How can I use Apache Spark with Python?

How can I use Apache Spark with Python?

You can use PySpark, which is the Python API for Apache Spark. It allows you to write Spark applications using Python.

What are the main features of PySpark?

PySpark provides features like distributed data processing, support for SQL queries, machine learning libraries, and data streaming.

Can you show me a simple example of using PySpark?

Sure! Here's a simple example:

from pyspark.sql import SparkSession

# Create a Spark session
spark = SparkSession.builder.appName('example').getOrCreate()

# Create a DataFrame
data = [(1, 'Alice'), (2, 'Bob')]
columns = ['id', 'name']
df = spark.createDataFrame(data, columns)

df.show()