Revision history [back]

To combine a series of PySpark dataframes using shared keys, you can follow the following procedure:

1. Import the PySpark module and create a SparkSession:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Combine DataFrames").getOrCreate()

1. Read in the dataframes from your desired sources:

df1 = spark.read.csv("path/to/df1.csv", header=True)
df2 = spark.read.csv("path/to/df2.csv", header=True)
df3 = spark.read.csv("path/to/df3.csv", header=True)

1. Use the join function to merge the dataframes using their shared columns:

combined_df = df1.join(df2, on="shared_column").join(df3, on="shared_column")

Note that the join function takes in the name of the shared column as an argument. You can continue chaining join functions as needed to combine more dataframes.

1. If necessary, you can rename columns using the .withColumnRenamed() function:

combined_df = combined_df.withColumnRenamed("old_column_name", "new_column_name")

1. Finally, you can write the combined DataFrame to a desired output format (e.g. CSV, Parquet, etc.):

combined_df.write.csv("path/to/output.csv", header=True, mode="overwrite")

Note: The above code assumes that you are using the PySpark API for version 2.0 or higher. If you are using an earlier version, some of the syntax may be different.