Revision history [back]
 | 1 | initial version |
Yes, Spark SQL can perform merge schema function for parquet files. When reading multiple parquet files with different schemas, Spark SQL can automatically merge the schemas by inferring the common schema. This is done by comparing the column names, data types, and nullable attributes of each schema to find common fields that can be combined into a single schema. The resulting schema will have all the fields from all the input schemas, with appropriate data types and nullable attributes. This can be done using the "mergeSchema" option in the Spark SQL configuration.