Revision history [back]

One option would be to decrease the row group sizes until the total allocation falls within the 95.00% limit of the driver_memory. This can be done by adjusting the parquet.block.size and spark.sql.parquet.row.group.size parameters.

Another option would be to increase the driver_memory to accommodate the total allocation. However, this may not be feasible if there are other constraints or limitations in the system.

It's important to note that row group sizes should generally be chosen based on the data characteristics and workload requirements, rather than solely based on memory constraints.