If I have set the driver_memory and there are 8 writers, how can I adjust the row group sizes to fit within 95.00% of the heap memory, which is equivalent to 1,020,054,720 bytes, despite the total allocation exceeding this limit?

edit

One option would be to decrease the row group sizes until the total allocation falls within the 95.00% limit of the driver_memory. This can be done by adjusting the parquet.block.size and spark.sql.parquet.row.group.size parameters.

Another option would be to increase the driver_memory to accommodate the total allocation. However, this may not be feasible if there are other constraints or limitations in the system.

It's important to note that row group sizes should generally be chosen based on the data characteristics and workload requirements, rather than solely based on memory constraints.