There is a known issue with the timestamp datatype in a Parquet file using pyarrow where timestamps with nanosecond precision cannot be read correctly by some software systems or versions. This is because, in some cases, pyarrow writes the timestamp values using a format that is not fully compliant with the Parquet specification. This can cause data loss or inaccurate results when the Parquet file is read by other systems that do not support this format. To resolve this issue, it is recommended to either use a timestamp with microsecond precision, or to manually adjust the timestamp values before writing them to the Parquet file.
Asked: 2022-01-25 11:00:00 +0000
Seen: 15 times
Last updated: Oct 09 '21