Ask Your Question
0

What is the issue with the timestamp datatype in a Parquet file using pyarrow?

asked 2022-01-25 11:00:00 +0000

nofretete gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
3

answered 2021-10-09 01:00:00 +0000

lakamha gravatar image

There is a known issue with the timestamp datatype in a Parquet file using pyarrow where timestamps with nanosecond precision cannot be read correctly by some software systems or versions. This is because, in some cases, pyarrow writes the timestamp values using a format that is not fully compliant with the Parquet specification. This can cause data loss or inaccurate results when the Parquet file is read by other systems that do not support this format. To resolve this issue, it is recommended to either use a timestamp with microsecond precision, or to manually adjust the timestamp values before writing them to the Parquet file.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2022-01-25 11:00:00 +0000

Seen: 15 times

Last updated: Oct 09 '21