Ask Your Question
1

Does the atomicity property apply to Spark's overwrite save mode?

asked 2023-06-03 23:54:19 +0000

ladyg gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2023-06-03 23:58:02 +0000

djk gravatar image

Yes, the atomicity property applies to Spark's overwrite save mode. In Spark's overwrite mode, the entire output directory is deleted and re-written with the new data. This operation is atomic, meaning that it is either completed successfully or is not executed at all, ensuring that data consistency is maintained. Therefore, if the write operation fails for any reason, the previous data is not lost and is still available.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2023-06-03 23:54:19 +0000

Seen: 18 times

Last updated: Jun 03 '23