Ask Your Question

What is the method to eliminate duplicate lines from a large Python file?

asked 2022-08-02 11:00:00 +0000

pufferfish gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2023-02-07 15:00:00 +0000

ladyg gravatar image

You can use the following Python code to eliminate duplicate lines from a large file:

with open('file.txt', 'r') as f:
    lines = set(f.readlines())

with open('file.txt', 'w') as f:
    for line in lines:

Explanation: 1. We open the file in read mode and use the readlines() method to extract all the lines of the file. 2. We use the set() method to eliminate duplicate lines from the list of lines. 3. We re-open the file in write mode and write the non-duplicate lines to the file using a loop.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer

Question Tools


Asked: 2022-08-02 11:00:00 +0000

Seen: 17 times

Last updated: Feb 07 '23