You can use the following Python code to eliminate duplicate lines from a large file:
with open('file.txt', 'r') as f:
lines = set(f.readlines())
with open('file.txt', 'w') as f:
for line in lines:
f.write(line)
Explanation:
1. We open the file in read mode and use the readlines()
method to extract all the lines of the file.
2. We use the set()
method to eliminate duplicate lines from the list of lines.
3. We re-open the file in write mode and write the non-duplicate lines to the file using a loop.
Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss
Asked: 2022-08-02 11:00:00 +0000
Seen: 11 times
Last updated: Feb 07
How can I set up Gunicorn with a Django Project?
Looking for a Python Module that finds Tags for a Text describing its Content
Need a Function in Python to remove entries less than 2 digits from an Array
How can I convert a Document in Python?
How can I program a Loop in Python?
How can I enable Python Code Highlighting in Askbot?