Revision history [back]

You can use the following Python code to eliminate duplicate lines from a large file:

with open('file.txt', 'r') as f:
    lines = set(f.readlines())

with open('file.txt', 'w') as f:
    for line in lines:
        f.write(line)

Explanation: 1. We open the file in read mode and use the readlines() method to extract all the lines of the file. 2. We use the set() method to eliminate duplicate lines from the list of lines. 3. We re-open the file in write mode and write the non-duplicate lines to the file using a loop.