Duplicate Line Remover

The Duplicate Line Remover deletes repeated lines from a list and keeps the first occurrence of each, in the original order. Paste email addresses, keywords, product codes, roll numbers or any line-by-line list, press Remove duplicates, and you get the unique entries plus a count of how many lines went in, came out and were removed.

How a duplicate is decided

Two lines are duplicates only if they are exactly the same, character for character:

• Case matters. "Apple" and "apple" are different lines, so both are kept.
• Spaces matter. "banana" and "banana " with a trailing space are different lines.
• Order is preserved. The first time a line appears it stays where it was; later copies are removed. Nothing is sorted.
• Blank lines are lines too. The first empty line is kept and any later empty lines are removed as duplicates of it.

Worked example with the three counters

Input (with two empty lines before "cherry"):

apple
banana
apple
Apple
banana


cherry

Output:

apple
banana
Apple

cherry

The stats read Lines in: 8, Lines out: 5, Removed: 3. The second "apple", the second "banana" and one of the empty lines were removed. "Apple" survived because of its capital A. If the second "banana" had ended with an invisible space, it would have survived too, which is the most common reason a list still looks duplicated after cleaning.

Getting a truly clean list

Most "the tool missed a duplicate" cases come from invisible differences. Normalise the list first, then remove duplicates:

1. Strip leading and trailing spaces with Remove Extra Spaces.
2. If capitalisation does not matter, as with email addresses or hashtags, convert everything to lowercase.
3. Remove duplicates here.
4. Sort the result if you want it alphabetical, using Text Sorter A–Z.

Phone numbers need extra care. "+91 98765 43210", "09876543210" and "9876543210" are the same number written three ways, and all three will be kept until you rewrite them in one format.

Everyday uses

• Merging two registration or attendance lists and keeping each name once.
• Cleaning an email list before a mail merge so nobody receives the same message twice.
• De-duplicating keyword lists gathered from several SEO tools.
• Reducing a log file to the distinct error messages it contains.
• Tidying tags, SKUs or pin codes copied from a spreadsheet column.

Limits to keep in mind

The comparison is on whole lines. For a CSV file, two rows count as duplicates only if every column matches; the tool cannot de-duplicate by a single column such as email while ignoring the name column. It also does not catch near-duplicates such as spelling variants. To remove repeated words inside a sentence rather than repeated lines, use Remove Duplicate Words.