Remove Duplicate Words

Remove Duplicate Words scans your text from left to right and deletes every word that has already appeared earlier, so the the quick brown fox fox becomes the quick brown fox. It works best on keyword lists, hashtag sets, tag clouds and transcripts full of stutters, and it follows a few matching rules worth knowing before you paste in a whole essay.

How the tool decides two words are the same

The text is split wherever there is whitespace. For each piece, the tool builds a comparison key by lowercasing it and throwing away every character that is not an English letter a to z or a digit 0 to 9. Two pieces with the same key count as duplicates. In practice that means:

• Case and punctuation are ignored: Yes, yes and YES! are one word.
• Some different words collide: it's and its both become "its", and U.S. and us both become "us", so the second one is deleted.
• Non-English scripts are never removed: a Hindi or Tamil word has no a to z letters, its key is empty, and the tool keeps it every time. नमस्ते नमस्ते दोस्त comes back unchanged. Emoji and stand-alone dashes behave the same way.
• The first copy wins: it stays in place with its original capitals and punctuation, while later copies disappear together with any comma or full stop stuck to them.

Worked example with punctuation

Input: Yes, yes, YES! We did it, we did.

Output: Yes, We did it,

The second and third "yes" go, the lowercase "we" matches the earlier "We", and "did." matches "did", taking the final full stop with it. On a hashtag list the result is cleaner: #travel #India #travel #goa travel #Goa becomes #travel #India #goa, because the plain word travel and #Goa reduce to keys that already exist.

Why your line breaks may disappear

After deleting words, the tool squeezes any run of two or more whitespace characters into a single space and trims both ends. A blank line between paragraphs is two line breaks, so it turns into a space, and a line break sitting next to a deleted word merges with the space beside it. red apple, red car, a blank line, then green apple comes out as red apple car green on one line. If your data is one item per line and you want to drop repeated lines rather than repeated words, the Duplicate Line Remover keeps the line structure intact.

When to use it, and when not to

It is a quick fix for speech-to-text output such as I I think the the meeting is is at five, which becomes I think the meeting is at five. The catch is that it removes every later occurrence, not only neighbouring doubles. In a normal paragraph, the second "the", "is" and "a" anywhere in the text will vanish and the prose will stop making sense. For essays, fix doubled words by hand, and use the word frequency counter first if you want to see how often each word appears before deleting anything.