How duplicates are detected
The text is split on line breaks, and each line is compared against a set of lines already seen. The first time a line appears it is kept, and every later copy is dropped, so the original order survives. Two checkboxes change what "the same" means.
- Trim lines (on by default): leading and trailing spaces are removed before comparing, and the cleaned version is what ends up in the output.
appleandappleare therefore one line. - Ignore case (off by default): the comparison key is lower-cased, so
AppleandAPPLEmatch. The line that survives keeps the capitalisation of its first appearance. - Sort A-Z: after removing duplicates the survivors are sorted with the browser's locale-aware comparison. Leave it off to keep your original sequence.
Worked example
Input of eight lines: apple, Banana, apple (leading space), APPLE, an empty line, cherry, another empty line, banana.
| Settings | Output | Removed |
|---|---|---|
| Trim on, ignore case off | apple, Banana, APPLE, (empty), cherry, banana | 2 |
| Trim on, ignore case on | apple, Banana, (empty), cherry | 4 |
The only difference between the rows is the case setting, which is why a list that "should" collapse sometimes does not: with case sensitivity on, APPLE and banana survive as separate entries.
Things to watch for
- Blank lines are lines too. The first empty line is kept as a normal entry and every later one is removed as a duplicate. If you want no blanks at all, use the sort tool, which has a "Remove empty lines" option.
- Whole lines only. The match is on the entire line.
apple pieandappleare different lines, and the tool never deduplicates words inside a line. - Trim off means exact. A trailing space or tab makes two visually identical lines different. If a deduplication seems to miss obvious copies, switch trimming on or clean the text first with Text Cleaner.
- Accents count as different.
caféandcafeare distinct, because the comparison only changes case.
The counters above the output show unique lines and the number removed, which makes it easy to confirm that, for example, a 5,000-address list lost the number of repeats you expected. For ordering the survivors afterwards, Sort Lines offers numeric and reverse options as well.