How it works
Each line of your text is compared against every line seen before it. The first occurrence of a line is kept; every repeat after that is removed, and the original line order is otherwise preserved (unless you turn on alphabetical sorting).
The options
- Case-sensitive: On by default. When off, "Apple" and "apple" count as the same line and only the first is kept.
- Trim whitespace: On by default. Removes leading and trailing spaces from each line before comparing, so "apple" and "apple " (with a trailing space) count as duplicates rather than being treated as different lines.
- Remove empty lines: Off by default. When on, blank lines are dropped entirely rather than being treated as one shared "empty" duplicate.
- Sort alphabetically: Off by default. When on, the deduplicated result is sorted rather than kept in its original order.
Common uses
Cleaning up a list of email addresses or usernames before importing them somewhere, removing repeated rows copied from a spreadsheet, deduplicating a keyword or tag list, or tidying up any list-style data where the same line ended up entered more than once.
Frequently asked questions
Does this tool keep the first or last copy of a duplicate line?
The first occurrence is always kept, and later repeats are removed. The original order of the remaining lines is preserved unless you turn on alphabetical sorting.
Will "Apple" and "apple" be treated as duplicates?
Only if you turn off the "Case-sensitive" option, which is on by default. With it off, different capitalizations of the same text are treated as the same line, and only the first is kept.
Does this handle Windows-style line endings?
Yes. Text with Windows-style (CRLF), Mac-style (CR), or Unix-style (LF) line endings is all split into lines correctly before comparing.