Remove Duplicate Lines
Strip repeated lines from a list, keeping the first or last occurrence, with trim, case-fold, blank-line and sort options. Shows how many lines were removed.
Lines in
1
Lines kept
1
Removed
0
Two lines are duplicates when they are exactly equal — after the optional trim and case-folding. “Keep first” preserves the original order of the lines that survive; “keep last” is useful when later entries override earlier ones (config files, key = value lists). This does not touch duplicate words within a line. Runs in your browser.
De-duplicating a list without losing order
Removing duplicate lines sounds simple, but two decisions shape the result: what counts as "the same line", and which copy to keep. This tool makes both explicit. Lines are compared as whole strings, optionally trimmed of surrounding whitespace and folded to a common case first, so you decide whether "Apple", " apple " and "apple" are one entry or three.
"Keep first" is the default and preserves the original order of the lines that survive — the natural choice for de-duplicating a mailing list, a set of URLs, or search keywords. "Keep last" is the right choice when later entries are meant to win: environment files, key = value configuration, or an append-only log where the newest value for a key should stand.
The optional extras handle the messy parts of pasted data. "Trim before comparing" stops trailing spaces from splitting entries that are really identical. "Drop blank lines" clears the gaps left by copying from a document. "Sort the result" applies a natural, locale-aware sort after de-duplication, so you can clean and alphabetise in one step. The tool also reports how many lines it removed, which is a quick sanity check on how much duplication your data had.
This works on whole lines only — it will not touch repeated words inside a line. For that, use Find and Replace. And because the whole operation is a single pass with a hash set, lists of tens of thousands of lines process instantly.
The text never leaves your browser. You can paste private lists and the tool still works with the network disconnected.
Worked example
Input (trim + ignore case on): anna@example.com Ben@example.com anna@example.com ben@example.com chris@example.com Output, keep first: anna@example.com Ben@example.com chris@example.com
With “keep last” the output would be anna@example.com, ben@example.com and chris@example.com — the later spelling of Ben wins. Keep last is useful when later lines are newer versions of the same record.
Cleaning real-world lists
- Email lists: turn on ignore case — email providers treat addresses case-insensitively in practice — and trim, because spreadsheet exports often carry trailing spaces.
- URLs: case matters in paths, so leave ignore case off; normalise trailing slashes first with Find and Replace if “/page” and “/page/” should count as one.
- Keyword lists: combine with sort to see near-duplicates (plurals, word order) next to each other, then decide by hand.
- Counting how many duplicates there were: compare the line counts before and after.
The same job in other tools
- Excel: Data → Remove Duplicates keeps the first occurrence of each row.
- Google Sheets: =UNIQUE(A:A) returns the distinct values in their original order.
- Command line: sort -u removes duplicates but also sorts the lines; awk '!seen[$0]++' file.txt keeps the original order and the first occurrence, like this tool’s default.
- Databases: SELECT DISTINCT, or GROUP BY when you also need a count of how many times each value appeared.
This page is quicker for a one-off list pasted from an email or a spreadsheet column, and nothing you paste is uploaded.
Before you deduplicate
Decide what “the same” means for your data first. Two lines that differ only by a trailing space, a capital letter or a trailing slash may or may not be duplicates depending on the job — a username list is usually case-insensitive, a list of file paths on Linux is not. Run the tool once with the options off, then turn them on one at a time and watch the kept count, so you can see exactly which rule removed which lines.
Related tools
These free tools pair well with this page — open them in a new tab to finish your workflow.
Frequently Asked Questions
When are two lines considered duplicates?
When they are exactly equal after the options you have chosen are applied — optionally trimmed of surrounding whitespace and folded to the same case. By default the comparison is case-sensitive and whitespace is trimmed.
What is the difference between "keep first" and "keep last"?
"Keep first" removes every repeat after the first time a line appears, preserving the original order of the survivors. "Keep last" keeps the final occurrence instead — useful for key = value lists or config lines where a later entry is meant to override an earlier one.
Does it remove blank lines?
Only if you tick "also drop blank lines". Otherwise blank lines are treated like any other line: the first blank is kept and later blanks are removed as duplicates.
Can I sort the output at the same time?
Yes — tick "sort the result" and the kept lines are sorted with natural, locale-aware ordering after deduplication. Leave it off to keep the lines in their original order.
Does it remove duplicate words within a line?
No. This tool works on whole lines. To change repeated text inside lines, use Find and Replace.
How many lines can it handle?
Tens of thousands of lines are fine — deduplication is a single pass with a hash set. Extremely large inputs are limited only by browser memory since everything is processed locally.
Is anything sent to a server?
No. The text never leaves your browser, and the tool works with the network disconnected.