Remove Duplicate Lines
Strip repeated lines from a list, keeping the first or last occurrence, with trim, case-fold, blank-line and sort options. Shows how many lines were removed.
Lines in
1
Lines kept
1
Removed
0
Two lines are duplicates when they are exactly equal — after the optional trim and case-folding. “Keep first” preserves the original order of the lines that survive; “keep last” is useful when later entries override earlier ones (config files, key = value lists). This does not touch duplicate words within a line. Runs in your browser.
De-duplicating a list without losing order
Removing duplicate lines sounds simple, but two decisions shape the result: what counts as "the same line", and which copy to keep. This tool makes both explicit. Lines are compared as whole strings, optionally trimmed of surrounding whitespace and folded to a common case first, so you decide whether "Apple", " apple " and "apple" are one entry or three.
"Keep first" is the default and preserves the original order of the lines that survive — the natural choice for de-duplicating a mailing list, a set of URLs, or search keywords. "Keep last" is the right choice when later entries are meant to win: environment files, key = value configuration, or an append-only log where the newest value for a key should stand.
The optional extras handle the messy parts of pasted data. "Trim before comparing" stops trailing spaces from splitting entries that are really identical. "Drop blank lines" clears the gaps left by copying from a document. "Sort the result" applies a natural, locale-aware sort after de-duplication, so you can clean and alphabetise in one step. The tool also reports how many lines it removed, which is a quick sanity check on how much duplication your data had.
This works on whole lines only — it will not touch repeated words inside a line. For that, use Find and Replace. And because the whole operation is a single pass with a hash set, lists of tens of thousands of lines process instantly.
The text never leaves your browser. You can paste private lists and the tool still works with the network disconnected.
Related tools
These free tools pair well with this page — open them in a new tab to finish your workflow.
Frequently Asked Questions
When are two lines considered duplicates?
When they are exactly equal after the options you have chosen are applied — optionally trimmed of surrounding whitespace and folded to the same case. By default the comparison is case-sensitive and whitespace is trimmed.
What is the difference between "keep first" and "keep last"?
"Keep first" removes every repeat after the first time a line appears, preserving the original order of the survivors. "Keep last" keeps the final occurrence instead — useful for key = value lists or config lines where a later entry is meant to override an earlier one.
Does it remove blank lines?
Only if you tick "also drop blank lines". Otherwise blank lines are treated like any other line: the first blank is kept and later blanks are removed as duplicates.
Can I sort the output at the same time?
Yes — tick "sort the result" and the kept lines are sorted with natural, locale-aware ordering after deduplication. Leave it off to keep the lines in their original order.
Does it remove duplicate words within a line?
No. This tool works on whole lines. To change repeated text inside lines, use Find and Replace.
How many lines can it handle?
Tens of thousands of lines are fine — deduplication is a single pass with a hash set. Extremely large inputs are limited only by browser memory since everything is processed locally.
Is anything sent to a server?
No. The text never leaves your browser, and the tool works with the network disconnected.