Duplicate Line Remover
Remove duplicate lines from pasted lists with switches for keep order, ascending/descending sort, case rules, trim, and blank-line handling.
It also supports Unicode-aware comparison, quick copy/download, and multilingual list cleanup.
What this tool does
Use it to deduplicate one-item-per-line lists such as tags, emails, URLs, product codes, or copied research notes. Keep the first appearance exactly where it appeared, or sort the unique result for a cleaner handoff.
How to use
- Paste your list into the input box, one item per line.
- Adjust order, case matching, trim, and blank-line handling if needed.
- Click Remove duplicates, then copy or download the cleaned result.
Examples
Remove duplicates while keeping the original order
Input
Alpha Beta Alpha Gamma Beta
Output
Alpha Beta Gamma
Ignore case + trim
Input
Alpha alpha ALPHA Beta
Output
Alpha Beta
Sort while removing duplicates
Input
item 10 item 2 item 2 item 1
Output
item 1 item 2 item 10
Glossary
Comparison key
The internal comparison value after trim, Unicode normalization, and case handling. It is separate from the original display line.
Trim
Removes leading and trailing spaces from each line before comparing. It does not rewrite inner spacing.
NFC
One Unicode normalization form that reduces differences between visually identical characters with different internal representations.
Locale-aware sort
Sorting that follows the active page language instead of raw code-point order. This can change how accents and mixed scripts are ordered.
FAQ
Does it keep the original order?
Yes. Keep order is the default. The tool keeps only the first occurrence of each line and preserves that first-appearance order unless you switch to ascending or descending sort.
Can I ignore case for multilingual text?
Yes, but case handling follows the current page language. For mixed-language lists, especially with locale-specific letters, results can differ from what you expect.
What changes when trim is on?
Leading and trailing spaces are removed before comparison. That makes `Alpha` and ` Alpha ` count as the same line.
How are blank lines handled?
Blank lines are excluded by default. If you turn that off, blank lines are compared too, and repeated blank lines collapse into a single kept blank line.
Can visually identical lines still fail to match?
Yes. NFC normalization helps with many composed/decomposed character differences, but zero-width characters and width differences can still keep lines apart.
Is my input sent to a server?
No. Everything runs locally in your browser, and neither the input text nor the result is stored in the URL.
Notes
- Ignore-case matching depends on the current page language, so mixed-language lists need extra care.
- Zero-width characters and full-width/half-width differences are not unified automatically. If duplicates remain visible, inspect the text with an invisible-character tool.
- This version keeps the first occurrence of each line. It does not offer a keep-the-last-occurrence rule.