Strip duplicate lines from text. Useful for cleaning lists, log files, and CSV data.
Cleaning email lists, deduplicating CSV exports, removing repeated log entries, finding unique URLs from a crawl, or building a unique vocabulary list from text are all common use cases. The tool preserves order by default (keeping the first occurrence) and offers options to sort or to keep the last occurrence instead.
By default, "Apple" and "apple" are different lines. Toggle "Ignore case" to treat them as duplicates.
On Linux/macOS: sort file.txt | uniq removes duplicates after sorting. awk '!seen[$0]++' file.txt preserves order.
Marketers cleaning a mailing list before a campaign use this to strip repeated email addresses that crept in from merging exports of several signup forms, avoiding the awkwardness (and reputation damage) of sending the same message to someone twice. Developers debugging log output paste in a chunk of repeated error lines to see only the distinct messages, which makes it much faster to spot how many unique failure types actually occurred versus how many times each one repeated. SEO researchers pull a list of URLs from a site crawl and dedupe it to get an accurate count of unique pages rather than one inflated by duplicate listings. Students and writers building a word list or glossary from a block of text use it to collect unique terms without a repeated entry cluttering the result. It's also useful for merging two contact lists, two inventories, or two spreadsheets exported separately, where overlapping entries need to collapse into one clean combined list. The option to keep the first versus last occurrence matters when the duplicate rows aren't identical — for example, keeping the most recently updated version of a repeated record rather than the oldest one.
Want more detail? Read How to Use Remove Duplicate Lines Tool: Practical Guide and Best Practices.