Duplicate Line Remover

Removed{{ resultsReady ? formatInteger(computation.values.summary.removed_lines) : '—' }} Duplicate groups{{ resultsReady ? formatInteger(computation.values.summary.duplicate_groups) : '—' }} Match{{ case_sensitive ? 'Exact case' : 'Ignore case' }}

Line list:

Paste emails, URLs, keywords, IDs, or other one-entry-per-line text.

{{ fileStatus || 'Drop TXT, CSV, or LOG onto the textarea. Maximum 512 KiB.' }}

Case-sensitive match:

Ignore case is useful for contact, keyword, and copied export lists.

Trim line edges:

Keep this on for pasted spreadsheet or CRM lists with stray edge spaces.

Ignore blank lines:

Dropping blanks keeps list exports compact.

Keep rule:

The rule applies independently to each matched duplicate group.

Output order:

Source order is best for audits; alphabetical order is useful for reusable lists.

Collapse inner whitespace for matching:

The neutral default keeps inner whitespace exact.

{{ normalize_internal_spaces ? 'Collapsed for match' : 'Exact inner whitespace' }}

{{ cleanedText }}

The chart renderer is unavailable. The same counts remain available in the summary and ledger.

Match key	Occurrences	Kept line	Removed lines	Kept text	Copy
{{ row.display }}	{{ row.occurrences }}	{{ row.kept_line_number \|\| 'None' }}	{{ formatLineNumbers(row.removed_line_numbers) }}	{{ row.kept_text \|\| 'Removed all occurrences' }}
No duplicate groups Every comparable line is unique under the current matching rules.

Duplicate removal sounds simple until two lines look alike but are not identical. Capitalisation, spaces at the edges, repeated spaces inside a value, and blank rows can all change whether a list contains one item or several. The right rule depends on what the lines represent.

Email addresses copied from a spreadsheet may need case-insensitive matching and trimmed edges. Source-code fragments may need exact case and spacing. A log or CSV export may not be suitable at all if one logical record spans several lines, because line-based cleanup treats every line break as an item boundary.

Identity rule: decide whether case and whitespace differences should count as meaningful.
Retention rule: keep the first occurrence, keep the last occurrence, or remove every value that appears more than once.
Order rule: preserve source order for auditability or sort only after the surviving lines have been chosen.

The safest workflow keeps the original list available until the cleaned output and duplicate groups have been reviewed. Deduplication can remove valid distinctions when the comparison rule is broader than the data's real identity rule.

How to Use This Tool:

Start with the narrowest matching rule that reflects the list, then inspect which original rows were grouped together.

Paste a one-entry-per-line list into Line list, or load a TXT, CSV, or LOG file no larger than 512 KiB.
Choose whether matching is case-sensitive, whether line edges are trimmed, and whether blank lines are ignored. Leave inner-space collapse off unless repeated spaces and tabs should compare as one space.
Select the Keep rule and Output order. Sorting happens after the retained occurrence in each group has been chosen.
Read the removed count, check Cleaned list, and use Duplicate ledger to verify the kept and removed source line numbers before copying the result.

Interpreting Results:

Removed counts rows excluded by the keep rule, while Duplicate groups counts distinct comparison keys that appeared more than once. These numbers answer different questions: one large group can remove many rows.

An empty duplicate ledger means every comparable line is unique under the current rules. It does not prove that the underlying records are unique by email column, customer ID, or another field hidden inside a wider line.

Technical Details:

Line cleanup is a stable grouping transformation. Line endings are first normalised to a single line-feed form. Each remaining row produces a comparison key, equal keys form a group, and the retention rule selects zero or one row from each group. Matching is global, so duplicate rows do not need to be adjacent.

Transformation Core:

Stages used to compare and retain lines
Stage	Transformation	Effect
Split	Carriage-return line endings become line feeds, then the text is split at every line break.	Every split row contributes to Original lines.
Trim	When enabled, leading and trailing whitespace is removed before comparison and from retained output.	A row containing `beta` is retained as `beta`.
Inner whitespace	When enabled, each run of whitespace inside a trimmed row becomes one space for comparison only.	The kept row can preserve its original inner spacing even when its key is collapsed.
Case	Case-insensitive mode lowercases the comparison key; case-sensitive mode leaves it unchanged.	`Alpha` and `alpha` either merge or remain separate.
Blank policy	Ignored blank rows are removed before grouping.	They increase Ignored blank lines but not Comparable lines.
Retention and sort	The keep rule selects survivors in source order; optional A-Z or Z-A sorting runs afterward.	Sorting never changes which source occurrence was kept.

Rule Core:

Duplicate group retention rules
Keep rule	Survivor from each comparison group
First occurrence	The earliest source row survives.
Last occurrence	The latest source row survives.
Only lines that appear once	A one-row group survives; every row in a repeated group is removed.

Formula Core:

The count identities are based on comparable rows after the blank-line rule. Kept and removed rows partition that comparable set.

RemovedLines = ComparableLines - KeptLines

For the lines alpha, beta, Alpha, beta , gamma, and beta, ignoring case and trimming edges creates three keys. Keeping the first occurrence retains three rows and removes three.

Inputs are limited to 512 KiB, 20,000 split lines, and 2,000 Unicode code points per line. A null control character is rejected. CSV content is handled as lines of text rather than parsed fields, so embedded line breaks and field-level duplicate checks require a CSV-aware workflow.

Privacy Notes:

Pasted text and selected files are read and processed in the current browser session; the cleanup path does not upload their contents. Keep the original available until the result is checked, and handle copied or downloaded lists according to the sensitivity of the entries.

FAQ:

Does this remove duplicate values from one CSV column?

No. Each physical CSV row is treated as one text line. Extract the target column first when duplicate identity belongs to a field rather than the whole row.