Clean a Long List: Deduplicate, Sort and Reverse Text Lines

#Three operations, three different jobs

Deduplication removes repeated lines, sorting reorders them by content, and reversing flips their positions — none of them substitutes for another, and picking the wrong one just scrambles your data. Each one uses a different rule to decide what happens:

The most common mix-up is using reverse to flip a log. To put the newest entries first, use descending order in the sorter, or sort on a timestamp column. Character reverse will flip the contents of every line too, leaving you with text nobody can read.

#Deduplication: confirm three defaults first

Deduplication treats two lines as duplicates only when they match exactly, character for character. Case, leading and trailing spaces, and invisible characters all count, so Apple and apple stay separate by default.

Three toggles cover the three kinds of near-duplicates you actually run into:

The output keeps the first occurrence of each line, including its original spelling and position, and the status bar reports how many lines were dropped.

These three lines look identical but are three different lines — the first and third end differently, the second differs in case:

Apple
apple
Apple 

Normalise case and spacing first, then deduplicate:

apple
apple

#Sorting: ascending means characters, not numbers

The default ascending sort is lexicographic. It compares the first character, then the second, and so on. Since the character 1 sorts below 9, the line 10 lands before 9.

That is not a bug, it is what lexicographic ordering means. The same set of filenames, in both modes:

Lexicographic (default ascending)
file1.txt
file10.txt
file2.txt
file9.txt

Natural sort (number aware)
file1.txt
file2.txt
file9.txt
file10.txt

Any line that carries a number should be sorted with natural order. Version numbers, chapter lists, order IDs and IP lists all fall into this category.

#Chinese text sorts by code point, not by pinyin

Chinese characters have no uppercase or lowercase, so ascending order compares code point values — the result is unrelated to how the characters are pronounced.

If you need pinyin order, the tool will not transliterate for you. Two options:

• Paste a pinyin column in front of each line with Find and Replace, sort, then strip the prefix
• Keep code point order, which is stable and reproducible — the same data sorts the same way on every machine

#Mixed case with ignore case enabled

With ignore case on, Apple, apple and APPLE collapse into a single key. Which spelling survives depends on which one appeared first in your input, not on a case rule. Leave the option off if you need the variants kept apart.

#Reversing: decide what you actually want flipped

Text Reverser has four modes — characters, lines, within-line and words — and only line mode leaves the contents of each line unchanged.

The same input, two modes, very different results:

input:      line 1 / line 2 / line 3
by lines:   line 3 / line 2 / line 1
by chars:   3 enil / 2 enil / 1 enil

CJK characters and emoji are handled as whole characters. That matters more than it sounds: a naive algorithm that reverses 16-bit units splits an emoji or a rare ideograph in half and prints a replacement box. This tool does not.

#Which tool for which job

#Frequently asked questions

#Does deduplicating change the order of my lines?

No. Deduplication keeps the first occurrence of each line, with its original position and spelling, so the output order matches the input order minus the removed duplicates. If you want the result alphabetised, run the sorter afterwards as a separate step.

#Why does the sorted Chinese text look random?

It is not broken — code point order simply differs from pronunciation order. Chinese characters have no case, so ascending order compares their code point values, and the sequence of characters follows those values rather than how they are read. For pinyin order, prepend a pinyin column, sort, then remove it.

#Why did Chinese characters or emoji turn into boxes after reversing?

Characters were split apart. Emoji and some rare ideographs occupy two 16-bit units internally, and reversing unit by unit cuts a single character in half. Text Reverser handles CJK and emoji safely. If you write your own script instead, iterate by code point rather than by 16-bit unit.

#A column pasted from a spreadsheet has extra leading spaces. How do I clean it first?

Use Find and Replace to strip the spaces, or a regex when the pattern is irregular. Then verify the line count with Text Statistics to make sure no whole line was lost, and only then deduplicate. Doing it in the opposite order leaves near-duplicates behind as separate lines.

#Does any of this data leave my browser?

No. All three tools run locally in the browser, and nothing is uploaded to a server, so they are safe for lists of phone numbers or email addresses. Thousands of lines process instantly; you only notice a delay when tens of thousands of result rows have to be rendered at once.

#Why do my line counts disagree between tools?

Inconsistent line endings are the usual cause. Windows files use CRLF and Linux uses LF, and some editors also treat a bare carriage return as a line break, so the same file reports different totals in different tools. Deduplication can also miss duplicates when an invisible character hides at the end of a line. Normalise line endings first, then confirm the count with Text Statistics.