Invisible Characters: URL Encoding, Entities, Fullwidth

#Symptom: identical appearance, different behaviour

Character bugs share one trait: they display correctly but behave incorrectly, because what changed is the bytes rather than the glyph. Comparing by eye will never find them; only a tool can expose the invisible part.

Four typical symptoms, each with a different cause:

#Fullwidth and halfwidth: nearly indistinguishable

A fullwidth digit and a halfwidth digit differ by a little width on screen and by a completely different character in code. Text copied from a PDF, Excel, a legacy system or certain input methods carries them often.

The rule is blunt: anything in U+FF01 to U+FF5E is a fullwidth ASCII variant, so subtracting 0xFEE0 gives the halfwidth form, and U+3000 becomes an ordinary space.

The fullwidth-halfwidth converter does it in one pass and reports how many characters actually changed - a count of zero means there was nothing to convert.

This matters in interface design: in mixed Chinese and Latin text a fullwidth comma occupies a full character width while a halfwidth comma occupies half, so mixing them makes the visual rhythm of a paragraph lurch. Decide which set you want and convert in bulk rather than editing by hand.

#HTML entities: decode exactly once

Entity decoding has to be a single scan, because step-by-step replacement decodes twice - and that is not "one layer too many", it is tampering. < should become <; a naive multi-pass turns it into a bare <.

Encoding covers four characters only: &, <, >, ". The order is fixed too: ampersand first, otherwise entities produced later get escaped by their own pass.

With the HTML entity encoder, two boundaries are worth knowing:

• A single quote is not encoded by default; add ' by hand when an attribute is wrapped in single quotes or JSON is embedded in onclick
• Named entities go far beyond these four:  , © and — are left as-is, because the tool does not carry an entity table

#URL encoding: plus is not a space

In URL encoding a space is %20; a plus sign means space only in form encoding (application/x-www-form-urlencoded). That historical split is the most common reason a parameter never arrives.

With the URL encoder and decoder, the rules follow encodeURIComponent: letters, digits and - _ . ! ~ * ' ( ) pass through, everything else is split into UTF-8 bytes and written as %XX per byte.

That makes the group count informative:

• One Chinese character is three groups (中 → %E4%B8%AD)
• One emoji is four groups (%F0%9F%98%80)

Seeing groups in threes usually means the source was Chinese text.

Two boundaries to remember:

• You cannot encode a whole URL with it. The : and / in https://a.com/p become %3A%2F%2F and the link dies. Keep scheme, host and path as they are and encode only the query keys and values
• Decoding never turns + back into a space. a+b copied from a query string decodes to a+b. For form semantics, replace + with a space yourself first

#The dollar sign in a replacement

A dollar sign in the replacement text is always special, whether or not regex mode is on - because the operation ends up in String.replace. That is the usual reason a replacement loses characters.

With find and replace, three rules cover it:

• In plain mode, typing $1 does not insert a literal $1; it references the first capture group, and plain mode has none, so the result is empty
• For a literal dollar sign, write $$
• To insert the whole match, write $&

Plain mode automatically escapes regex metacharacters in the search text, so 价格(元) matches literally. Regex mode does not escape, and a broken pattern reports the engine's own message instead of silently returning the input.

Write the pattern itself without /.../gi delimiters or flags - case sensitivity and replace-all are controlled by checkboxes, and anything written into the pattern is matched as literal characters.

#Which tool to use

A sensible order: check the characters first (fullwidth?), then the escape depth (decoded how many times?), then the encoding rules (plus sign? percent sign?). Those three steps locate almost every "looks the same but is not" bug.

#Frequently asked questions

#The number I copied is correct. Why does validation fail?

Almost certainly fullwidth digits. １２３ and 123 look nearly identical but have different bytes, so a \d regex, parseInt or a database column will all reject one of them. Run it through the fullwidth-halfwidth converter: the changed-character count tells you immediately whether this was the problem.

#The page shows a literal <. How do I fix it?

The string's ampersand was escaped to &, so the browser decodes once and leaves <, which then renders as text. Either escape one time fewer, or only escape the four necessary characters when storing. Decode once with the HTML entity encoder to see what the original content really was.

#Should a space in a URL be %20 or +?

It depends on context. In a path only %20 is valid; in a query string, form encoding accepts +, but under strict RFC 3986 semantics + is an ordinary character. Across systems, %20 is never wrong, while + requires the receiver to parse form rules.

#Why did $1 become empty in my replacement?

Because plain mode has no capture groups, so $1 references nothing. Either switch to regex mode with a real group such as (\d+) and use $1, or escape the literal text as $$1. The behaviour comes from String.replace, not from the tool.

#Why is one Chinese character so long in a URL?

In UTF-8 a common Chinese character takes three bytes, and each byte becomes one %XX group. That is the price of UTF-8, paid in exchange for one encoding shared by every language. GBK uses two bytes per character but travels far worse across systems.