Text Diff

How the comparison is computed

Both texts are split into lines, and the lines they have in common are found as a longest common subsequence - the same idea the Unix diff command is built on. Everything in that subsequence keeps its place; whatever is left is reported as removed from the left or added on the right. Where a removal and an addition end up adjacent, they are paired and shown as a modification rather than as two unrelated events.

Each modified pair then gets a second pass at word level, so a long line with one changed number highlights that number instead of turning the whole line red. This is the difference between a diff you can read and a diff you have to squint at, and it is why line-level tools feel unhelpful on prose and configuration files.

Before any of that, identical leading and trailing lines are trimmed off. Two thousand-line files that differ in one place are therefore compared as a handful of lines, which is both faster and clearer.

When two identical-looking lines are marked as different

This is the most common surprise, and the cause is always a character you cannot see. Trailing spaces are the usual culprit, which is why ignoring them is on by default. Line endings are next: a file written on Windows ends its lines with a carriage return and line feed, one written on macOS or Linux with a line feed alone, and a diff that compares the raw bytes will call every line changed.

After that come tabs against runs of spaces, which the ignore-whitespace option collapses; the non-breaking space that word processors and some web pages insert in place of a normal one; curly quotation marks and en dashes substituted by autocorrect; and a byte order mark sitting invisibly at the start of the first line. Any of these will make two lines that look identical on screen genuinely different in the file.

The size ceiling

Longest common subsequence is quadratic: comparing m lines against n lines fills an m by n table. At a few hundred lines that is trivial, at ten thousand against ten thousand it is a hundred million cells, and the page would stop responding while it filled them.

The comparison is therefore budgeted at four million cells after the common prefix and suffix have been trimmed, and a comparison that would exceed it is refused with a message rather than attempted. In practice this means files of several thousand lines compare instantly as long as they are broadly similar, and only two genuinely unrelated large documents hit the limit. When they do, the right tool is a command-line diff that can stream.

Side by side or unified

Side by side keeps the two versions in their own columns and is the better view for prose, translations, and anything where you want to read both texts as texts. Unified interleaves them with markers, which is the format patches and code review use, and is easier to follow when changes are small and scattered through a long file.

Neither view changes what was computed; they are two presentations of the same result.

The algorithms behind a diff

The classic approach, published by Eugene Myers in 1986, finds the shortest edit script between two sequences - the smallest set of insertions and deletions that turns one into the other. It is what the Unix diff command and Git's default implementation use, and it is optimal in the sense of producing the fewest changes.

Fewest changes is not always most readable. Myers can align unrelated lines that happen to match - a closing brace here, a blank line there - producing a diff that interleaves two functions instead of showing one removed and one added. Git's patience algorithm addresses this by anchoring on lines that appear exactly once in both versions and recursing between those anchors, which usually matches what a reader considers the real correspondence.

The histogram algorithm, now Git's recommended alternative, extends patience with a frequency heuristic and is faster on large files. If a code review diff ever looks nonsensical, setting diff.algorithm to histogram frequently produces a clearer result from exactly the same two files.

Two-way differences and three-way merges

Comparing two files answers what changed. Merging needs a third input - the common ancestor - because without it there is no way to tell which side changed a line and which side simply has the original. That is why version control keeps a merge base and why three-way merge tools show three panes.

A conflict arises when both sides changed the same region relative to that ancestor. Everything else merges automatically, including changes that are adjacent but not overlapping, which is why two people can edit the same file all day without conflict and then collide on one line.

Understanding this changes how you resolve conflicts. The markers in a conflicted file show your version and theirs, but the useful question is what each side was trying to do relative to the base - and git checkout --conflict=diff3 adds the base to the markers, which usually makes the resolution obvious rather than a guess between two alternatives.

Diffing things that are not lines of text

Line-based comparison assumes meaningful line boundaries. Minified JavaScript, a single-line JSON payload and a long paragraph of prose all break that assumption: a one-character change reports the entire line as different. Pretty-printing both sides first restores the structure the diff needs, and for JSON a canonical form - sorted keys, consistent indentation - makes the comparison meaningful rather than incidental.

CSV has the opposite problem: line-based diff works but reports a column reordering as a change to every row. Sorting by a key column first, or using a CSV-aware tool, separates the change you care about from the noise.

Binary and near-binary formats need their own converters. A .docx is a zip of XML, a notebook is JSON with embedded output, and a PDF is largely opaque. Git supports textconv filters that convert a file to text before diffing, which is how teams get readable diffs of documents and notebooks without storing a second copy.

Reading a diff someone else wrote

A reviewable diff is mostly a matter of what is not in it. Formatting churn - a reformatter run over a file, line endings changed, an import sorter applied - buries the three lines that matter under three hundred that do not, and the usual remedy is to separate the mechanical commit from the substantive one so each can be read at its own speed.

When you are handed a mixed diff anyway, whitespace-insensitive comparison is the fastest way through it: git diff -w, or the ignore options here. Turn it back on before approving, because in some files whitespace is content - Python indentation, YAML structure, Makefile tabs, and any test that asserts on exact output.

Large renames are the other case worth handling deliberately. A file moved and edited appears as a deletion plus an addition unless the tool detects the rename, and Git's -M flag or a rename-aware review interface turns a thousand-line diff back into the ten lines that actually changed.

Comparing text as an editorial task

Not every diff is code. Comparing two drafts of a contract, two translations, or a document before and after an edit is a common use, and the side-by-side view exists for it: you want to read both texts as texts, not as a patch.

For prose, word-level highlighting carries most of the value, because a rewritten sentence shows as a rewritten sentence rather than a deleted line and an added one. The tools that do this well within a line are the ones people find usable for editing work, which is why the comparison here runs a second pass at word level inside each changed pair.

Two practical notes for documents. Text pasted from a word processor frequently carries curly quotation marks, en dashes and non-breaking spaces that the other copy does not, so differences appear where the text looks identical. And a document exported twice from the same source can differ in line breaks alone, which is why normalising line endings before comparing saves a lot of confusion.