Propositional and predicate logic
Every precondition, loop guard and test assertion you have ever written is a formula in a logic that has been fully worked out since 1879. Knowing its rules turns "I think this condition is right" into "this condition is provably equivalent to that one".
A proposition is a statement that is definitely true or definitely false. 7 is prime is a proposition. x > 3 is not — it has a free variable, so its truth depends on x. That distinction is the whole difference between propositional logic and predicate logic, and it is also the difference between a boolean constant and a boolean-valued function in your code.
The connectives, and the only one people get wrong
Five connectives generate everything: negation ¬, conjunction ∧, disjunction ∨, implication →, and biconditional ↔. Four of them match intuition. Implication does not.
The two rows that trouble everyone are the last two: P → Q is true whenever P is false. This is not a philosophical claim about causation; it is a definition chosen so that "for all n, if n > 100 then n > 10" comes out true rather than being invalidated by n = 5. An implication only ever makes a promise about the cases where its hypothesis holds. When the hypothesis fails, the promise was never triggered, so it was not broken — the statement is said to be vacuously true.
This has an immediate programming consequence. A loop body that never executes satisfies any postcondition you like; a filter over an empty list satisfies every predicate. all([]) is True in Python and Stream.allMatch on an empty stream is true in Java for exactly this reason, and it is not a quirk — it is the truth table.
Equivalences you should be able to apply without thinking
Two formulas are logically equivalent (written ≡) when they have identical truth tables. The core set:
De Morgan is the one you will use daily. Inverting a compound condition is not a matter of adding a ! at the front and hoping: !(a && b) is !a || !b, and getting this wrong while "simplifying" an if is one of the most common bug sources in refactoring.
Contraposition (P → Q ≡ ¬Q → ¬P) is a genuine equivalence. The converse (Q → P) and the inverse (¬P → ¬Q) are not. "If it is a square then it is a rectangle" is true; its converse is false. Almost every fallacious argument you will meet is a converse being smuggled in as an equivalence.
Predicates and quantifiers
A predicate P(x) is a proposition with holes. Quantifiers fill them: ∀x P(x) asserts P holds for every x in the domain, ∃x P(x) asserts it holds for at least one. The domain is never optional — ∀x (x² ≥ x) is true over the integers and false over the reals (take x = ½).
Negation pushes through quantifiers by flipping them, which is De Morgan generalised to infinite conjunctions and disjunctions:
Reading that in engineering terms: the negation of "every request completes in under 200 ms" is not "every request takes at least 200 ms" — it is "some request takes at least 200 ms". One counterexample refutes a universal claim; refuting an existential claim requires exhausting the domain. That asymmetry is exactly why testing can disprove a specification but not prove it.
Quantifier order is not commutative
This is the single most valuable thing in the chapter. ∀x ∃y and ∃y ∀x say entirely different things.
In the first, y may depend on x; in the second it may not. Recognising this distinction is what separates "every user has a session" from "there is a session shared by every user", and it is the formal content of the difference between a per-request lock and a global lock. Same-type quantifiers do commute (∀x∀y ≡ ∀y∀x); mixed ones never do.
Definitions in analysis and algorithms are quantifier sandwiches. Big-O is ∃c ∃n₀ ∀n ≥ n₀; a limit is ∀ε ∃δ ∀x; linearizability is ∀ histories ∃ a serial order. If you can read the quantifier prefix, you can reconstruct the definition; if you cannot, you are memorising.
Normal forms and satisfiability
Every formula can be rewritten in conjunctive normal form (an AND of ORs of literals) or disjunctive normal form (an OR of ANDs). CNF is what SAT solvers eat; DNF is what a truth table's true-rows read off as directly. The conversion is mechanical — eliminate → and ↔, push negations inward with De Morgan, then distribute — but distribution can blow up exponentially, which is why the Tseytin transformation (introducing fresh variables for subformulas) is used in practice to get an equisatisfiable CNF of linear size.
A formula is satisfiable if some assignment makes it true, a tautology if every assignment does, and unsatisfiable if none does. These are linked: φ is a tautology exactly when ¬φ is unsatisfiable, which is why a single SAT solver answers both questions and why validity checking is co-NP-complete while satisfiability is NP-complete.
# Column for variable i is a 2^n-bit integer whose bit r says
# what variable i is in row r. All 2^n rows evaluate at once.
def columns(n):
full = (1 << (1 << n)) - 1 # all rows true
cols = []
for i in range(n):
c, block = 0, 1 << i # run length of the pattern
for r in range(1 << n):
if (r >> i) & 1: c |= 1 << r
cols.append(c)
return cols, full
# NOT p -> full ^ p p AND q -> p & q
# p OR q -> p | q p -> q -> (full ^ p) | q
# tautology <=> result == full satisfiable <=> result != 0That trick — one machine word holding a whole truth-table column — is worth remembering beyond logic. It is the same idea as a bitset DP, and it makes brute-force checking of formulas over n ≤ 5 variables free.
- Theory
Build the truth table of (P → Q) ∧ (Q → R) → (P → R) and classify it as tautology, contingency or contradiction.
Solution
It is a tautology (hypothetical syllogism). Rather than eight rows, argue by contradiction on the outer implication: it can only be false if the antecedent (P→Q)∧(Q→R) is true and P→R is false. The latter forces P = T, R = F. Then P→Q with P = T forces Q = T, and Q→R with Q = T forces R = T — contradicting R = F. No falsifying row exists.
- Theory
Show P → (Q → R) ≡ (P ∧ Q) → R without a truth table.
Solution
Apply material implication twice and associate:
P → (Q → R) ≡ ¬P ∨ (¬Q ∨ R) ≡ (¬P ∨ ¬Q) ∨ R ≡ ¬(P ∧ Q) ∨ R ≡ (P ∧ Q) → R.
This equivalence is currying: a function of two arguments and a function returning a function are the same thing, and the logical statement is the Curry–Howard image of that fact.
- Theory
Negate and simplify so that no negation stands outside a quantifier or a parenthesis: ∀x ∃y (P(x,y) ∧ ¬Q(y)).
Solution
¬∀x∃y(P ∧ ¬Q) ≡ ∃x ∀y ¬(P(x,y) ∧ ¬Q(y)) ≡ ∃x ∀y (¬P(x,y) ∨ Q(y)) ≡ ∃x ∀y (P(x,y) → Q(y)).
Rule of thumb: walk left to right, flip every quantifier you pass, then apply De Morgan to the matrix.
- Theory
Over the domain of real numbers, decide the truth of each and justify: (a) ∀x∃y (x·y = 1), (b) ∃y∀x (x·y = 0), (c) ∀x∀y∃z (x < z ∧ z < y).
Solution
- (a) False: x = 0 has no multiplicative inverse. It becomes true over the non-zero reals — the domain carries the content.
- (b) True: take y = 0, which annihilates every x. Note this is an ∃∀ that genuinely holds, because a single witness works uniformly.
- (c) False as stated: it requires x < y, and fails for x = y or x > y. Restricted to pairs with x < y it is true (density of the reals) with z = (x+y)/2.
- Theory
State the converse, inverse and contrapositive of "if the cache is warm then the request is fast", and say which are equivalent to the original.
Solution
- Converse: if the request is fast then the cache is warm.
- Inverse: if the cache is not warm then the request is not fast.
- Contrapositive: if the request is not fast then the cache is not warm.
Only the contrapositive is equivalent to the original. Converse and inverse are equivalent to each other (each is the contrapositive of the other), but not to the original — a fast request could be fast because the payload was tiny.
- Theory
Convert (A ∧ B) ∨ (C ∧ D) to CNF and count the clauses. Generalise to n conjunctive blocks of size 2 and give the growth rate.
Solution
Distributing: (A∨C) ∧ (A∨D) ∧ (B∨C) ∧ (B∨D) — 4 clauses. With n blocks of size 2 you must pick one literal from each block, giving 2ⁿ clauses. This exponential blow-up is why solvers use the Tseytin encoding: name each subformula with a fresh variable and assert the definitional equivalences, producing an equisatisfiable CNF of size linear in the original formula.
- Theory
Prove that {¬, ∧} is functionally complete — every truth function can be written with those two alone. Then show NAND alone suffices.
Solution
Any truth function on n variables is the disjunction of one conjunctive term per true row (its DNF), so {¬, ∧, ∨} is complete. De Morgan removes ∨: P∨Q ≡ ¬(¬P ∧ ¬Q). Hence {¬,∧} is complete.
For NAND, write P ↑ Q = ¬(P∧Q). Then ¬P = P↑P and P∧Q = ¬(P↑Q) = (P↑Q)↑(P↑Q). Since {¬,∧} is complete, so is {↑} — which is why a chip fab can build every gate from one cell.
- Theory
An engineer replaces
if (!(a > 0 && b != null))withif (a <= 0 && b == null). Identify the fault and give the correct rewrite, including the case whereais a floating-point NaN.Solution
De Morgan turns a negated AND into an OR, not an AND. The correct rewrite is
if (a <= 0 || b == null). The buggy version only fires when both fail, silently skipping the case of a positiveawith a nullb.With floating point there is a second fault:
a > 0anda <= 0are both false whenais NaN, so ¬(a > 0) is not a ≤ 0. IEEE comparison is not a total order, so classical logic's law of trichotomy does not apply — negate by writing!(a > 0)literally. - Theory
Express "f is injective" and "f is surjective" as quantified formulas, then write the negation of each in positive form.
Solution
Injective: ∀x∀y (f(x) = f(y) → x = y). Negation: ∃x∃y (f(x) = f(y) ∧ x ≠ y) — a collision, which is exactly the object a hash-function attacker looks for.
Surjective: ∀b ∃a (f(a) = b). Negation: ∃b ∀a (f(a) ≠ b) — an unreachable value.
- Program
Write a function that takes a formula over variables
p,q,r,…(up to 20) as an expression tree and returnsTAUTOLOGY,SATISFIABLEorUNSATISFIABLEby exhaustive evaluation. State the complexity.Solution
Enumerate all 2ⁿ assignments as bitmasks and evaluate the tree in O(size) each: O(2ⁿ · |φ|) time, O(depth) space. Practical to n ≈ 20–24.
def classify(node, n): sat = taut = True any_true = False for mask in range(1 << n): v = ev(node, mask) any_true |= v taut &= v if taut: return "TAUTOLOGY" if any_true: return "SATISFIABLE" return "UNSATISFIABLE" def ev(t, m): k = t[0] if k == 'var': return (m >> t[1]) & 1 if k == 'not': return 1 - ev(t[1], m) a = ev(t[1], m); b = ev(t[2], m) if k == 'and': return a & b if k == 'or': return a | b if k == 'imp': return (1 - a) | b if k == 'iff': return int(a == b)Note the bitmask
mdoubles as the assignment: bit i is the value of variable i. Returning the first satisfyingmgives you a model, not just a yes. - Program
Two boolean expressions over the same n ≤ 16 variables are given as strings. Decide whether they are logically equivalent in O(2ⁿ) total using the bitmask-column technique from the chapter.
Solution
Evaluate each formula once, over all rows simultaneously, by making every variable a 2ⁿ-bit integer. The formulas are equivalent iff the two resulting integers are equal.
FULL = (1 << (1 << n)) - 1 col = [0]*n for r in range(1 << n): for i in range(n): if (r >> i) & 1: col[i] |= 1 << r def ev(t): k = t[0] if k == 'var': return col[t[1]] if k == 'not': return FULL ^ ev(t[1]) a, b = ev(t[1]), ev(t[2]) return {'and': a & b, 'or': a | b, 'imp': (FULL ^ a) | b, 'iff': FULL ^ (a ^ b)}[k] equivalent = (ev(f1) == ev(f2)) # differing rows, if you want a counterexample: diff = ev(f1) ^ ev(f2) if diff: row = (diff & -diff).bit_length() - 1The big-integer operations cost O(2ⁿ/w) machine words each, so this is roughly 64× faster than looping rows, and
diffhands you a concrete counterexample for free. - Program
Implement CNF conversion by the naive method (eliminate →, push negations, distribute) and by Tseytin. Compare output clause counts on (a₁∧b₁) ∨ (a₂∧b₂) ∨ … ∨ (aₙ∧bₙ) for n = 1..12.
Solution
Naive produces 2ⁿ clauses; Tseytin produces about 4n. The Tseytin idea: for each gate introduce a variable and assert its definition in CNF.
# t_i <-> a_i AND b_i becomes three clauses: # (¬t_i ∨ a_i) (¬t_i ∨ b_i) (t_i ∨ ¬a_i ∨ ¬b_i) # top <-> t_1 OR ... OR t_n becomes: # (¬t_i ∨ top) for each i, and (¬top ∨ t_1 ∨ ... ∨ t_n) # finally assert the unit clause (top)The two CNFs are not logically equivalent — Tseytin adds variables — but they are equisatisfiable, and any model of the Tseytin form restricted to the original variables is a model of the original. That is all a SAT solver needs, and it is why every real toolchain uses it.
- Program
Given a 2-CNF formula with n ≤ 10⁵ variables and m ≤ 10⁶ clauses, decide satisfiability in linear time and output an assignment. (This is 2-SAT; it is the one tractable island in an NP-complete sea.)
Solution
Each clause (x ∨ y) is two implications ¬x → y and ¬y → x. Build the implication graph on 2n literal-nodes, take strongly connected components with Tarjan or Kosaraju. It is unsatisfiable iff some variable and its negation land in the same component. Otherwise set x true when
comp[x] > comp[¬x]in reverse topological order.// node 2i = x_i, node 2i+1 = ¬x_i void addClause(int a, bool na, int b, bool nb){ add(2*a+na, 2*b+!nb); // ¬a → b add(2*b+nb, 2*a+!na); // ¬b → a } // after SCC (Tarjan gives reverse topological order in comp[]): for (int i = 0; i < n; i++){ if (comp[2*i] == comp[2*i+1]) return UNSAT; val[i] = comp[2*i] < comp[2*i+1]; // Tarjan numbering }Correctness rests on the symmetry of the implication graph: if u → v is an edge then so is ¬v → ¬u — contraposition, appearing as a graph automorphism.
- Program
Model this as logic and solve programmatically: in a 9×9 Sudoku, encode "every cell has exactly one value" in CNF. How many clauses does the exactly-one constraint need, and how would you reduce it?
Solution
Variables xₕ,ₖ,ₔ = "cell (r,c) holds value v", 729 in total. At least one: one clause (xₕₖ₁ ∨ … ∨ xₕₖ₉) per cell. At most one: pairwise negation, C(9,2) = 36 clauses per cell, so 81 × 37 = 2,997 clauses just for cells, before rows, columns and boxes.
The pairwise encoding is O(k²). For large k use a sequential (ladder) encoding with auxiliary variables sᵢ meaning "one of the first i is true", giving O(k) clauses:
(¬x_i ∨ s_i) (¬s_{i-1} ∨ s_i) (¬x_i ∨ ¬s_{i-1})At k = 9 pairwise is fine; at k = 1000 it is the difference between running and not.
- Program
Write a checker that, given a claimed logical equivalence between two conditions extracted from source code (say
!(a && b) || cand!a || !b || c), reports whether a refactor is safe — and explain why the answer changes if the operands can have side effects or short-circuit.Solution
Treating
a,b,cas opaque booleans, evaluate both over all 8 assignments; here both columns agree, so the rewrite is sound in classical logic.It is not necessarily sound as a program transformation, for two reasons an interviewer will want you to name:
- Short-circuit order. In
!(a && b),bis evaluated only whenais true. In!a || !b,bis evaluated only whenais true as well — here they match, but reordering the disjuncts to!b || !achanges which expression runs, and ifbislist.get(0)guarded byabeing a non-empty check, the rewrite throws. - Side effects and non-boolean semantics. If evaluating
bmutates state or blocks, duplicating or dropping an evaluation changes behaviour even though the truth tables agree. Three-valued logics (SQL'sNULL) break the equivalences outright: in SQL,NOT (a AND b)and(NOT a) OR (NOT b)do agree, but the law of excluded middlea OR NOT ais not a tautology whenaisNULL.
The general lesson: logical equivalence licenses a refactor only under the assumption of pure, total, two-valued operands. State the assumption, then use the theorem.
- Short-circuit order. In
Direct proof, contraposition, contradiction and cases
Four proof shapes cover the overwhelming majority of arguments in computer science. Knowing which shape a statement wants is most of the work; the algebra afterwards is usually routine.
A proof is a chain of implications from things already accepted to the claim, where each link is either a definition, a previously proved result, or a rule of inference. It is not a computation and not a verification on examples — checking a claim for n = 1..1000 is evidence, not proof, and the history of number theory is littered with conjectures that first fail past 10¹⁰.
What makes proof a practical engineering skill rather than an academic ritual is that the shape of the statement determines the shape of the proof. Learn the mapping and you always know how to start.
| Statement shape | How you start |
|---|---|
| ∀x P(x) | "Let x be arbitrary." Prove P(x) using nothing specific about x. |
| ∃x P(x) | Construct a witness, or prove one must exist non-constructively. |
| P → Q | Assume P, derive Q. Or assume ¬Q, derive ¬P. |
| P ↔ Q | Two proofs, or a chain of equivalences where every step is reversible. |
| ¬P | Assume P, derive a contradiction. |
| A = B for sets | A ⊆ B and B ⊆ A. |
| Uniqueness | Assume two objects work, prove they are equal. |
Direct proof
Assume the hypothesis, unfold the definitions, do algebra, arrive at the conclusion. The step that beginners skip is unfolding the definitions — almost every stuck direct proof is stuck because a word like "divides" or "even" has not been replaced by what it means.
Claim: if a | b and b | c then a | c. Unfold: b = a·k and c = b·m for integers k, m. Substitute: c = a·(km). Since km is an integer, a | c. The whole proof was the substitution; the definition did the work.
Contraposition
Because P → Q ≡ ¬Q → ¬P, you may prove either. Choose whichever hypothesis is more usable. "If n² is even then n is even" is awkward directly — you would have to reason backwards from a square. Contrapositively: "if n is odd then n² is odd", and now you have n = 2k+1 in hand and n² = 4k²+4k+1 = 2(2k²+2k)+1 falls out.
If the hypothesis is a negative or an existential you cannot get hold of, and the conclusion is a positive statement with structure, try the contrapositive. You are trading which side gives you an object to manipulate.
Contradiction
Assume the claim is false, derive something impossible. The classic is the irrationality of √2: assume √2 = p/q in lowest terms, square to get p² = 2q², deduce p is even, write p = 2r, get 2r² = q² so q is even too — contradicting "lowest terms".
Contradiction is the natural tool for impossibility results, which is most of what a computer scientist proves: no comparison sort beats n log n, no algorithm decides halting, no consensus protocol is safe and live under an asynchronous network with one crash failure. Each assumes the thing exists and extracts an absurdity.
A caution worth carrying: contradiction is often used where contraposition would be cleaner. If your "contradiction" proof assumes P ∧ ¬Q and then never uses P, you actually wrote a contrapositive proof with extra steps.
Proof by cases and WLOG
Split the domain into finitely many exhaustive cases and prove each. The obligations are that the cases cover everything and that you actually do all of them. "Without loss of generality" is a legitimate shortcut only when a symmetry genuinely maps one case to the other — if you say WLOG a ≤ b, the statement must be symmetric in a and b, and you should be able to say why.
Claim: for reals a,b: |a + b| ≤ |a| + |b| (triangle inequality)
Case 1: a,b ≥ 0 |a+b| = a+b = |a|+|b| equality
Case 2: a,b ≤ 0 |a+b| = -(a+b) = |a|+|b| equality
Case 3: opposite signs, WLOG a ≥ 0 > b
|a+b| ≤ max(a,-b) ≤ a + (-b) = |a|+|b| strict unless one is 0
WLOG is justified here because swapping a and b leaves both sides unchanged.Existence, uniqueness, and constructive versus not
To prove ∃x P(x) you may exhibit an x (constructive) or show that no-x leads to contradiction (non-constructive). The distinction is not pedantry for a programmer: a constructive proof is an algorithm. The extended Euclidean algorithm is a constructive proof of Bézout's identity; the pigeonhole principle proves collisions exist without telling you where.
Uniqueness has its own template: suppose x₁ and x₂ both satisfy the property, then show x₁ = x₂. Proving "exactly one" is two separate obligations, and forgetting the second is the most common gap in student proofs.
"Prove your algorithm is correct" is a proof-shape question in disguise. Greedy correctness is an exchange argument (a proof by contradiction: take an optimal solution differing from greedy at the earliest point, exchange, show it is no worse). Dynamic programming correctness is induction on subproblem size. Naming the shape before you start is what makes the answer sound rehearsed rather than improvised.
- Theory
Prove that the sum of a rational and an irrational number is irrational.
Solution
By contradiction. Let q be rational, i irrational, and suppose q + i = r is rational. Then i = r - q. The rationals are closed under subtraction (a/b - c/d = (ad-cb)/bd, and bd ≠ 0), so i is rational — contradiction. Note that the analogous claim for two irrationals is false: √2 + (-√2) = 0.
- Theory
Prove √3 is irrational, and explain precisely where the same argument fails for √4.
Solution
Suppose √3 = p/q in lowest terms, so p² = 3q². Then 3 | p². Because 3 is prime, Euclid's lemma gives 3 | p; write p = 3r, so 9r² = 3q², i.e. q² = 3r², so 3 | q — contradicting lowest terms.
For √4 the step "4 | p² ⇒ 4 | p" is false (p = 2 is a counterexample), because 4 is not prime. The primality of the radicand is doing all the work, which is why the general theorem is: √n is irrational unless n is a perfect square.
- Theory
Prove there are infinitely many primes. Then adapt the argument to show there are infinitely many primes of the form 4k+3.
Solution
Euclid. Suppose p₁,…,pₖ were all of them. Let N = p₁p₂…pₖ + 1. N > 1 so it has a prime factor p; but p cannot be any pᵢ, since each leaves remainder 1. Contradiction. (Note the standard misreading: N itself need not be prime — 2·3·5·7·11·13+1 = 30031 = 59·509.)
Form 4k+3. Every odd prime is 4k+1 or 4k+3, and a product of numbers all ≡ 1 (mod 4) is ≡ 1. Suppose q₁,…,qₖ are all the 4k+3 primes; put N = 4q₁…qₖ - 1 ≡ 3 (mod 4). N is odd, and if all its prime factors were ≡ 1 (mod 4) then N ≡ 1, false. So some prime factor is ≡ 3 (mod 4), and it is not any qᵢ (each would have to divide 1). Contradiction.
- Theory
Prove: if n is an integer and n³ + 5 is odd, then n is even. Which technique is natural and why?
Solution
Contraposition: assume n is odd and show n³+5 is even. If n = 2k+1 then n³ is odd (product of odds), so n³+5 = odd + odd = even. Direct proof would require reasoning from the parity of a cube back to the parity of n, which is the same work done backwards; contraposition hands you the algebraic object (2k+1) immediately.
- Theory
Prove that among any two consecutive integers exactly one is even, using a proof by cases on n mod 2. Then state what could go wrong if you omitted a case.
Solution
By the division algorithm, n = 2q or n = 2q+1, exhaustively. Case A: n even, then n+1 = 2q+1 odd. Case B: n odd, then n+1 = 2q+2 = 2(q+1) even. In both cases exactly one of the pair is even.
Exhaustiveness is the hidden obligation: the proof is valid only because the division algorithm guarantees the two cases cover ℤ. Omitting a case yields a proof of a statement restricted to a subdomain — which is precisely the bug pattern where an
if/else ifchain lacks a finalelseand silently does nothing on the uncovered input. - Theory
Show that for any 5 points placed in a 2×2 square, some two are at distance at most √2. Is your proof constructive?
Solution
Cut the square into four 1×1 quadrants. Five points into four quadrants means some quadrant holds two (pigeonhole), and the diameter of a unit square is √2. So those two are within √2.
It is non-constructive in the pigeonhole step: it proves a crowded quadrant exists without naming it. It becomes constructive the moment you actually bucket the points, which is exactly what the grid method for the closest-pair problem does — the proof turns into an O(n) algorithm.
- Theory
Prove that if a₁,…,aₙ are positive reals with product 1, then their sum is at least n. Identify the proof technique.
Solution
This is AM–GM: (a₁+…+aₙ)/n ≥ (a₁…aₙ)^(1/n) = 1. A self-contained argument is induction on n with a case split: for n = 1 trivial; for the step, since the product is 1 some aᵢ ≥ 1 and some a₃ ≤ 1. Replace the pair by their product aᵢa₃, reducing to n-1 terms; the inequality (aᵢ-1)(1-a₃) ≥ 0 rearranges to aᵢ + a₃ ≥ aᵢa₃ + 1, so the sum did not increase when we merged. Apply the induction hypothesis.
The technique is induction plus a smoothing/exchange step — the same shape as a greedy exchange argument.
- Theory
A greedy algorithm schedules jobs by earliest finishing time to maximise the count of non-overlapping intervals. Write the correctness proof as a formal exchange argument.
Solution
Let G = g₁,g₂,… be greedy's choices in order and O = o₁,o₂,… an optimal solution sorted by finish time, chosen among all optima to agree with G on the longest prefix. Suppose |O| > |G| and let k be the first index where they differ. Greedy picked the compatible interval with the smallest finish time, so finish(gₖ) ≤ finish(oₖ). Replace oₖ by gₖ in O: everything after oₖ starts at or after finish(oₖ) ≥ finish(gₖ), so the result is still feasible and the same size — hence still optimal, but it agrees with G on a longer prefix, contradicting the choice of O.
Therefore no such k exists, G is a prefix of an optimum, and since greedy stops only when nothing compatible remains, |G| = |O|. Note the structure: contradiction on the outside, construction on the inside.
- Program
Write a program that searches for a counterexample to "n² + n + 41 is prime for all natural n". Find the smallest one and explain what the experiment does and does not establish.
Solution
The smallest counterexample is n = 40: 40²+40+41 = 1681 = 41². (n = 41 also fails, visibly, since every term is divisible by 41.)
def is_prime(x): if x < 2: return False d = 2 while d*d <= x: if x % d == 0: return False d += 1 return True n = 0 while is_prime(n*n + n + 41): n += 1 print(n) # 40The search disproves a universal claim with one witness — that is logically complete. Running it to n = 10⁶ and finding nothing would not prove the claim: verifying ∀ over an infinite domain by enumeration is impossible. This asymmetry (tests refute, proofs establish) is the whole reason correctness arguments exist alongside test suites.
- Program
Implement a checker for the claim "every even integer 4 ≤ n ≤ 10⁶ is the sum of two primes" (Goldbach) in near-linear time overall, and report the pair with the largest smaller-prime.
Solution
Sieve once to 10⁶, then for each even n scan primes upward until the complement is prime. Empirically the first prime works almost immediately, so the total cost is close to O(N log log N) for the sieve plus a small constant per n.
N = 10**6 comp = bytearray(N+1) for i in range(2, int(N**0.5)+1): if not comp[i]: comp[i*i::i] = bytearray(len(comp[i*i::i])) for j in range(i*i, N+1, i): comp[j] = 1 primes = [i for i in range(2, N+1) if not comp[i]] worst = (0, 0, 0) for n in range(4, N+1, 2): for p in primes: if p > n // 2: break if not comp[n-p]: if p > worst[0]: worst = (p, n-p, n) break print(worst)Again: exhaustive verification to 10⁶ is strong evidence and zero proof. Goldbach remains open.
- Program
Given a set of intervals, implement the earliest-finish-time greedy and, separately, an O(n²) DP that computes the true optimum. Randomly generate 10⁵ instances and assert the two agree — a computational cross-check of the exchange argument.
Solution
def greedy(iv): iv = sorted(iv, key=lambda t: t[1]) end, k = float('-inf'), 0 for s, f in iv: if s >= end: end, k = f, k + 1 return k def dp(iv): # exact, O(n^2) iv = sorted(iv, key=lambda t: t[1]) n = len(iv); best = [1]*n for i in range(n): for j in range(i): if iv[j][1] <= iv[i][0]: best[i] = max(best[i], best[j] + 1) return max(best) if n else 0 import random for _ in range(100000): iv = [tuple(sorted(random.sample(range(20), 2))) for _ in range(random.randint(0,8))] assert greedy(iv) == dp(iv), ivThis is stress testing, the competitive-programming discipline that corresponds to proof-checking: a brute force you trust, a fast solution you do not, and random small inputs. It finds counterexamples fast when the greedy is wrong (try sorting by start time instead and watch it fail within a few hundred iterations).
- Program
Write a program to find the smallest positive integer n such that n is expressible as a sum of two positive cubes in two different ways, and describe why brute force needs a bound argument to terminate.
Solution
The answer is 1729 = 1³+12³ = 9³+10³, the Hardy–Ramanujan number.
from collections import defaultdict LIM = 100 # a^3 with a <= LIM seen = defaultdict(list) for a in range(1, LIM+1): for b in range(a, LIM+1): seen[a*a*a + b*b*b].append((a,b)) ans = min(k for k,v in seen.items() if len(v) > 1 and k <= LIM**3) print(ans) # 1729The subtlety: a search over a,b ≤ LIM only certifies minimality up to LIM³, because a smaller n could in principle use a cube larger than LIM³ — except it cannot, since each cube is at most n. So the correct bound is a,b ≤ n^(1/3), and restricting the reported answer to k ≤ LIM³ is what makes the brute force a proof rather than a guess. Stating such a bound is the difference between "I searched" and "I searched exhaustively".
- Program
Implement a verifier for uniqueness: given a function
solve(instance)and a brute-force enumerator of all valid answers, check on random instances that the answer is unique, and explain how non-uniqueness changes what a judge can accept.Solution
for _ in range(TRIALS): inst = random_instance() all_valid = [a for a in enumerate_candidates(inst) if valid(inst, a)] assert all_valid, "no solution but solve() returned one" if len(all_valid) > 1: print("NOT UNIQUE:", inst, all_valid[:3]) assert solve(inst) in all_validThe last assertion is the point. When answers are not unique, comparing your output byte-for-byte with a reference is wrong — the checker must verify the property (feasibility plus optimal value), which is exactly why contest judges use special checkers for "print any valid answer" problems. Formally, you have separated proving ∃x P(x) from proving ∃!x P(x), and only the second licenses exact-match judging.
- Program
Prove computationally and then mathematically: the equation x² ≡ 2 (mod 5) has no solution. Generalise your program to report, for each prime p < 100, which residues are squares.
Solution
Computationally, square every residue mod 5: 0,1,4,4,1 — the set of squares is {0,1,4}, and 2 is absent. That is a complete proof, because a solution would have to be one of the 5 residue classes; exhaustive search over a finite domain is a proof, unlike the infinite case in problem 9.
for p in primes_below(100): qr = sorted({(x*x) % p for x in range(p)}) print(p, len(qr), qr[:10]) # for odd p you always get exactly (p+1)/2 values, including 0Mathematically: the non-zero squares mod an odd prime p form the image of x ↦ x², which is 2-to-1 (since x and -x collide and nothing else does), so there are exactly (p-1)/2 non-zero quadratic residues. For p = 5 that is 2 residues, {1,4}, and 2 is a non-residue. Chapter 24 turns this counting fact into Euler's criterion.
Induction, strong induction and well-ordering
Induction is the proof rule that mirrors recursion, and strong induction is the one that mirrors divide and conquer. If you can write the recursion you can write the proof, because they are the same object read in opposite directions.
The principle of mathematical induction says: if P(0) holds, and P(n) → P(n+1) for every n, then P(n) holds for all n ≥ 0. It is not a trick — it is one of the Peano axioms, the axiom that pins down what "the natural numbers" means and excludes stray elements not reachable by counting.
Every induction proof has three named parts and a discipline about each:
If the step never uses the hypothesis you have not written an induction, you have written a direct proof with decoration. If the base case is wrong you have proved nothing, however impeccable the step — the classic joke proof that all horses are the same colour has a flawless step and a broken n = 2 case.
A worked example, and where the step really happens
Claim: 1 + 2 + … + n = n(n+1)/2.
Notice the shape: you write the k+1 quantity, split off the last term so the hypothesis applies to the rest, substitute, and simplify to the target form. That splitting move is 90% of routine inductions.
Strong induction
Sometimes P(k) alone is not enough and you need all of P(b), …, P(k). That is strong induction, and it is logically equivalent to ordinary induction (apply ordinary induction to Q(n) = P(b) ∧ … ∧ P(n)), but far more convenient whenever the recursion splits the problem non-uniformly.
Every integer n ≥ 2 has a prime factorisation: if n is prime, done; otherwise n = ab with 1 < a,b < n, and the hypothesis applies to a and b — which are not n-1, so ordinary induction cannot reach them. Mergesort's correctness has the same shape: the recursive calls are on ⌊n/2⌋ and ⌈n/2⌉.
Match the induction to the recursion. Loop → ordinary induction on the iteration count. Divide and conquer → strong induction on size. Tree recursion → structural induction on the tree. Mutual recursion → simultaneous induction on a pair of statements. When a proof feels impossible, it is usually because the induction is not shaped like the algorithm.
Well-ordering, and why termination is a proof obligation
The well-ordering principle — every non-empty set of non-negative integers has a least element — is equivalent to induction, and it is the standard way to prove termination. Find a quantity (a variant or monovariant) that is a non-negative integer and strictly decreases every iteration. It cannot decrease forever, so the loop stops.
while b != 0:
a, b = b, a % b
return a
VARIANT b is a non-negative integer and strictly decreases:
the new b is a mod b < b whenever b > 0.
By well-ordering the loop cannot run forever.
INVARIANT gcd(a, b) is unchanged by the body, because
gcd(a, b) = gcd(b, a mod b).
True before the loop; preserved by each iteration.
ON EXIT b = 0, so the invariant reads gcd(a₀, b₀) = gcd(a, 0) = a.
Hence the returned a is the gcd. ∎That template — invariant for correctness, variant for termination — proves every loop you will ever write, and it is Floyd–Hoare logic in miniature. A while (true) with a break deep inside is hard to reason about precisely because the variant is hidden.
Strengthening the hypothesis
A counter-intuitive but essential technique: when an induction will not close, prove something stronger. A stronger claim gives you a stronger hypothesis in the step, and the extra assumption is often exactly what was missing.
Try proving ∑ 1/i² ≤ 2 by induction and it stalls — the hypothesis Sₖ ≤ 2 gives Sₖ₊₁ ≤ 2 + 1/(k+1)², which overshoots. Prove instead Sₖ ≤ 2 - 1/k. Now Sₖ₊₁ ≤ 2 - 1/k + 1/(k+1)² ≤ 2 - 1/(k+1), because 1/(k+1)² ≤ 1/k - 1/(k+1) = 1/(k(k+1)). The stronger statement proves itself while the weaker one does not.
"Why is your DP correct?" wants an induction on subproblem order: state the invariant that dp[i] equals the true optimum for the subproblem, show it for the base indices, and show the transition preserves it given that all indices it reads are smaller in the DP order. Two sentences, and it is the difference between "I've seen this pattern" and "I know why it works".
- Theory
Prove by induction that ∑ᵢ₋₁ⁿ i² = n(n+1)(2n+1)/6.
Solution
Base n = 1: 1 = 1·2·3/6 = 1. ✓
Step: assume Sₖ = k(k+1)(2k+1)/6. Then
Sₖ₊₁ = k(k+1)(2k+1)/6 + (k+1)² = (k+1)[k(2k+1) + 6(k+1)]/6 = (k+1)(2k²+7k+6)/6 = (k+1)(k+2)(2k+3)/6,
which is the formula at n = k+1. ∎ The factoring of 2k²+7k+6 into (k+2)(2k+3) is the only non-mechanical step; pulling (k+1) out first is what makes it appear.
- Theory
Prove 2ⁿ > n² for all n ≥ 5. Why does the base case have to be 5?
Solution
Base n = 5: 32 > 25. ✓ (At n = 2,3,4 the claim is false or equal: 4 = 4, 8 < 9, 16 = 16 — so 5 is genuinely the first valid base.)
Step: assume 2ᵏ > k² for some k ≥ 5. Then 2ᵏ⁺¹ = 2·2ᵏ > 2k². It suffices that 2k² ≥ (k+1)², i.e. k² - 2k - 1 ≥ 0, which holds for k ≥ 1+√2 ≈ 2.41, so certainly for k ≥ 5. ∎
The step is valid from k = 3 onward, but the chain has to start somewhere true — this is the standard trap where a correct step is anchored to a false base.
- Theory
Prove that any 2ⁿ × 2ⁿ board with one square removed can be tiled by L-shaped trominoes.
Solution
Base n = 1: a 2×2 board minus one square is an L-tromino. ✓
Step: given a 2ᵏ⁺¹ × 2ᵏ⁺¹ board with one square missing, cut it into four 2ᵏ×2ᵏ quadrants. The missing square lies in one of them. Place a single tromino at the centre covering one cell of each of the other three quadrants. Now every quadrant is a 2ᵏ×2ᵏ board with exactly one cell unavailable, so the hypothesis tiles each. ∎
This proof is an algorithm: recursing gives an O(4ⁿ) constructive tiling, optimal since that is the output size.
- Theory
Prove by strong induction that every integer n ≥ 12 can be written as 4a + 5b with a,b ≥ 0.
Solution
Bases n = 12 = 4·3, 13 = 4·2+5, 14 = 4+5·2, 15 = 5·3. ✓ (Four bases, because the step reaches back by 4.)
Step: for n ≥ 16, n-4 ≥ 12, so by the strong hypothesis n-4 = 4a+5b; then n = 4(a+1)+5b. ∎
The count of base cases must equal the reach-back of the step. This is the Chicken McNugget / Frobenius setting: for coprime p,q the largest unrepresentable value is pq-p-q, here 20-4-5 = 11, which is exactly why 12 is the threshold.
- Theory
Prove that the Fibonacci numbers satisfy F₁+F₂+…+Fₙ = Fₙ₊₂ - 1, and that gcd(Fₔ, Fₙ) = F₋ᶜᵔ₃ₓᵢ₋ₙ₋ is plausible by checking small cases.
Solution
Base n = 1: F₁ = 1 and F₃ - 1 = 2-1 = 1. ✓
Step: ∑ᵢ₋₁ᵏ⁺¹ Fᵢ = (Fₖ₊₂ - 1) + Fₖ₊₁ = Fₖ₊₃ - 1 by the Fibonacci recurrence. ∎
For the gcd identity: gcd(F₁₂,F₁₈) = gcd(144,1597) = 1 = F₆? No — gcd(12,18) = 6 and F₆ = 8, and indeed gcd(144, 2584) = 8 once you index consistently. The lesson is to fix an indexing convention (F₁=F₂=1) before testing identities; off-by-one in Fibonacci indexing invalidates more "counterexamples" than any actual error.
- Theory
Find the flaw: "Claim: all horses have the same colour. Base: one horse, trivially. Step: given k+1 horses, the first k share a colour and the last k share a colour, so all k+1 do."
Solution
The step fails at k+1 = 2. The argument needs the two groups of size k to overlap in at least one horse to chain the colours; with k = 1 the sets {h₁} and {h₂} are disjoint and nothing links them. So P(1) → P(2) is not established, and the chain breaks at its first link even though every later link is fine.
The general moral: a step that is valid "for large k" must still be valid at the specific k where the chain starts. This is the same failure mode as an off-by-one in a loop that only manifests on inputs of size 1.
- Theory
State a loop invariant for insertion sort and use it to prove correctness.
Solution
Invariant: at the start of the iteration with index i, the subarray
A[0..i-1]contains the original elements ofA[0..i-1], in sorted order.- Initialisation: at i = 1 the subarray is a single element, trivially sorted and a permutation of itself.
- Maintenance: the body shifts every element of
A[0..i-1]greater thanA[i]one position right and insertsA[i]into the gap. The result is sorted (the inserted key is ≥ everything to its left and < everything to its right) and is a permutation of the old contents plusA[i]. So the invariant holds at i+1. - Termination: the loop ends at i = n, where the invariant states that
A[0..n-1]is a sorted permutation of the input. ∎
This is induction on i, dressed in the vocabulary CLRS uses; the "maintenance" clause is the inductive step.
- Theory
Prove that a binary tree with n internal nodes (each with exactly 2 children) has n+1 leaves, by structural induction.
Solution
Structural induction on the tree. Base: a single leaf has n = 0 internal nodes and 1 leaf; 0+1 = 1. ✓
Step: a non-leaf tree is a root with subtrees T₁, T₂ having n₁, n₂ internal nodes and, by hypothesis, n₁+1, n₂+1 leaves. The whole tree has n₁+n₂+1 internal nodes and (n₁+1)+(n₂+1) = (n₁+n₂+1)+1 leaves. ∎
This is why a Huffman tree with k symbols has exactly k-1 merges, and why a segment tree over n leaves needs < 2n nodes — the same counting identity twice.
- Theory
Prove that strong induction and the well-ordering principle are equivalent.
Solution
WO → SI. Suppose the strong-induction premises hold but P(n) fails for some n. Let S = {n : ¬P(n)}, non-empty, so by well-ordering it has a least element m. Every j < m satisfies P(j), so the strong-induction step yields P(m) — contradicting m ∈ S. Hence S is empty.
SI → WO. Let S be a set of naturals with no least element; we show S is empty by strong induction on P(n) = "n ∉ S". Assume 0,…,n-1 ∉ S. If n ∈ S it would be the least element of S (everything smaller is excluded), contradiction; so n ∉ S. By strong induction no n is in S. ∎
This is why "take a minimal counterexample" and "induct" are interchangeable proof openings.
- Program
Implement Towers of Hanoi and prove by induction that your solution uses exactly 2ⁿ - 1 moves; verify computationally for n ≤ 20.
Solution
def hanoi(n, a, b, c, out): if n == 0: return hanoi(n-1, a, c, b, out) out.append((a, c)) hanoi(n-1, b, a, c, out) # note the source/aux swap for n in range(21): out = [] hanoi(n, 'A', 'B', 'C', out) assert len(out) == (1 << n) - 1Proof. Let M(n) be the move count. M(0) = 0 = 2⁰-1. The code gives M(n) = 2M(n-1) + 1, so by induction M(n) = 2(2ⁿ⁻¹-1)+1 = 2ⁿ-1. ∎
Optimality is a separate induction: the largest disc must move at least once, and immediately before it does, all n-1 smaller discs must be stacked on the spare peg — costing at least M(n-1) before and M(n-1) after, giving M(n) ≥ 2M(n-1)+1.
- Program
Write mergesort and annotate it with the invariant and variant that prove it correct and terminating. Then write a stress test against the language's built-in sort.
Solution
def msort(a): # VARIANT: len(a) strictly decreases on every recursive call # and is a non-negative integer -> recursion terminates. if len(a) <= 1: return a[:] # BASE of the strong induction m = len(a) // 2 L, R = msort(a[:m]), msort(a[m:]) # HYPOTHESIS: both are sorted out, i, j = [], 0, 0 # INVARIANT: out is sorted, and every element of out is <= every # remaining element of L[i:] and R[j:]. while i < len(L) and j < len(R): if L[i] <= R[j]: out.append(L[i]); i += 1 else: out.append(R[j]); j += 1 out += L[i:]; out += R[j:] return out # STEP: hence out is sorted import random for _ in range(20000): a = [random.randrange(-9, 10) for _ in range(random.randrange(0, 12))] assert msort(a) == sorted(a)Note that the induction is strong: the calls are on ⌊n/2⌋ and ⌈n/2⌉, not n-1. Note also that
<=rather than<in the comparison is what makes the sort stable — an invariant about equal elements that the correctness proof above does not capture, and therefore has to be stated separately. - Program
Given n ≤ 10⁵, count the ways to tile a 2×n strip with 1×2 dominoes. Derive the recurrence, prove it by induction, and compute the answer mod 10⁹+7.
Solution
Let T(n) be the count. Consider the leftmost column: either one vertical domino covers it (leaving T(n-1)) or two horizontal dominoes cover the first two columns (leaving T(n-2)). These cases are exhaustive and disjoint, so T(n) = T(n-1) + T(n-2), T(0)=1, T(1)=1 — the Fibonacci numbers.
MOD = 10**9 + 7 a, b = 1, 1 # T(0), T(1) for _ in range(n - 1): a, b = b, (a + b) % MOD print(b if n >= 1 else 1)The induction that makes this a proof rather than a pattern-match is the claim "every tiling of the 2×n strip has a unique leftmost-column configuration from those two", which needs the observation that a horizontal domino in row 1 forces one in row 2 (nothing else can cover the cell below it without leaving a hole). For n up to 10¹⁸ use matrix exponentiation from Chapter 48.
- Program
Implement a function that, given a recurrence aₙ = c₁aₙ₋₁ + … + cₖaₙ₋ₖ and initial terms, verifies a claimed closed form for n = 0..10⁵ — a computational check of an inductive claim.
Solution
def verify(c, init, closed, N=10**5, mod=None): a = list(init) k = len(c) for n in range(N + 1): if n >= k: v = sum(c[i] * a[n-1-i] for i in range(k)) if mod: v %= mod a.append(v) if a[n] != closed(n): return ("MISMATCH", n, a[n], closed(n)) return "OK" # e.g. a_n = a_{n-1} + a_{n-2}, a_0=0,a_1=1 vs Binet rounded print(verify([1,1], [0,1], lambda n: round(((1+5**.5)/2)**n / 5**.5), N=70))Two lessons this experiment teaches directly. First, Binet's formula fails past n ≈ 70 in double precision — a closed form correct in ℝ is not correct in
float64, which is Chapter 66's subject. Second, agreement on 10⁵ terms is not proof; the actual proof is a two-line induction verifying that the closed form satisfies both the recurrence and the initial conditions, and that these two facts determine the sequence uniquely. - Program
Prove and implement: the number of binary strings of length n with no two consecutive 1s is Fₙ₊₂. Then generate all such strings for n = 20 without generating and filtering all 2²⁰.
Solution
Proof. Let S(n) be the count. Strings ending in 0 are any valid string of length n-1 with 0 appended: S(n-1). Strings ending in 1 must have 0 before it, so they are a valid string of length n-2 with "01" appended: S(n-2). Hence S(n) = S(n-1)+S(n-2), with S(0)=1, S(1)=2, giving S(n) = Fₙ₊₂ under F₁=F₂=1. ∎
def gen(n): out = [] def rec(pos, last, cur): if pos == n: out.append(''.join(cur)); return cur.append('0'); rec(pos+1, 0, cur); cur.pop() if last == 0: cur.append('1'); rec(pos+1, 1, cur); cur.pop() rec(0, 0, []) return out # |gen(20)| == 17711 == F_22, and the recursion visits only valid prefixesThe point of the generator is that pruning at the branch (
if last == 0) means the search tree has exactly O(S(n)) leaves rather than 2ⁿ — the proof and the pruning are the same fact, which is the general relationship between counting arguments and backtracking efficiency. - Program
Write a general "variant checker" harness: instrument a loop with a claimed variant expression and assert on every iteration that it is a non-negative integer and strictly decreased. Apply it to binary search and to the Collatz iteration; explain the different outcomes.
Solution
def run_with_variant(step, state, variant, limit=10**6): prev = variant(state); assert prev >= 0 for _ in range(limit): if done(state): return "terminated", state state = step(state) v = variant(state) assert 0 <= v < prev, f"variant failed: {prev} -> {v}" prev = v return "no proof of termination within limit", stateBinary search: the variant
hi - lois a non-negative integer that at least halves each iteration, so it strictly decreases while positive — the harness passes, and this is the termination proof (plus the ⌈log₂n⌉ bound, since halving a value in [0,n] reaches 0 in that many steps).Collatz: no known variant exists. The value itself increases on odd inputs (3n+1), and no monotone measure has ever been found. The harness reports "no proof" forever, which is the honest state of mathematics: Collatz terminates for every integer anyone has tested past 2⁶⁸, and remains unproved. The contrast is the punchline of the chapter — termination is not observable, it is provable or it is open.
Sets, relations and functions
Sets are the substrate everything else is built on, relations are what tables and graphs actually are, and functions are relations with a determinism constraint. Getting the vocabulary exact removes an entire class of modelling mistakes.
A set is an unordered collection of distinct objects. That definition contains two engineering facts: no duplicates and no order. A HashSet honours both; a List honours neither. Choosing the wrong one is choosing the wrong mathematical object, and the bugs that follow (duplicate emails, order-dependent test failures) are the model leaking.
The standard operations and their identities:
Those are the same laws as Chapter 1 because a set is a predicate in disguise: x ∈ A ∩ B unfolds to x ∈ A ∧ x ∈ B. Any set identity can be proved by unfolding to logic, applying a propositional equivalence, and folding back. That is the standard technique and it never fails.
Proving set equality
To show A = B you show A ⊆ B and B ⊆ A. Each direction starts "let x ∈ A be arbitrary" and ends "hence x ∈ B". When both directions are the same chain read backwards, you may write a single chain of ↔ — but only if every step is genuinely reversible, which is a real obligation people skip.
Relations
A binary relation from A to B is simply a subset of A × B. That is the entire definition, and it is worth sitting with: a database table is a relation (hence "relational"), an adjacency matrix is a relation, ≤ is a relation, and so is "follows on social media". The properties a relation on a single set may have:
| Property | Definition | Example / non-example |
|---|---|---|
| Reflexive | ∀a: aRa | ≤ yes; < no |
| Irreflexive | ∀a: ¬aRa | < yes; "is a sibling of" yes |
| Symmetric | aRb → bRa | "same connected component" yes; "parent of" no |
| Antisymmetric | aRb ∧ bRa → a=b | ≤ yes; "same age" no |
| Transitive | aRb ∧ bRc → aRc | ≤ yes; "friend of" famously no |
| Total | ∀a,b: aRb ∨ bRa | ≤ on ℤ yes; ⊆ no |
Symmetric and antisymmetric are not opposites: equality is both, "divides" on positive integers is antisymmetric only, and a relation can be neither.
A comparator you pass to a sort must define a strict weak ordering: irreflexive, transitive, and with transitive incomparability. Violate it — return true for both cmp(a,b) and cmp(b,a), or compare with <= where < is required — and C++'s std::sort has undefined behaviour that reads out of bounds, while Java throws "Comparison method violates its general contract". The exception is a relation-theory violation reported at runtime.
Closures
The transitive closure R⁺ is the smallest transitive relation containing R — reachability, if R is an edge set. Reflexive-transitive closure R* adds the diagonal. Computing it is Floyd–Warshall on booleans in O(n³), or O(n³/64) with bitsets, and knowing that "reachability = transitive closure" lets you move between graph and relational phrasings of the same problem freely.
Functions
A function f : A → B is a relation where every a ∈ A relates to exactly one b. "At most one" makes it a partial function; "at least one" makes it total. A Map is a partial function; a total function on a finite domain is an array.
Composition (g ∘ f)(x) = g(f(x)) is associative but not commutative, and it preserves injectivity and surjectivity in the directions you would expect. The formal inverse f⁻¹ exists exactly when f is a bijection — which is precisely why a hash function has no inverse (not injective) and why an encryption function must be one (a bijection on the message space for a fixed key).
"Design a URL shortener" is a question about functions. You need an injective map from long URLs to short codes (else two URLs collide and one breaks), and you must decide whether it is a computed function (hash, needing collision handling because it is not injective) or a stored relation (a counter plus a table, injective by construction). Saying that out loud reframes the whole design in one sentence.
- Theory
Prove A ∖ (B ∪ C) = (A ∖ B) ∩ (A ∖ C) by the unfold-to-logic method.
Solution
x ∈ A∖(B∪C) ↔ x∈A ∧ ¬(x∈B ∨ x∈C) ↔ x∈A ∧ (¬x∈B ∧ ¬x∈C) (De Morgan) ↔ (x∈A ∧ ¬x∈B) ∧ (x∈A ∧ ¬x∈C) (idempotence of ∧ on x∈A) ↔ x ∈ (A∖B) ∩ (A∖C). Every step is a biconditional, so both inclusions are proved at once. ∎
- Theory
Show |A ∪ B| = |A| + |B| - |A ∩ B| and extend to three sets. What is the pattern for n sets?
Solution
Each element of A∪B is counted once on the right if it lies in exactly one of A, B, and twice-minus-once = once if it lies in both. For three sets, elements in all three are counted 3 - 3 + 1 = 1 time:
|A∪B∪C| = |A|+|B|+|C| - |A∩B| - |A∩C| - |B∩C| + |A∩B∩C|.
The general pattern is inclusion–exclusion: alternate signs over all non-empty subsets of the index set, |∪Aᵢ| = ∑∅≠S (-1)^(|S|+1) |∩i∈S Aᵢ|, proved in Chapter 30. The binomial identity that makes it work is ∑k=1..m (-1)^(k+1) C(m,k) = 1 for every m ≥ 1.
- Theory
Classify each relation on ℤ as reflexive / symmetric / antisymmetric / transitive: (a) a ≡ b (mod 5), (b) a | b, (c) a ≠ b, (d) a · b > 0.
Solution
- (a) reflexive, symmetric, transitive — an equivalence relation. Not antisymmetric (2 ≡ 7 and 7 ≡ 2 but 2 ≠ 7).
- (b) reflexive, transitive, antisymmetric on positive integers (a partial order); on all of ℤ antisymmetry fails since 2 | -2 and -2 | 2. The domain matters.
- (c) symmetric only. Not reflexive, not transitive (1≠2, 2≠1, but 1=1).
- (d) symmetric and transitive, but not reflexive because 0·0 = 0. A relation that is symmetric and transitive but not reflexive is the standard trap: the "proof" that aRb → bRa → aRa silently assumes some b exists related to a.
- Theory
How many binary relations are there on an n-element set? How many are reflexive? Symmetric? Functions?
Solution
- All relations: subsets of n² pairs, so 2^(n²).
- Reflexive: the n diagonal pairs are forced, the other n²-n are free: 2^(n²-n).
- Symmetric: choose freely on the diagonal (n) and on unordered off-diagonal pairs (C(n,2)): 2^(n(n+1)/2).
- Functions A→A: each of n inputs picks one of n outputs: nⁿ. Of these, n! are bijections.
The gap between 2^(n²) and nⁿ is the price of determinism, and it is why a lookup table is exponentially cheaper to represent than an arbitrary relation.
- Theory
Prove that if g ∘ f is injective then f is injective, and give a counterexample showing g need not be.
Solution
Suppose f(x) = f(y). Applying g, g(f(x)) = g(f(y)), and injectivity of the composite gives x = y. So f is injective. ∎
Counterexample for g: let f : {1} → {1,2} with f(1)=1, and g : {1,2} → {1} constant. Then g∘f is injective on a one-element domain, but g is not injective. Dually, if g∘f is surjective then g is, but f need not be.
- Theory
Show |P(A)| = 2^|A| by a bijection with binary strings, and explain the connection to bitmask enumeration.
Solution
Fix an ordering a₀,…,aₙ₋₁ of A. Map each subset S to the length-n string whose i-th bit is 1 iff aᵢ ∈ S. This is injective (different subsets differ on some element, hence on some bit) and surjective (every string names the subset of positions where it is 1), so it is a bijection, and there are 2ⁿ strings. ∎
That bijection is subset enumeration by bitmask:
for (int m = 0; m < (1<<n); m++)iterates over P(A) exactly once each, andm & (1<<i)tests membership in O(1). Enumerating submasks of m is the same bijection restricted:for (int s = m; s; s = (s-1) & m). - Theory
Define the image and preimage of a set under f. Prove f⁻¹(X ∩ Y) = f⁻¹(X) ∩ f⁻¹(Y) but that images only satisfy f(X ∩ Y) ⊆ f(X) ∩ f(Y).
Solution
f(X) = {f(x) : x ∈ X} and f⁻¹(Y) = {x : f(x) ∈ Y} (defined for any f, not just bijections).
Preimage: x ∈ f⁻¹(X∩Y) ↔ f(x) ∈ X ∧ f(x) ∈ Y ↔ x ∈ f⁻¹(X) ∧ x ∈ f⁻¹(Y). ∎ Preimages commute with all of union, intersection and complement — which is exactly why filtering by a predicate on the input composes cleanly.
Image: ⊆ is clear. For strictness take f(x)=x², X = {1}, Y = {-1}. Then X∩Y = ∅ so the left side is empty, while f(X)∩f(Y) = {1}. Equality holds for all X,Y precisely when f is injective.
- Program
Implement set operations over sorted integer arrays in O(n+m) and over bitsets in O(n/64). Benchmark both for n = 10⁶.
Solution
// sorted two-pointer intersection vector<int> inter(const vector<int>& a, const vector<int>& b){ vector<int> r; size_t i = 0, j = 0; while (i < a.size() && j < b.size()){ if (a[i] < b[j]) i++; else if (b[j] < a[i]) j++; else { r.push_back(a[i]); i++; j++; } } return r; } // bitset: universe of size N, membership by bit bitset<1000000> A, B; auto I = A & B; // intersection ~N/64 word ops auto U = A | B; // union auto D = A & ~B; // difference auto S = A ^ B; // symmetric differenceThe bitset wins by a factor near 64 when the universe is dense and small enough to fit in memory; the sorted-array version wins when the sets are sparse relative to the universe (a 10⁶-bit set costs 125 KB whether it holds 3 elements or 10⁶). The crossover is roughly when |A|+|B| > N/64 — a genuinely useful rule when choosing a representation.
- Program
Given a directed graph with n ≤ 1000 nodes, compute the transitive closure. Give both the O(n³) Floyd–Warshall version and a bitset version, and state the practical speed-up.
Solution
// bitset transitive closure: O(n^3 / 64) bitset<1000> reach[1000]; for (int i = 0; i < n; i++) reach[i][i] = 1; // reflexive-transitive for (int k = 0; k < n; k++) for (int i = 0; i < n; i++) if (reach[i][k]) reach[i] |= reach[k];The inner loop over j collapses into one
|=over 1000 bits — about 16 machine words. At n = 1000 that is 10⁹/64 ≈ 1.6·10⁷ word operations, well under a second, where the scalar triple loop is 10⁹ and takes seconds. Theif (reach[i][k])guard is not an optimisation detail: it is the observation that if i cannot reach k, routing through k adds nothing. - Program
Write a comparator-contract validator: given a comparator and a small array, check irreflexivity, antisymmetry and transitivity by brute force over all triples, and report the violating triple.
Solution
def check(cmp_lt, xs): for a in xs: if cmp_lt(a, a): return ("not irreflexive", a) for a in xs: for b in xs: if cmp_lt(a,b) and cmp_lt(b,a): return ("not antisymmetric", a, b) for a in xs: for b in xs: for c in xs: if cmp_lt(a,b) and cmp_lt(b,c) and not cmp_lt(a,c): return ("not transitive", a, b, c) # transitivity of incomparability (strict weak ordering): eq = lambda x,y: not cmp_lt(x,y) and not cmp_lt(y,x) if eq(a,b) and eq(b,c) and not eq(a,c): return ("incomparability not transitive", a, b, c) return "ok"Run it on the classic broken comparator
lambda a,b: a.score <= b.scoreand it reports "not antisymmetric" on any two equal scores. Run it on a comparator that ranks bya.scorebut treats a missing score as 0 in one branch and as infinity in another, and it reports an intransitive triple — the exact input that makesstd::sortread out of bounds. This O(k³) check on a 20-element sample is cheap insurance in a test suite. - Program
Given n ≤ 20 items, enumerate every subset and, for each, every one of its submasks. Prove the total work is 3ⁿ rather than 4ⁿ.
Solution
for (int m = 0; m < (1 << n); m++) for (int s = m; ; s = (s - 1) & m){ process(m, s); if (s == 0) break; }Count. The pair (m, s) with s ⊆ m is determined by assigning each of the n elements one of three states: in s (hence in m), in m but not s, or in neither. Three independent choices per element gives exactly 3ⁿ pairs. ∎
At n = 20 that is 3.5·10⁹ — too slow; at n = 16 it is 4.3·10⁷ and fine. This is the workhorse loop for subset-sum DP over subsets (SOS DP is the O(2ⁿ n) refinement when you only need sums, not all pairs).
- Program
Implement a bijection between the set of all finite subsets of ℕ and ℕ itself, in both directions, and verify round-tripping for the first 10⁵ naturals.
Solution
The bijection is the binary-representation one: subset S maps to ∑i∈S 2ᵢ.
def encode(S): return sum(1 << i for i in S) def decode(n): return {i for i in range(n.bit_length()) if (n >> i) & 1} assert all(encode(decode(n)) == n for n in range(10**5))It is a bijection because binary representation is unique. The consequence is important for Chapter 6: the set of finite subsets of ℕ is countable, while P(ℕ) — all subsets, including infinite ones — is not. The finiteness restriction is exactly what makes the encoding a finite integer.
- Program
Model a "user has role" relation and answer: which users have all of a required role set? Implement with bitmasks and state the complexity against a nested-loop approach.
Solution
// roles[u] is a bitmask of u's roles; need is the required mask vector<int> who(const vector<int>& roles, int need){ vector<int> r; for (int u = 0; u < (int)roles.size(); u++) if ((roles[u] & need) == need) r.push_back(u); // superset test return r; }O(n) with a one-instruction subset test, versus O(n·k) scanning a list of roles per user. The mathematical content is that "need ⊆ roles[u]" is
(roles[u] & need) == need— subset as a bit operation. Note the direction:(a & b) == bmeans b ⊆ a, and reversing it is the single most common bug in permission code. - Program
Given two large files of records, compute the symmetric difference by key without loading both fully into memory. What set-theoretic property makes a streaming solution possible?
Solution
Sort both files by key externally, then do a single merge pass emitting keys present in exactly one stream:
i = j = 0 while i < len(A) or j < len(B): if j == len(B) or (i < len(A) and A[i] < B[j]): emit(A[i], 'only-A'); i += 1 elif i == len(A) or B[j] < A[i]: emit(B[j], 'only-B'); j += 1 else: i += 1; j += 1 # present in both: not in the symmetric differenceThe property that licenses streaming is that ▵ is decomposable over an ordering: whether a key belongs to A ▵ B depends only on that key's membership in each set, not on any other key. So a merge that sees each key's full multiplicity in one place suffices, and memory is O(1) beyond the sort. Union, intersection and difference are decomposable too; something like "the 10 most common keys in A but not B" is not, and needs a second pass or a sketch.
Equivalence relations, partial orders and lattices
Equivalence relations partition; partial orders arrange. Union-find is the first, topological sort is the second, and a surprising number of algorithm design questions are really asking which of the two structures your data has.
Two special kinds of relation carry most of the weight in computer science, and they are near-mirror images. An equivalence relation is reflexive, symmetric and transitive. A partial order is reflexive, antisymmetric and transitive. Swapping symmetry for antisymmetry is the whole difference between "these things are interchangeable" and "these things are ranked".
Equivalence relations and partitions
The central theorem: equivalence relations on A correspond exactly to partitions of A. Given ~, the equivalence class [a] = {x : x ~ a}, and the classes are non-empty, cover A, and are pairwise disjoint. Conversely any partition defines an equivalence by "in the same block".
The disjointness proof is the one to know: if [a] ∩ [b] ≠ ∅, take c in both, so c~a and c~b; symmetry and transitivity give a~b, and then [a] = [b]. Classes are never partially overlapping — they are equal or disjoint, and there is no third case.
Notice the last row: every group-by is an equivalence relation, and the key function is a witness. Two items are equivalent iff key(x) = key(y), and relations of that form ("kernel of a function") are automatically equivalences. This gives a fast test: if you can produce a canonical form, you have an equivalence, and you can group in O(n) with a hash map instead of comparing pairs in O(n²).
To group by an equivalence, find a canonical form. Anagrams → sorted letters. Congruent triangles → sorted side lengths. Isomorphic rooted trees → a canonical string encoding of the sorted children encodings. Whenever grouping looks quadratic, the missing piece is a canonical form.
Union-find: an equivalence relation you can grow
Disjoint-set union maintains an equivalence relation under incremental merges, with O(α(n)) amortised operations — inverse Ackermann, effectively constant.
int p[N], sz[N];
void init(int n){ for (int i = 0; i < n; i++) p[i] = i, sz[i] = 1; }
int find(int x){ return p[x] == x ? x : p[x] = find(p[x]); } // compress
bool unite(int a, int b){
a = find(a); b = find(b);
if (a == b) return false; // already equivalent
if (sz[a] < sz[b]) swap(a, b); // attach smaller to larger
p[b] = a; sz[a] += sz[b];
return true;
}The three relation axioms map onto the code: reflexivity is p[i] = i, symmetry is the fact that unite is order-insensitive, and transitivity is the tree structure — sharing a root is transitive by construction.
Partial orders
A poset (P, ≤) is reflexive, antisymmetric, transitive. The word partial means some pairs are simply incomparable: under ⊆, the sets {1} and {2} are neither. A total order additionally compares everything.
Posets are drawn as Hasse diagrams: the transitive reduction, edges only for covering relations, everything implied by transitivity left out. A DAG is exactly a poset's Hasse diagram plus orientation, and this is why:
Every finite poset has at least one linear extension (that is what topological sort computes), and counting them is #P-complete — which is why "how many valid orderings" problems are hard while "give me one" is linear time.
Lattices, and why they matter for static analysis
A lattice is a poset where every pair has a least upper bound (join, ∨) and a greatest lower bound (meet, ∧). Subsets under ⊆ form a lattice with union and intersection; divisors under divisibility form one with lcm and gcd.
Lattices are the backbone of dataflow analysis. The compiler's abstract state lives in a finite-height lattice, transfer functions are monotone, and the fixed-point iteration terminates because a monotone function on a finite-height lattice reaches a fixed point in bounded steps (Knaster–Tarski). Type inference joins types the same way; the "common supertype" of a ternary expression is a join in the subtyping lattice.
"Given task dependencies, find the minimum time with unlimited workers" is a poset question: the answer is the longest chain, computable by DP over a topological order. "Maximum tasks in parallel" is the largest antichain, which by Dilworth's theorem equals the minimum number of chains covering the poset — solvable as bipartite matching. Recognising which of the two you were asked is the whole problem.
- Theory
Prove that equivalence classes are either identical or disjoint.
Solution
Suppose c ∈ [a] ∩ [b], so c~a and c~b. By symmetry a~c, and with transitivity a~b. Now take any x ∈ [a]: x~a~b gives x ∈ [b], so [a] ⊆ [b]; the symmetric argument gives the reverse. Hence [a] = [b]. Therefore if the classes meet at all, they coincide. ∎
- Theory
How many equivalence relations are there on a set of size 4? Of size n?
Solution
Equivalence relations correspond to partitions, so the count is the Bell number Bₙ. B₀..B₅ = 1, 1, 2, 5, 15, 52, so the answer for n = 4 is 15. The recurrence is Bₙ₊₁ = ∑ₖ C(n,k) Bₖ — choose the k elements sharing a block with a distinguished element, partition the rest. Chapter 33 develops this.
Contrast with the 2^(n²) relations overall: the three axioms are enormously restrictive.
- Theory
Is "|a - b| ≤ 1" on the integers an equivalence relation? Justify precisely.
Solution
No. It is reflexive and symmetric but not transitive: |1-2| ≤ 1 and |2-3| ≤ 1 but |1-3| = 2. This is the classic "approximately equal" trap, and it has a direct engineering consequence: floating-point comparison by epsilon (
abs(a-b) < eps) is not an equivalence relation, so you cannot use it as a hash-map key or a sort key without producing inconsistent results. The standard fix is snapping to a grid (round(x/eps)), which is an equivalence because it is the kernel of a function. - Theory
Draw the Hasse diagram of the divisors of 36 under divisibility. Identify the join and meet, the maximal and minimal elements, and the longest chain.
Solution
Divisors: 1, 2, 3, 4, 6, 9, 12, 18, 36. Covering edges: 1→2, 1→3; 2→4, 2→6; 3→6, 3→9; 4→12; 6→12, 6→18; 9→18; 12→36; 18→36.
Join = lcm, meet = gcd. Unique minimum 1, unique maximum 36 (so it is a bounded lattice). The longest chain is 1 | 2 | 4 | 12 | 36, length 5 — equal to Ω(36)+1 = 4+1 where Ω counts prime factors with multiplicity, since each covering step multiplies by exactly one prime.
Structurally this poset is {0,1,2} × {0,1} ordered componentwise (exponents of 2 and 3) — a product of chains, which is the general shape of any divisor lattice.
- Theory
Prove that every finite non-empty poset has a maximal element, and explain why "maximal" and "maximum" differ.
Solution
Take any x₀. If it is not maximal, some x₁ > x₀ exists; repeat. Antisymmetry and transitivity make the sequence strictly increasing, so no element repeats, and finiteness forces the process to stop — at a maximal element. ∎ (For infinite posets this needs Zorn's lemma and can fail without it.)
Maximal means nothing is strictly above it; maximum means it is above everything. In the poset {a, b} with no relations, both a and b are maximal and there is no maximum. In scheduling terms: several tasks can be "last possible" without one being "the last".
- Theory
State Dilworth's theorem and use it to solve: given n tasks with a dependency partial order, what is the minimum number of workers each of whom executes a chain of dependent tasks?
Solution
Dilworth: in a finite poset, the minimum number of chains needed to cover all elements equals the maximum size of an antichain.
So the answer is the size of the largest set of mutually independent tasks. Computationally: build the transitive closure, then a bipartite graph with left copy and right copy of each element, edge u_L - v_R when u < v. The minimum chain cover is n - (maximum matching) (this is minimum path cover on a DAG). By König's theorem the matching equals the minimum vertex cover, and Dilworth follows.
Complexity: O(n³/64) for the closure plus Hopcroft–Karp at O(E√V).
- Theory
Show that the set of all partitions of a set, ordered by refinement, forms a lattice. What are the meet and join?
Solution
Order: P ≤ Q iff every block of P is contained in a block of Q (P is finer). The meet P ∧ Q is the partition whose blocks are the non-empty intersections Bᴵ ∩ B_Q — the coarsest partition refining both. The join P ∨ Q is the transitive closure of the union of the two equivalence relations — the finest partition that both refine.
The join is exactly what union-find computes when you feed it the edges of both partitions, which is why merging two groupings is a linear-time operation rather than a search. The bottom element is all singletons; the top is one block.
- Program
Implement union-find with path compression and union by size, and use it to count connected components of a graph with 10⁶ nodes and 10⁶ edges. Then extend it to support "number of nodes in my component" and "number of edges in my component".
Solution
struct DSU { vector<int> p, sz, ecnt; DSU(int n): p(n), sz(n,1), ecnt(n,0) { iota(p.begin(), p.end(), 0); } int find(int x){ while (p[x] != x) x = p[x] = p[p[x]]; return x; } // halving void unite(int a, int b){ int ra = find(a), rb = find(b); if (ra == rb) { ecnt[ra]++; return; } // internal edge if (sz[ra] < sz[rb]) swap(ra, rb); p[rb] = ra; sz[ra] += sz[rb]; ecnt[ra] += ecnt[rb] + 1; } int compSize(int x){ return sz[find(x)]; } int compEdges(int x){ return ecnt[find(x)]; } };Any value aggregated at the root works as long as it is mergeable and you only ever query it at the root. A component is a tree iff
compEdges == compSize - 1, which is a one-line cycle detector. - Program
Group 10⁶ strings into anagram classes. Give an O(total length · logσ) solution and explain why comparing pairs is not viable.
Solution
from collections import defaultdict groups = defaultdict(list) for s in words: key = tuple(sorted(s)) # canonical form; or a 26-tuple of counts groups[key].append(s)Pairwise comparison is Θ(n²) = 10¹² comparisons — hopeless. The canonical form converts the equivalence test into an equivalence key, and hashing does the grouping in one pass. For lowercase words a 26-element count tuple is faster than sorting: O(|s| + 26) per word with no comparison sort.
The general principle from the chapter: an equivalence relation given by a canonical form is the kernel of a function, and kernels are groupable in linear time.
- Program
Implement topological sort (Kahn's algorithm), detect cycles, and additionally output the lexicographically smallest linear extension.
Solution
import heapq def toposort(n, adj): indeg = [0]*n for u in range(n): for v in adj[u]: indeg[v] += 1 h = [u for u in range(n) if indeg[u] == 0] heapq.heapify(h) # min-heap => lexicographically smallest order = [] while h: u = heapq.heappop(h); order.append(u) for v in adj[u]: indeg[v] -= 1 if indeg[v] == 0: heapq.heappush(h, v) return order if len(order) == n else None # None => a cycle existsO((V+E) log V) with the heap, O(V+E) with a plain queue if any extension will do. The cycle test is exact: Kahn's algorithm emits every vertex iff the relation is antisymmetric-and-acyclic, i.e. iff it really is a partial order. A graph with a cycle is not a poset, because a ≤ b ≤ a with a ≠ b violates antisymmetry.
- Program
Given a DAG of n ≤ 10⁵ tasks with durations, compute the minimum makespan with unlimited parallelism, and list one critical path.
Solution
order = toposort(n, adj) finish = [0]*n; pred = [-1]*n for u in order: finish[u] = max((finish[p] for p in rev[u]), default=0) + dur[u] pred[u] = max(rev[u], key=lambda p: finish[p], default=-1) makespan = max(finish) # reconstruct the critical path backwards v = max(range(n), key=lambda i: finish[i]); path = [] while v != -1: path.append(v); v = pred[v] path.reverse()O(V+E). Mathematically the makespan is the maximum weight of a chain in the poset, and the critical path is that chain. Every task off the critical path has slack; reducing any single non-critical task's duration cannot reduce the makespan, which is the scheduling fact this identity makes obvious.
- Program
Implement DFA minimisation by partition refinement (Hopcroft or Moore) — a direct application of computing the coarsest equivalence relation satisfying a stability condition.
Solution
# Moore's algorithm: O(n^2 * |Σ|), simple and adequate for n ≤ few thousand part = [1 if accepting[s] else 0 for s in range(n)] # initial 2 blocks while True: sig = [(part[s], tuple(part[delta[s][c]] for c in alphabet)) for s in range(n)] remap, new = {}, [] for s in range(n): if sig[s] not in remap: remap[sig[s]] = len(remap) new.append(remap[sig[s]]) if new == part: break part = newEach round splits blocks whose members behave differently on some symbol, so the partition only ever refines; since it cannot refine past singletons, it terminates in at most n rounds. The fixed point is the coarsest equivalence that is stable under the transition function — the Myhill–Nerode relation — and the quotient automaton is the unique minimal DFA. Hopcroft's refinement of this is O(n |Σ| log n) by always processing the smaller half of a split block.
- Program
Given n intervals, define a < b iff a ends before b starts. Find the longest chain and the largest antichain, and interpret both.
Solution
Under this order, a chain is a set of pairwise non-overlapping intervals, and an antichain is a set of pairwise overlapping intervals.
# longest chain = maximum non-overlapping set: greedy by end time, O(n log n) iv.sort(key=lambda t: t[1]); end = -inf; chain = 0 for s, f in iv: if s >= end: end, chain = f, chain + 1 # largest antichain = maximum simultaneous overlap: sweep line, O(n log n) ev = sorted([(s, +1) for s, f in iv] + [(f, -1) for s, f in iv]) cur = best = 0 for x, d in ev: cur += d; best = max(best, cur)Interpretation: the chain is the maximum number of non-conflicting meetings one room can host; the antichain is the minimum number of rooms needed. By Dilworth's dual (Mirsky's theorem) the poset can be covered by exactly that many antichains, which is precisely the greedy room-assignment algorithm. Interval orders are the case where both quantities are computable greedily instead of by matching.
- Program
Write a fixed-point iterator over a finite lattice for a reaching-definitions dataflow analysis, and argue termination.
Solution
IN = [set() for _ in blocks] OUT = [set() for _ in blocks] changed = True while changed: changed = False for b in blocks: newin = set().union(*(OUT[p] for p in preds[b])) if preds[b] else set() newout = (newin - kill[b]) | gen[b] if newin != IN[b] or newout != OUT[b]: IN[b], OUT[b], changed = newin, newout, TrueTermination. The lattice is P(D) for the finite set D of definitions, ordered by ⊆, of height |D|. The transfer function f(X) = (X ∖ kill) ∪ gen is monotone, and the join is union, so each variable's value only grows. A strictly increasing chain in a lattice of height |D| has at most |D|+1 elements, so across all |B| blocks the loop runs at most |B|·|D| times and the least fixed point is reached — Knaster–Tarski, made concrete. Replacing union with intersection (available expressions) flips the lattice and requires initialising to top rather than bottom; getting that initialisation backwards is the classic dataflow bug.
Cardinality, countability and the pigeonhole principle
The pigeonhole principle is the most useful trivial theorem in computer science, and Cantor's diagonal argument is the proof template behind undecidability. Both are statements about comparing sizes without counting.
Two sets have the same cardinality when a bijection exists between them. For finite sets this recovers ordinary counting. For infinite sets it produces results that look paradoxical until you accept the definition and stop appealing to intuition.
The pigeonhole principle
If n items go into m boxes and n > m, some box holds at least two items. The proof is one line by contraposition: if every box held at most one, the total would be at most m. The generalised form: some box holds at least ⌈n/m⌉ items — because if every box held at most ⌈n/m⌉ - 1, the total would be at most m(⌈n/m⌉-1) < n.
Trivial to state, and it proves an astonishing range of things:
Pigeonhole proofs are non-constructive: they say a collision exists without saying where. The engineering consequence is that they give lower bounds and impossibility results for free, but never algorithms. When a problem says "show that two of these must…", the answer is almost always pigeonhole, and the work is choosing the right boxes.
Choosing the boxes is the entire skill. For "among n+1 numbers from 1..2n, one divides another": write each number as 2ᵃ · q with q odd. There are only n odd values in 1..2n, so two numbers share the same q — and then the one with the smaller exponent divides the other. The boxes were the odd parts, and finding that is the proof.
Countable sets
A set is countable if it is finite or has a bijection with ℕ. Surprises follow immediately:
- ℤ is countable: 0,1,-1,2,-2,… — zigzag enumeration.
- ℕ × ℕ is countable: enumerate along anti-diagonals. The explicit bijection is the Cantor pairing function π(x,y) = (x+y)(x+y+1)/2 + y.
- ℚ is countable: each rational is a pair of integers, and a countable union of countable sets is countable.
- The set of all finite strings over a finite alphabet is countable: enumerate by length, then lexicographically.
- Therefore the set of all programs is countable, since a program is a finite string.
Cantor's diagonal argument
The reals are not countable. Suppose they were, and list the reals in [0,1) as r₁, r₂, r₃, … with decimal expansions. Build a new number d whose k-th digit differs from the k-th digit of rₖ (say, 5 unless that digit is 5, else 6 — avoiding the 0.999… ambiguity). Then d differs from every rₖ in position k, so d is not in the list, contradicting completeness of the list. ∎
This is the single most reused proof template in theoretical computer science. Since programs are countable and functions ℕ → {0,1} are not, most functions are not computable — a cardinality argument that gives an impossibility result before any machine model is defined. The halting problem's proof is the diagonal argument applied to a table of "does program i halt on input j", and Russell's paradox, Gödel's incompleteness and the Recursion theorem are all the same move.
Suppose H(p, x) decides whether program p halts on input x.
Build:
D(p): if H(p, p) then loop forever else halt
Now ask: does D(D) halt?
If D(D) halts, then H(D,D) was true, so D loops. contradiction
If D(D) loops, then H(D,D) was false, so D halts. contradiction
So no such H exists. The "diagonal" is the entry H(p, p).Cantor's theorem and the hierarchy
For any set A, |A| < |P(A)|. Proof: given any f : A → P(A), consider D = {a : a ∉ f(a)}. If D = f(b) for some b, then b ∈ D ↔ b ∉ f(b) = D — contradiction. So no f is surjective. ∎ The same diagonal, one level up, and it gives an infinite hierarchy of infinities.
"Can you write a program that detects infinite loops in any program?" is asking whether you know the halting problem. The strong answer names the diagonal construction in three lines, then adds the engineering reality: undecidable in general, but decidable for restricted languages (loop bounds, terminating type systems), and approximable in practice by timeouts and static analysers that are allowed to answer "don't know". Rice's theorem generalises: every non-trivial semantic property of programs is undecidable.
- Theory
Prove that among any 51 numbers chosen from 1..100, two are consecutive.
Solution
Box the integers into the 50 pairs {1,2}, {3,4}, …, {99,100}. With 51 chosen numbers and 50 boxes, some box contributes both its members — and those two are consecutive. ∎ The bound is tight: the 50 odd numbers contain no consecutive pair, so 51 is exactly the threshold.
- Theory
Prove that among any n+1 numbers chosen from 1..2n, one divides another.
Solution
Write each chosen number uniquely as 2ᵃq with q odd. Each q lies in {1,3,5,…,2n-1}, only n possible values. With n+1 numbers, two share the same odd part: 2ᵃq and 2ᵇq with a < b. Then the first divides the second. ∎
Tight: {n+1, n+2, …, 2n} has n elements and no such pair, since the ratio of any two is less than 2.
- Theory
Prove the Erdős–Szekeres theorem: any sequence of n²+1 distinct reals has a monotone subsequence of length n+1.
Solution
For each position i, let (aᵢ, dᵢ) be the lengths of the longest increasing and longest decreasing subsequences ending at i. If no monotone subsequence of length n+1 exists, then aᵢ, dᵢ ∈ {1,…,n}, giving only n² possible pairs for n²+1 positions. By pigeonhole two positions i < j share a pair. But comparing xᵢ and x₃: if xᵢ < x₃ then a₃ ≥ aᵢ+1; otherwise d₃ ≥ dᵢ+1. Either way the pairs differ — contradiction. ∎
Note the boxes were pairs of DP values, which is why this proof is essentially the correctness argument for the O(n log n) LIS algorithm read backwards.
- Theory
Prove that no lossless compression algorithm can shrink every input, and quantify how many inputs can shrink.
Solution
A lossless compressor is injective on the set of inputs. There are 2ⁿ strings of length exactly n, but only 2ⁿ - 1 strings of length less than n in total (1 + 2 + 4 + … + 2ⁿ⁻¹). By pigeonhole an injective map from 2ⁿ inputs into 2ⁿ-1 shorter outputs is impossible, so at least one length-n input does not shrink. ∎
Quantitatively: at most 2ⁿ⁻ᵏ - 1 inputs of length n can shrink by k or more bits, so the fraction shrinking by 10 bits is under 2⁻¹⁰ ≈ 0.1%. Compression works only because real data is concentrated on a tiny, highly non-uniform subset of the input space — Chapter 68 makes this precise with entropy.
- Theory
Prove ℚ is countable and ℝ is not, and state exactly which step of the diagonal argument fails if you try it on ℚ.
Solution
ℚ: each rational is p/q in lowest terms with q > 0, so it injects into ℤ × ℕ, which is countable by the Cantor pairing function; an infinite subset of a countable set is countable.
ℝ: Cantor's diagonal as in the chapter.
Applying the diagonal to a list of rationals does construct a real number differing from every listed rational — but that number need not be rational (its digit sequence is not eventually periodic, in general), so no contradiction arises. The diagonal argument only refutes a listing when the diagonal object is guaranteed to be in the set being listed, and that is the step that fails.
- Theory
Prove that the set of computable functions ℕ → ℕ is countable, and conclude that uncomputable functions exist.
Solution
Every computable function is computed by at least one program; every program is a finite string over a finite alphabet; the set of finite strings is countable (enumerate by length, then lexicographically). So the computable functions are the image of a countable set under a surjection, hence countable.
But the set of all functions ℕ → {0,1} has cardinality |P(ℕ)| > |ℕ| by Cantor's theorem. A countable set cannot cover an uncountable one, so uncomputable functions exist — in fact "almost all" functions are uncomputable. ∎
This is a pure cardinality proof: it names no specific uncomputable function and requires no machine model. Producing an explicit one (the halting predicate) needs the diagonal construction instead.
- Theory
You hash n keys into m buckets uniformly. Give the pigeonhole bound on the maximum load, and contrast it with the probabilistic bound.
Solution
Pigeonhole (worst case, deterministic): some bucket holds at least ⌈n/m⌉ keys — and an adversary who knows your hash function can force all n keys into one bucket, since the function is fixed and its preimages are infinite. This is exactly the hash-flooding DoS attack, and the mitigation is a keyed hash chosen at startup so the adversary cannot compute preimages.
Probabilistic (random hash, n = m): the maximum load is Θ(log n / log log n) with high probability — the balls-in-bins result of Chapter 40. With two hash functions and placement in the lesser-loaded bucket it drops to Θ(log log n) ("the power of two choices").
The contrast is the whole point: pigeonhole gives an unavoidable worst case, randomness gives a typical case, and neither substitutes for the other.
- Theory
Prove that in any group of 6 people, there are 3 mutual acquaintances or 3 mutual strangers (R(3,3) ≤ 6).
Solution
Fix a person v. Of the 5 others, by pigeonhole at least ⌈5/2⌉ = 3 share the same relation to v — say three are acquaintances of v (the stranger case is symmetric). If any two of those three know each other, they plus v form a mutual-acquaintance triangle. Otherwise all three are mutual strangers. ∎
Tightness: on 5 vertices, the 5-cycle and its complement (also a 5-cycle) contain no triangle, so R(3,3) = 6 exactly. In graph terms: every 2-colouring of K₆'s edges contains a monochromatic triangle.
- Program
Implement the Cantor pairing function and its inverse; verify it is a bijection ℕ² → ℕ for all pairs up to 500.
Solution
def pair(x, y): s = x + y return s * (s + 1) // 2 + y def unpair(z): import math w = (math.isqrt(8*z + 1) - 1) // 2 # largest w with w(w+1)/2 <= z t = w * (w + 1) // 2 y = z - t return (w - y, y) seen = set() for x in range(500): for y in range(500): z = pair(x, y) assert unpair(z) == (x, y) assert z not in seen; seen.add(z) assert min(seen) == 0 # and the image has no gaps below maxUse
math.isqrt, notint(sqrt())— the floating-point square root is wrong for z near 2⁵² and the bijection silently breaks. That is a Chapter 66 issue appearing inside a Chapter 6 construction, and it is exactly the kind of failure that survives a 500×500 test and dies in production. - Program
Given n ≤ 10⁵ integers, decide in O(n) whether two of them are congruent mod n-1, without a hash set. Justify with pigeonhole.
Solution
There are only n-1 residue classes mod n-1, and n numbers, so by pigeonhole a collision is guaranteed — the answer is always yes, and the work is only to exhibit the pair.
seen = [-1] * (n - 1) # direct-address array, not a hash set for i, v in enumerate(a): r = v % (n - 1) if seen[r] != -1: print(seen[r], i); break seen[r] = iO(n) time and memory with no hashing, because the key space is bounded by n-1 and can be direct-addressed. The pigeonhole guarantee also means the loop provably breaks before exhausting the array, so no "not found" branch is needed — a rare case where a counting argument removes a code path.
- Program
Implement Floyd's tortoise-and-hare cycle detection for a functional graph f : {0..n-1} → {0..n-1}, returning the cycle start and length in O(1) memory. Explain why pigeonhole guarantees a cycle.
Solution
Iterating f from any start produces an infinite sequence in an n-element state space; by pigeonhole some state repeats within the first n+1 steps, and once a state repeats the sequence is eventually periodic. So a cycle always exists — this is why the algorithm never needs a "no cycle" branch.
def floyd(f, x0): slow, fast = f(x0), f(f(x0)) while slow != fast: slow, fast = f(slow), f(f(fast)) mu, slow = 0, x0 # find cycle start while slow != fast: slow, fast, mu = f(slow), f(fast), mu + 1 lam, cur = 1, f(slow) # find cycle length while cur != slow: cur, lam = f(cur), lam + 1 return mu, lamWhy the meeting point works: if the tail has length μ and the cycle length λ, the pointers meet at a position ≡ 0 (mod λ) steps into the cycle, so restarting one pointer at x₀ and advancing both one step at a time makes them meet exactly at the cycle entrance. This is the basis of Pollard's rho in Chapter 23.
- Program
Empirically find the birthday-paradox threshold: for a 32-bit hash, how many random keys before a collision is more likely than not? Compare to the theoretical ≈ 1.177√m.
Solution
import random def trial(bits): m, seen, k = 1 << bits, set(), 0 while True: h = random.randrange(m); k += 1 if h in seen: return k seen.add(h) trials = sorted(trial(24) for _ in range(200)) # 24 bits keeps memory sane print("median first collision:", trials[100]) print("theory 1.177*sqrt(m) =", 1.177 * (1 << 12))For 24 bits the median lands near 4,800 versus the predicted 1.177·4096 ≈ 4,820. Extrapolating to 32 bits gives ≈ 77,000 keys for a coin-flip chance of collision — which is why a 32-bit hash is unusable as a unique identifier at any real scale, and why UUIDs carry 122 random bits (collision-safe past 10¹⁹ ids).
Note the contrast with pigeonhole: pigeonhole says a collision is certain at m+1 keys; the birthday bound says it is likely at √m. The square-root gap is the difference between a guarantee and a risk, and security proofs live in that gap.
- Program
Write a program that enumerates all finite binary strings in a well-defined order (a bijection with ℕ) and, given a string, returns its index. Use it to argue that the set of programs is countable.
Solution
def index(s): # "" -> 0, "0" -> 1, "1" -> 2, "00" -> 3, ... return (1 << len(s)) - 1 + int(s or '0', 2) * (len(s) > 0) def at(n): L = 0 while (1 << (L + 1)) - 1 <= n: L += 1 off = n - ((1 << L) - 1) return format(off, '0%db' % L) if L else '' assert all(index(at(n)) == n for n in range(10000))This is a explicit bijection ℕ → {0,1}*, so the set of finite binary strings is countable. A program is a finite string in some encoding, so the set of programs injects into {0,1}* and is countable. Combined with problem 6, that settles the existence of uncomputable functions — and the enumeration above is exactly the "list all programs" device used in proofs of the Recursion theorem and in Kolmogorov complexity arguments.
- Program
Implement a busy-beaver-style bounded halting checker: run each of the 2ᵏ programs of length k for at most T steps and report how many halt. Explain what this experiment cannot tell you.
Solution
halted = 0 for code in range(1 << k): if run_with_step_limit(decode(code), steps=T) is not TIMEOUT: halted += 1 print(halted, "of", 1 << k, "halted within", T, "steps")What it gives you: a lower bound on the number of halting programs of length k, and a sound over-approximation of the non-halting set. Bounded halting is decidable — that is the whole point of the step limit.
What it cannot tell you: whether any timed-out program halts eventually. Raising T never resolves this in general, because the maximum runtime of a halting k-bit program (the busy beaver function BB(k)) grows faster than any computable function, so no computable choice of T is ever sufficient. That is the halting problem in quantitative form: not merely unsolved, but with a provably non-computable resource requirement.
- Program
Given n points with integer coordinates in [0,10⁹]², prove and detect: among any 5, two have a midpoint with integer coordinates. Implement the detection in O(n).
Solution
Proof. The midpoint of (x₁,y₁) and (x₂,y₂) is integral iff x₁ ≡ x₂ and y₁ ≡ y₂ (mod 2). There are only 2×2 = 4 parity classes, so among 5 points two share a class. ∎
buckets = {} for i, (x, y) in enumerate(pts): key = (x & 1, y & 1) if key in buckets: print("pair:", buckets[key], i); break buckets[key] = iO(n), and by pigeonhole it terminates within the first 5 points regardless of n. In d dimensions the same argument needs 2ᵈ+1 points — the parity vector is the box, and the box count is what generalises.
Invariants, monovariants and the extremal principle
Three problem-solving tools that answer "is this reachable?", "does this terminate?" and "what does the best case look like?". They are the mathematical form of loop invariants, termination measures and greedy exchange arguments.
These three techniques are the working core of olympiad-style problem solving, and each has an exact counterpart in program reasoning. They are grouped here because in practice you try them in order: what stays the same, what steadily decreases, what does the extreme case look like.
Invariants: proving something is unreachable
An invariant is a quantity or property preserved by every legal move. If the start and the target differ in an invariant, the target is unreachable — and this is the only clean way to prove a negative about a process.
The classic: a mutilated 8×8 chessboard with two opposite corners removed cannot be tiled by dominoes. Colour it normally; the removed corners share a colour, so 30 squares of one colour and 32 of the other remain. Every domino covers exactly one of each. The difference (#black - #white) is invariant at 0 for any tiling, but the board has difference 2. Unreachable. ∎
Note what this proof does not do: it does not search. A brute force over placements would take exponential time to conclude the same thing, and would not generalise to 2n × 2n. The invariant is a proof of impossibility for every board size at once.
The 15-puzzle, in full
State: a permutation of 15 tiles + blank, on a 4×4 grid.
Move : slide a tile into the blank = transpose that tile with the blank.
Each move is a single transposition, so it flips the parity of the
permutation. Each move also changes the blank's row by 0 or ±1.
Define I = (number of inversions among the 15 tiles)
+ (row index of the blank, 0-based from the top)
A horizontal move: permutation unchanged in the "reading order" sense
→ inversions unchanged, blank row unchanged → I unchanged mod 2.
A vertical move: the blank jumps over 3 tiles → inversions change by
an odd amount (±1 or ±3); blank row changes by 1 → I unchanged mod 2.
So I mod 2 is INVARIANT. The solved state has I even.
Any start with I odd is provably unreachable — exactly half of them.Monovariants: proving a process ends
A monovariant changes in one direction only. If it is a non-negative integer that strictly decreases, well-ordering forces termination — this is the loop variant of Chapter 3, used on processes rather than code.
Example: n people sit in a circle; repeatedly pick two adjacent people who are out of order by height and swap them. Does this stop? Yes: each swap removes exactly one inversion, inversions are a non-negative integer, so at most C(n,2) swaps occur. That is simultaneously a termination proof and a tight bound on bubble sort.
Monovariants need not be integers, but they must not approach a limit without reaching it. "The total energy decreases" proves nothing if it decreases by 1/2ᵏ each step. Either bound the decrease from below by a positive constant, or use a well-founded ordering (lexicographic on a tuple of naturals is the standard trick for nested loops).
For nested or interleaved processes, use a lexicographic variant: a tuple of non-negative integers, ordered lexicographically, that strictly decreases. This proves termination of algorithms where no single counter decreases — Knuth–Bendix completion, the simplex method with Bland's rule, and the potential-function proofs of Chapter 12 all rely on it.
The extremal principle
Consider the largest, the smallest, the leftmost, the closest pair. In a finite non-empty set, an extremum exists (well-ordering again), and the extremal object often cannot behave the way the problem's negation requires.
Proof that a finite graph with all degrees ≥ 2 contains a cycle: take a longest path v₀ … vₖ. The endpoint vₖ has degree at least 2, so it has a neighbour besides vₖ₋₁. That neighbour cannot be outside the path (the path would be longer, contradicting maximality), so it is some vᵢ on the path — giving a cycle. ∎ Nothing was constructed; the extremal choice did the work.
Extremal arguments are how greedy proofs and exchange arguments are written: "consider an optimal solution agreeing with greedy on the longest prefix" is an extremal choice, and the contradiction comes from showing you could extend the agreement. If you can state that sentence, you can prove most greedy algorithms correct on the spot.
- Theory
A blackboard holds the numbers 1..20. Repeatedly erase two numbers a, b and write |a-b|. What is the parity of the final number, and why?
Solution
Invariant: the parity of the sum. Replacing a, b by |a-b| changes the sum by a+b-|a-b| = 2min(a,b), an even amount. So the sum's parity never changes.
Initially 1+2+…+20 = 210, which is even. The final single number therefore must be even. (Every even value from 0 to 20 is achievable, so the invariant is exactly the obstruction and nothing more.)
- Theory
Prove that a 8×8 board with two opposite corners removed cannot be tiled by 1×2 dominoes, and generalise: which pairs of removed squares can be tiled?
Solution
The colouring argument is in the chapter: opposite corners share a colour, leaving 30/32, while every domino covers one of each.
Generalisation (Gomory's theorem): removing any two squares of opposite colours always leaves a tileable board. Proof: draw a closed Hamiltonian cycle through all 64 cells (easy to construct on any even board). Removing two cells of opposite colour splits the cycle into two paths, each of even length — because along the cycle colours alternate, so the two removed cells are an odd distance apart in both directions. An even path is trivially tiled by consecutive dominoes. ∎
So the colour invariant is not merely necessary here — it is sufficient, which is unusual and worth noting.
- Theory
Three piles hold 10, 15 and 20 stones. A move takes one stone from each of two piles and adds two to the third. Can you reach 15, 15, 15?
Solution
Look at the pile sizes mod 3. A move changes a pair by -1,-1 and the third by +2. Since +2 ≡ -1 (mod 3), every pile decreases by 1 mod 3 — so the pairwise differences mod 3 are invariant.
Start: (10,15,20) ≡ (1,0,2) (mod 3), all distinct. Target: (15,15,15) ≡ (0,0,0), all equal. Differences mod 3 cannot change from "all distinct" to "all equal". Unreachable. ∎
- Theory
Prove that bubble sort performs exactly inv(A) swaps, where inv is the number of inversions, and that this is optimal for adjacent-transposition sorting.
Solution
Each bubble-sort swap exchanges an adjacent out-of-order pair, which removes exactly that inversion and changes no other pair's relative order. So swaps and inversions decrease in lockstep, and the process ends when inversions reach 0 — after exactly inv(A) swaps.
Optimality: any algorithm restricted to adjacent transpositions changes the inversion count by exactly ±1 per operation. Going from inv(A) to 0 therefore needs at least inv(A) operations. ∎ The maximum, C(n,2), occurs for a reversed array — which is why insertion sort is Θ(n + inv) and genuinely linear on nearly-sorted data.
- Theory
Prove that any 15-puzzle position with an odd inversion-plus-blank-row parity is unsolvable, and compute what fraction of the 16! arrangements are solvable.
Solution
The invariant argument is in the chapter's code block. Since the invariant partitions the state space into two classes and every move preserves it, no position in the odd class reaches the solved (even) position.
Fraction: exactly half, 16!/2 = 10{,}461{,}394{,}944{,}000 solvable arrangements. That the split is exactly even follows because transposing any two non-blank tiles flips the invariant, giving a bijection between the two classes. The converse (every even position is solvable) is true but needs a separate constructive argument.
- Theory
n points in the plane, no three collinear, each joined to its nearest neighbour. Prove no point has more than 5 edges pointing at it.
Solution
Suppose points a₁,…,aₖ all have p as their nearest neighbour. Take any two, aᵢ, a₃. Since p is nearest to aᵢ, we have |aᵢa₃| > |aᵢp|, and symmetrically |aᵢa₃| > |a₃p|. In triangle aᵢ p a₃, the largest side is opposite the largest angle, so the angle at p exceeds both others, hence ∠aᵢpa₃ > 60°.
The angles at p around a full turn sum to 360°, and each pair of consecutive rays subtends more than 60°, so k < 360/60 = 6, i.e. k ≤ 5. ∎ This bounds the in-degree of a nearest-neighbour graph by 5, which is why such graphs have O(n) edges and can be built and traversed in linear time once computed.
- Theory
Use the extremal principle to prove that every finite DAG has a vertex with in-degree 0.
Solution
Take a longest directed path v₀ → v₁ → … → vₖ (one exists: the graph is finite, and path lengths are bounded by n since acyclicity forbids repeats). If v₀ had an incoming edge from u, then either u is off the path — giving a longer path, contradiction — or u is some vᵢ on the path, giving a cycle, contradicting acyclicity. So v₀ has in-degree 0. ∎
This is the existence proof underlying Kahn's algorithm: the queue is never empty while vertices remain, so topological sort always succeeds on a DAG.
- Program
Implement a 15-puzzle solvability checker in O(n²) or O(n log n), generalised to k × k boards.
Solution
def solvable(board, k): # board: flat list, 0 = blank tiles = [t for t in board if t != 0] inv = sum(1 for i in range(len(tiles)) for j in range(i+1, len(tiles)) if tiles[i] > tiles[j]) if k % 2 == 1: return inv % 2 == 0 # odd width: parity of inv alone blank_row_from_bottom = k - (board.index(0) // k) return (inv + blank_row_from_bottom) % 2 == 1The width parity matters because a vertical move jumps the blank over k-1 tiles: when k is odd that is an even number of inversions changed (invariant is inversions alone), when k is even it is odd (so the blank's row must be folded in). Replace the O(n²) inversion count with a BIT for O(n log n) on large boards.
- Program
Count inversions in an array of up to 10⁶ elements in O(n log n), two ways: merge sort and a Fenwick tree.
Solution
# merge sort: count pairs crossing the split def count(a): if len(a) < 2: return a, 0 m = len(a)//2 L, x = count(a[:m]); R, y = count(a[m:]) out, i, j, c = [], 0, 0, x + y while i < len(L) and j < len(R): if L[i] <= R[j]: out.append(L[i]); i += 1 else: out.append(R[j]); j += 1; c += len(L) - i # L[i:] all beat R[j] return out + L[i:] + R[j:], c # Fenwick: sweep right-to-left, count already-seen smaller values def count_bit(a): r = {v:i+1 for i,v in enumerate(sorted(set(a)))} # coordinate compress bit = [0]*(len(r)+1); total = 0 for v in reversed(a): k = r[v]-1 while k: total += bit[k]; k -= k & -k # query prefix k = r[v] while k <= len(r): bit[k] += 1; k += k & -k # update return totalBoth are O(n log n). The merge version needs no compression and is cache-friendlier; the Fenwick version generalises to weighted counts and to online queries. The key line in the merge version is
c += len(L) - i: whenR[j]is emitted, every remaining element of L is larger and to its left, so all of them are inversions — the whole algorithm is that observation. - Program
Given a chocolate bar of m × n squares, prove that breaking it into unit squares always takes exactly mn - 1 breaks, and write a verifier that plays random break sequences.
Solution
Monovariant / invariant: each break increases the number of pieces by exactly 1. Starting at 1 piece and ending at mn, the number of breaks is mn - 1 regardless of strategy. ∎ No search needed, and no strategy is better than another — which is the surprising part.
import random def play(m, n): pieces, breaks = [(m, n)], 0 while any(p != (1,1) for p in pieces): i = random.choice([i for i,p in enumerate(pieces) if p != (1,1)]) a, b = pieces.pop(i); breaks += 1 if a > 1 and (b == 1 or random.random() < .5): k = random.randint(1, a-1); pieces += [(k,b), (a-k,b)] else: k = random.randint(1, b-1); pieces += [(a,k), (a,b-k)] return breaks assert all(play(m,n) == m*n - 1 for m in range(1,7) for n in range(1,7))This is the cleanest possible illustration that an invariant can settle a question that looks like it needs game analysis.
- Program
Implement a solver for the "water jug" problem (jugs of capacity a and b, target c) using BFS, and separately decide reachability in O(log) using an invariant.
Solution
Invariant: every reachable amount is an integer combination of a and b, so it is a multiple of g = gcd(a,b). Conversely every multiple of g up to max(a,b) is reachable (Bézout, Chapter 15). So:
def reachable(a, b, c): from math import gcd return c == 0 or (c <= max(a, b) and c % gcd(a, b) == 0)The BFS over states (x,y) with 0 ≤ x ≤ a, 0 ≤ y ≤ b and six moves (fill either, empty either, pour either way) explores O(ab) states and confirms this empirically. For a,b ≤ 10⁹ only the invariant version is viable — the classic case where a number-theoretic invariant collapses an exponential search to a gcd.
- Program
Write a stress test that searches for an invariant automatically: given a small state space and a move relation, compute the reachable set from a start state by BFS and report which simple functions (sum mod k, parity of a coordinate, …) are constant on it.
Solution
from itertools import product reach = bfs(start, moves) # set of reachable states cands = {} for k in range(2, 13): vals = {sum(s) % k for s in reach} if len(vals) == 1: cands[f"sum mod {k}"] = vals.pop() for i in range(dim): vals = {s[i] % 2 for s in reach} if len(vals) == 1: cands[f"coord {i} parity"] = vals.pop() print(cands)This is a genuinely useful contest technique: brute-force the reachable set for small parameters, look for a constant statistic, then prove it preserved by each move and generalise. It converts "guess the invariant" from inspiration into search. The caveat is that a statistic constant on a small reachable set may not be invariant in general — the program suggests, the proof decides.
- Program
Prove and implement: a knight on an infinite board cannot return to its start in an odd number of moves. Verify by BFS on an 8×8 board.
Solution
Invariant: colour the board like a chessboard. Every knight move changes colour (a (1,2) displacement changes x+y by an odd amount). So after k moves the knight is on its starting colour iff k is even. Returning to the start requires the starting colour, hence an even k. ∎
from collections import deque dist = {(0,0): 0}; q = deque([(0,0)]) D = [(1,2),(2,1),(-1,2),(-2,1),(1,-2),(2,-1),(-1,-2),(-2,-1)] while q: x, y = q.popleft() for dx, dy in D: n = (x+dx, y+dy) if 0 <= n[0] < 8 and 0 <= n[1] < 8 and n not in dist: dist[n] = dist[(x,y)] + 1; q.append(n) assert all((x+y) % 2 == d % 2 for (x,y), d in dist.items())The assertion is the invariant, checked on every reachable square: shortest-path parity equals coordinate-sum parity. It also proves BFS distances on a knight graph are never both even and odd for the same square — the graph is bipartite, which is the general statement of this colouring argument.
- Program
Implement a lexicographic termination checker: given a multi-counter loop, assert that a tuple variant strictly decreases lexicographically each iteration. Apply it to the Euclidean algorithm on pairs and to a nested loop with a reset.
Solution
def lex_lt(a, b): return a < b # tuples compare lexicographically prev = variant(state) while not done(state): state = step(state) cur = variant(state) assert all(v >= 0 for v in cur), "variant left the well-founded domain" assert lex_lt(cur, prev), f"not decreasing: {prev} -> {cur}" prev = curEuclid on pairs: variant (a, b) with a ≥ b — the pair strictly decreases lexicographically even though
aalone does not on the swap step.Nested loop with reset: an inner counter that resets to n whenever the outer counter drops. No single counter decreases (the inner one jumps up), but (outer, inner) decreases lexicographically every iteration, and lexicographic order on ℕᵏ is well-founded — so it terminates. This is exactly the pattern that defeats naive termination checkers and exactly what the tuple variant is for.
Asymptotic notation, formally
Big-O is a set of functions defined by a quantifier sandwich, not a synonym for "roughly". Getting the definition exact fixes the three mistakes almost everyone makes: confusing O with Θ, asserting tightness that was never proved, and reasoning about constants that the notation deliberately discarded.
The definitions, stated as the quantified formulas they are:
Two structural facts are visible in the quantifiers. First, O and Ω begin with ∃c, so one constant suffices; o and ω begin with ∀c, which is far stronger. Second, everything is eventual: behaviour below n₀ is irrelevant, so no finite amount of measurement can establish or refute an asymptotic claim.
The = sign is an abuse of notation. O(g) is a set of functions and the honest statement is f ∈ O(g). The abuse is harmless in one direction and dangerous in the other: n = O(n²) and n² = O(n²) are both true, but you may not conclude n = n². Read = as "is", never as "equals".
The three mistakes
1. Saying O when you mean Θ. "Quicksort is O(n log n)" is true but weak; it is also O(n²) and O(2ⁿ), all correct upper bounds. The precise statements are: worst case Θ(n²), expected case Θ(n log n).
2. Confusing case with bound. Best/average/worst case selects which function of n you are describing; O/Θ/Ω describes how tightly you are describing it. They are orthogonal, and "the worst case is Ω(n log n)" is a perfectly sensible sentence that people wrongly correct.
3. Believing asymptotics predict runtime. An O(n log n) algorithm with a 500× constant loses to an O(n²) one until n is large. Galactic algorithms — matrix multiplication at O(n²·³⁷²), for example — have constants so large they are never run.
Asymptotics answer "how does cost scale?" not "how long does it take?". In competitive programming the useful translation is the operations budget: roughly 10⁸ simple operations per second. At n = 10⁵, an O(n²) solution is 10¹⁰ and fails; O(n log n) is 1.7·10⁶ and passes comfortably. That translation, not the notation, is what you actually use under time pressure.
The growth hierarchy
| Class | n=10⁶ | Typical source |
|---|---|---|
| O(1) | 1 | hash lookup, arithmetic |
| O(α(n)) | < 5 | union-find |
| O(log n) | 20 | binary search, balanced tree |
| O(√n) | 1,000 | trial division, sqrt decomposition |
| O(n) | 10⁶ | one pass |
| O(n log n) | 2·10⁷ | sorting, divide and conquer |
| O(n√n) | 10⁹ | Mo's algorithm |
| O(n²) | 10¹² | all pairs — dead above n ≈ 10⁴ |
| O(2ⁿ) | — | subsets — dead above n ≈ 25 |
| O(n!) | — | permutations — dead above n ≈ 11 |
The reverse reading of this table is the one that wins contests: given the constraint, infer the intended complexity. n ≤ 20 means bitmask DP. n ≤ 500 means O(n³). n ≤ 10⁵ means O(n log n). n ≤ 10¹⁸ means O(log n) or a closed form.
Comparing growth rates
The limit test decides most cases: if lim f/g = 0 then f = o(g); if the limit is a positive constant, f = Θ(g); if it is ∞, then f = ω(g). If the limit does not exist, the functions may still be comparable, or genuinely incomparable — f(n) = n^(1+sin n) is neither O(n) nor Ω(n²).
log^k n = o(n^ε) for every k and every ε > 0 (logs beat nothing)
n^k = o(c^n) for every k and every c > 1 (polys lose to exps)
c^n = o(n!) n! ≈ (n/e)^n √(2πn)
n! = o(n^n)
log(n!) = Θ(n log n) Stirling
Σ_{i=1..n} 1/i = ln n + γ + o(1) = Θ(log n) harmonic
a^log b = b^log a same base throughout
log_a n = Θ(log_b n) ⟹ the base of a log inside Θ is irrelevant
BUT 2^log₂n = n while 2^log₁₀n = n^0.30
— the base matters in an exponentWhen asked for a complexity, give the tight bound, name the case, and say what dominates: "Θ(n log n) expected, dominated by the sort; worst case Θ(n²) if the pivot is adversarial, which randomising the pivot removes." Three clauses, and it demonstrates you know what the notation is hiding.
- Theory
Prove from the definition that 3n² + 10n + 7 = Θ(n²), giving explicit constants.
Solution
Upper: for n ≥ 1, 10n ≤ 10n² and 7 ≤ 7n², so f(n) ≤ 20n². Take c₂ = 20, n₀ = 1.
Lower: all terms are non-negative for n ≥ 0, so f(n) ≥ 3n². Take c₁ = 3, n₀ = 1.
Hence 3n² ≤ f(n) ≤ 20n² for n ≥ 1, so f = Θ(n²). ∎ Note the constants are not unique and no smallest one is required — only existence.
- Theory
Is 2ⁿ⁺¹ = O(2ⁿ)? Is 2²ⁿ = O(2ⁿ)? Justify both.
Solution
2ⁿ⁺¹ = 2·2ⁿ, so with c = 2 it is O(2ⁿ) — yes. A constant factor in the exponent's offset is a constant factor overall.
2²ⁿ = (2ⁿ)², and 2²ⁿ/2ⁿ = 2ⁿ → ∞, so no constant c works — no, it is ω(2ⁿ). A constant factor in the exponent's coefficient is an exponential factor overall.
This is why "O(2ⁿ) so the constant does not matter" is wrong: halving the exponent (meet-in-the-middle, O(2^{n/2})) is the difference between n = 20 and n = 40.
- Theory
Order by growth rate, grouping ties: n log n, n^1.01, 2^{log₂ n}, n / log n, (log n)^{log n}, n^{1/log n}, 2^{√(log n)}.
Solution
Take logs of everything: log n + log log n; 1.01 log n; log n (since 2^{log₂n} = n); log n - log log n; (log n)(log log n); (log n)/(log n) = 1, i.e. the function is the constant 2; √(log n).
n^{1/log n} = 2 constant 2^{√log n} sub-polynomial, super-polylog n/log n 2^{log₂ n} = n (tie with nothing else here) n log n n^{1.01} beats n log n since 0.01 log n → ∞ beats log log n (log n)^{log n} = 2^{(log log n)(log n)}, super-polynomialThe technique — compare logarithms, since log is strictly increasing — is the general method and it handles every case above mechanically.
- Theory
Prove or disprove: if f = O(g) then 2^{f} = O(2^{g}).
Solution
False. Take f(n) = 2n and g(n) = n. Then f = O(g) with c = 2, but 2^{f} = 4ⁿ and 2^{g} = 2ⁿ, and 4ⁿ/2ⁿ = 2ⁿ → ∞. ∎
The reason: O hides multiplicative constants, and exponentiation turns a multiplicative constant into an exponential one. The correct statement needs f ≤ g + O(1), not f = O(g). The same trap kills "f = O(g) ⇒ f! = O(g!)".
- Theory
Prove that O(f) + O(g) = O(max(f,g)) and use it to justify why analysing a sequence of phases means taking the maximum.
Solution
Let h₁ ≤ c₁f for n ≥ n₁ and h₂ ≤ c₂g for n ≥ n₂. For n ≥ max(n₁,n₂), h₁+h₂ ≤ c₁f + c₂g ≤ (c₁+c₂)·max(f,g). ∎
Consequence: an algorithm that sorts (Θ(n log n)) then does a linear scan (Θ(n)) is Θ(n log n) overall — the phases do not add up in the notation, the largest simply absorbs the rest. This is also why micro-optimising a linear pass inside an O(n²) algorithm is wasted effort, and why max and + are interchangeable in asymptotic sums.
- Theory
Give functions f, g with neither f = O(g) nor g = O(f).
Solution
Let f(n) = n and g(n) = n^{1 + sin(n)}, so g oscillates between n⁰ and n² infinitely often. Then f/g is unbounded and g/f is unbounded, so neither bound holds.
A cleaner discrete version: f(n) = n, and g(n) = n² for even n, 1 for odd n. ∎
The point is that asymptotic comparison is a partial order (Chapter 5), not a total one. Every real algorithm's cost function is comparable in practice, which is why this is easy to forget until a proof needs the general case.
- Theory
Prove log(n!) = Θ(n log n) without invoking Stirling's formula.
Solution
Upper: n! ≤ nⁿ, so log(n!) ≤ n log n.
Lower: the top half of the factors are each at least n/2, so n! ≥ (n/2)^{n/2}, giving log(n!) ≥ (n/2)(log n - 1) = Ω(n log n).
Hence Θ(n log n). ∎ This is the bound behind the Ω(n log n) comparison-sort lower bound of Chapter 13: a decision tree with n! leaves has depth at least log₂(n!).
- Theory
An algorithm runs in T(n) = 10⁶ n and another in T(n) = n². For which n is the first faster? Discuss what this means for asymptotic reasoning in practice.
Solution
10⁶n < n² iff n > 10⁶. So the "better" linear algorithm is slower for every input below a million.
What it means: asymptotics are a statement about the limit, and the crossover point is invisible to them. Real engineering answers this with hybrid algorithms —
std::sortswitches to insertion sort below ~16 elements, Karatsuba multiplication reverts to schoolbook below ~32 limbs, and Timsort has a whole minrun heuristic. Each is an explicit acknowledgement that the constant beats the exponent in the range that actually occurs. - Program
Write a program that empirically estimates the exponent k in T(n) ≈ c nᵏ by timing an algorithm at several sizes and fitting log T against log n. Apply it to bubble sort and to
sorted().Solution
import time, random, math def timeit(f, n, reps=3): best = float('inf') for _ in range(reps): a = [random.random() for _ in range(n)] t = time.perf_counter(); f(a); best = min(best, time.perf_counter()-t) return best def exponent(f, sizes): xs = [math.log(n) for n in sizes] ys = [math.log(timeit(f, n)) for n in sizes] mx, my = sum(xs)/len(xs), sum(ys)/len(ys) num = sum((x-mx)*(y-my) for x, y in zip(xs, ys)) den = sum((x-mx)**2 for x in xs) return num/den # slope = estimated k print(exponent(bubble, [200, 400, 800, 1600])) # ≈ 2.0 print(exponent(sorted, [10**5, 2*10**5, 4*10**5, 8*10**5])) # ≈ 1.05The sort measures ~1.05 rather than 1.0 because n log n looks like n^{1+log log n/log n} over any finite range — a genuine limitation of the method: it cannot distinguish n log n from n^{1.05} empirically. That is the experimental face of the fact that asymptotic claims are not falsifiable by measurement.
- Program
Given a problem constraint, output the intended complexity class. Write the lookup as code and justify each threshold with the 10⁸ ops/second budget.
Solution
def intended(n): if n <= 11: return "O(n!) permutations / brute force" if n <= 25: return "O(2^n) subset enumeration, meet-in-middle at 40" if n <= 100: return "O(n^4) or O(n^3 log n)" if n <= 500: return "O(n^3) Floyd-Warshall, matrix ops" if n <= 5000: return "O(n^2) DP over pairs" if n <= 10**5: return "O(n sqrt n) or O(n log^2 n)" if n <= 10**6: return "O(n log n) sort, heap, divide and conquer" if n <= 10**7: return "O(n) single pass, no allocation per element" if n <= 10**12: return "O(sqrt n) trial division, Pollard rho" return "O(log n) or closed form"The thresholds come from 10⁸ operations/second and a typical 1–2 s limit: n² at n = 5000 is 2.5·10⁷ (fine), at n = 10⁵ is 10¹⁰ (hopeless). Reading the constraint first is the single most useful contest habit, because it tells you which algorithms are even candidates before you think about the problem.
- Program
Implement three algorithms for maximum subarray sum — O(n³), O(n²) and Kadane's O(n) — and measure the largest n each handles in one second.
Solution
def kadane(a): # O(n) best = cur = a[0] for x in a[1:]: cur = max(x, cur + x) # extend or restart best = max(best, cur) return best def quad(a): # O(n^2) best = a[0] for i in range(len(a)): s = 0 for j in range(i, len(a)): s += a[j]; best = max(best, s) return bestRough limits at 10⁸ ops/s in a compiled language: cubic ~ n = 500, quadratic ~ n = 10⁴, Kadane ~ n = 10⁸. In Python divide each by roughly 50.
Kadane's correctness is the invariant "
curis the maximum sum of a subarray ending exactly at the current index" — proved by induction, since such a subarray either extends the previous one or starts fresh. The whole n³ → n improvement is that one observation, which is typical: complexity classes are crossed by finding structure, not by optimising code. - Program
Write a benchmark demonstrating that an O(n) algorithm with poor cache behaviour can lose to an O(n log n) one with good locality. Explain what the model omits.
Solution
// A: O(n) — random access into a large array (cache-hostile) long long a = 0; for (int i = 0; i < n; i++) a += data[perm[i]]; // random order // B: O(n log n) — sort a copy, then a sequential scan sort(copy.begin(), copy.end()); long long b = 0; for (int i = 0; i < n; i++) b += copy[i]; // sequentialAt n = 10⁸ with 4-byte ints (400 MB), loop A incurs a cache miss on nearly every access — about 100 ns each, so ~10 s — while the sort plus sequential scan streams at memory bandwidth and finishes in ~2 s despite doing log n = 27× more "operations".
What the RAM model omits: it charges 1 for every memory access, but real hardware has a 100× spread between L1 and DRAM. The external-memory (cache-oblivious) model fixes this by counting block transfers instead, and it is why B-trees beat binary search trees on disk and why array-of-structs versus struct-of-arrays changes runtime without changing complexity.
- Program
Implement a function that counts the exact number of basic operations an algorithm performs, and use it to verify a claimed Θ bound by checking that T(2n)/T(n) approaches the expected ratio.
Solution
class Counter: def __init__(self): self.n = 0 def tick(self, k=1): self.n += k def count_ops(alg, n): c = Counter(); alg(list(range(n)), c); return c.n for n in [1000, 2000, 4000, 8000, 16000]: print(n, count_ops(alg, n), count_ops(alg, 2*n) / count_ops(alg, n))Expected ratios T(2n)/T(n): 2 for linear, slightly above 2 for n log n (precisely 2 + 2/log₂n), 4 for quadratic, 8 for cubic. Counting operations rather than timing removes constant factors, JIT warm-up and cache effects, so the ratio converges cleanly — this is the right way to check an implementation matches its claimed complexity in a test suite. Assert
1.9 < ratio < 2.3and a future refactor that accidentally makes the algorithm quadratic fails CI. - Program
Given n ≤ 40 integers, find whether a subset sums to S. Naive is O(2ⁿ) — implement meet-in-the-middle at O(2^{n/2} · n) and explain the asymptotic significance.
Solution
def subset_sum(a, S): n = len(a); h = n // 2 def sums(xs): r = [0] for x in xs: r += [v + x for v in r] return r L, R = sums(a[:h]), sorted(sums(a[h:])) import bisect for v in L: i = bisect.bisect_left(R, S - v) if i < len(R) and R[i] == S - v: return True return FalseAt n = 40: naive is 2⁴⁰ ≈ 10¹² (hours); meet-in-the-middle is 2²⁰ = 10⁶ per half plus a sort, well under a second. The improvement is not a constant factor and not a change of complexity class in the usual sense — it halves the exponent, which is exactly the distinction problem 2 makes. This is the standard escape hatch when n sits in the 30–45 window: too big for 2ⁿ, too small for anything polynomial to exist.
- Program
Write a script that, given two implementations, finds the crossover input size where the asymptotically better one starts winning, using binary search over n.
Solution
def crossover(fast, slow, lo=1, hi=10**7): # invariant: slow wins at lo, fast wins at hi (verify before searching) while lo < hi: mid = (lo + hi) // 2 if timeit(fast, mid) < timeit(slow, mid): hi = mid else: lo = mid + 1 return loTwo caveats that make this harder than it looks. First, the predicate must be monotone for binary search to be valid — timing noise breaks monotonicity, so take the minimum of several runs and require a margin (say fast < 0.9·slow) before deciding. Second, the crossover moves with the machine, the allocator and the data distribution, so the number is an observation about one environment, not a property of the algorithms.
Used properly this is how you pick the cutoff for a hybrid algorithm: measure once, hard-code the threshold, and re-measure when the hardware changes.
Summations and closed forms
Every loop nest is a summation, and every summation you can close is a complexity you can state exactly. A dozen closed forms and three manipulation techniques cover essentially all of algorithm analysis.
Analysing an algorithm means writing its cost as a sum and then evaluating or bounding that sum. The translation is mechanical: a loop is a ∑, nesting is nesting, and the bounds come from the loop bounds.
Note the reindexing in the last step: substituting k = n-i+1 turned a decreasing sum into the standard triangular one. Reindexing, splitting and bounding are the three moves, and they suffice for nearly everything.
The closed forms to know cold
The geometric sum deserves emphasis because it is the one that most often collapses an analysis. 1 + 2 + 4 + … + 2ᵏ = 2ᵏ⁺¹ - 1 < 2·2ᵏ: a geometric series is dominated by its largest term, up to a constant. That single fact explains why doubling a dynamic array is amortised O(1), why the total work in a divide-and-conquer with geometrically shrinking levels is dominated by one level, and why a geometric-backoff retry loop has bounded total delay.
If the terms grow or shrink geometrically, the sum is Θ(largest term). If they grow polynomially, the sum is Θ(n · largest term) up to the degree shift. Recognising which regime you are in answers most summation questions without evaluating anything.
Three techniques
1. Splitting. To bound ∑ᵢ₋₁ⁿ 1/i from above, split into blocks of size 1, 2, 4, 8, …; each block sums to at most 1, and there are log₂n blocks, giving Hₙ ≤ 1 + log₂n. Blocking with the same trick from below gives Hₙ ≥ 1 + (log₂n)/2. Hence Hₙ = Θ(log n) with no calculus.
2. Integral comparison. For monotone f, ∫₁ⁿ⁺¹ f ≤ ∑ᵢ₋₁ⁿ f(i) ≤ f(1) + ∫₁ⁿ f (increasing f; reverse for decreasing). This gives ∑ iᵏ ≈ n^{k+1}/(k+1) and Hₙ ≈ ln n immediately, and is the fastest route to the leading term of almost any sum.
3. Perturbation. To evaluate S = ∑ᵢ₋₀ⁿ i·2ᵢ, write 2S, shift the index, subtract. The telescoping leaves a geometric sum you already know:
S = Σ_{i=0..n} i·2ⁱ
2S = Σ_{i=0..n} i·2^{i+1} = Σ_{j=1..n+1} (j-1)·2ʲ
2S - S = (n)·2^{n+1} - Σ_{j=1..n} 2ʲ ← after aligning indices
S = n·2^{n+1} - (2^{n+1} - 2)
= (n-1)·2^{n+1} + 2
Check n = 3: 0 + 2 + 8 + 24 = 34, and (3-1)·16 + 2 = 34. ✓Always check a closed form at a small n. Off-by-one errors in summation bounds are the single most common analysis mistake, and a 5-second check at n = 1, 2, 3 catches nearly all of them.
Sums that appear in real analyses
| Sum | Value | Where it appears |
|---|---|---|
| ∑ n/2ᵢ | < 2n | heapify is O(n), not O(n log n) |
| ∑ 2ᵢ to n | < 2n | dynamic array doubling |
| ∑ n/i | n Hₙ ≈ n ln n | sieve of Eratosthenes, harmonic loops |
| ∑ log i | Θ(n log n) | building a BST by n insertions |
| ∑ i | Θ(n²) | insertion sort, naive string concat |
| ∑ √i | Θ(n¹·⁵) | sqrt decomposition rebuilds |
The first and third rows are the ones people get wrong. for i in 1..n: for j in i..n step i runs ∑ n/i = Θ(n log n) times, not Θ(n²) — the harmonic series is why the sieve is nearly linear.
"Why is building a heap O(n) when each sift-down is O(log n)?" The answer is a summation: nodes at height h number about n/2Ḥ⁺¹ and cost O(h), so the total is n ∑ h/2Ḥ ≤ 2n. The naive n · log n multiplies the worst case by the count instead of summing the actual costs, and spotting that difference is the whole question.
- Theory
Evaluate ∑ᵢ₋₁ⁿ i(i+1) in closed form.
Solution
Split: ∑ i² + ∑ i = n(n+1)(2n+1)/6 + n(n+1)/2 = n(n+1)[(2n+1)+3]/6 = n(n+1)(n+2)/3.
Check n = 2: 1·2 + 2·3 = 8, and 2·3·4/3 = 8. ✓ The result n(n+1)(n+2)/3 = 2·C(n+2,3) is not a coincidence — i(i+1) = 2·C(i+1,2) and hockey stick gives the sum directly, which is the slicker derivation.
- Theory
Compute ∑ᵢ₋₀ⁿ i²·2ᵢ using perturbation.
Solution
Let S = ∑ i²2ᵢ. Then 2S = ∑₃₋₁ⁿ⁺¹ (j-1)²2₌, so
S = 2S - S = n²2ⁿ⁺¹ + ∑₃₋₁ⁿ[(j-1)² - j²]2₌ = n²2ⁿ⁺¹ - ∑₃₋₁ⁿ(2j-1)2₌.
Using ∑ j2₌ = (n-1)2ⁿ⁺¹+2 and ∑ 2₌ = 2ⁿ⁺¹-2:
S = n²2ⁿ⁺¹ - 2[(n-1)2ⁿ⁺¹+2] + (2ⁿ⁺¹-2) = (n²-2n+3)2ⁿ⁺¹ - 6.
Check n = 2: 0 + 2 + 16 = 18, and (4-4+3)·8 - 6 = 18. ✓
- Theory
Prove Hₙ = Θ(log n) by the blocking argument, giving explicit constants.
Solution
Upper: group terms as 1, (1/2+1/3), (1/4+…+1/7), … The k-th block has 2ᵏ terms each at most 2⁻ᵏ, so it sums to at most 1. With ⌈log₂(n+1)⌉ blocks, Hₙ ≤ 1 + log₂n.
Lower: the same blocks each have 2ᵏ terms each at least 2^{-(k+1)}, so each sums to at least 1/2, giving Hₙ ≥ 1 + (log₂n)/2 for n = 2ᵏ.
So (1/2)log₂n ≤ Hₙ ≤ 1 + log₂n and Hₙ = Θ(log n). ∎ The true constant is ln n = 0.693 log₂n, comfortably inside these bounds.
- Theory
Show that the sieve of Eratosthenes performs Θ(n log log n) operations, not Θ(n log n).
Solution
The work is ∑p ≤ n, p prime n/p = n ∑p ≤ n 1/p. Mertens' second theorem gives ∑p ≤ n 1/p = ln ln n + M + o(1) with M ≈ 0.2615. Hence Θ(n log log n). ∎
The naive bound ∑i ≤ n n/i = n Hₙ = Θ(n log n) is what you get if you cross out multiples of every integer rather than only primes — the difference between log n and log log n is exactly the sparsity of the primes. At n = 10⁸, log log n ≈ 2.9 versus log n ≈ 18, a 6× difference that is very visible in practice.
- Theory
Prove that heapify is O(n) by summing the actual sift-down costs.
Solution
In a heap of n elements, the number of nodes at height h is at most ⌈n/2^{h+1}⌉, and sifting down from height h costs O(h). Total:
∑h=0..⌊log n⌋ (n/2^{h+1})·O(h) = O(n ∑h≥0 h/2Ḥ) = O(n · 2) = O(n),
using ∑ h xḤ = x/(1-x)² at x = 1/2, which is 2. ∎
The intuition to state aloud: half the nodes are leaves and cost nothing, a quarter cost 1, an eighth cost 2 — the expensive nodes are exponentially rare, and the geometric series converges. Inserting n elements one at a time is genuinely Θ(n log n) because there the expensive operations are the common ones.
- Theory
What is ∑ᵢ₋₁ⁿ ∑₃₋₁ᵢ 1/j? Give the asymptotic.
Solution
Swap the order of summation. The pair (i,j) is counted when j ≤ i ≤ n, so
∑₃₋₁ⁿ (1/j)·|{i : j ≤ i ≤ n}| = ∑₃₋₁ⁿ (n-j+1)/j = (n+1)Hₙ - n.
So the value is (n+1)Hₙ - n = Θ(n log n). ∎
Exchanging the order of summation is the most reliable technique for double sums: rewrite the index constraint as a region, then integrate over it in the other order. It is the discrete Fubini theorem, and it turns many analyses that look quadratic into n log n.
- Theory
An algorithm at each step of size k does k work and recurses on k/2. Write the total as a sum and evaluate. Repeat when it does k² work, and when it does √k.
Solution
Work k: n + n/2 + n/4 + … = n∑2⁻ᵢ < 2n = Θ(n) — geometric, dominated by the top.
Work k²: n² + n²/4 + n²/16 + … = n²·(4/3) = Θ(n²) — even more strongly top-dominated.
Work √k: √n(1 + 1/√2 + 1/2 + …) = √n · 1/(1-1/√2) ≈ 3.41√n = Θ(√n) — still geometric, still top-dominated.
All three are Θ(top-level work), and that is the general rule for a single recursive call on a constant fraction: the levels form a geometric series with ratio < 1. It changes only when the branching factor multiplies the work back up, which is the master theorem of Chapter 10.
- Program
Write a program that verifies a claimed closed form against direct summation for n = 1..10⁵, using exact integer arithmetic. Test it on the four Faulhaber formulas.
Solution
def check(term, closed, N=100000): s = 0 for n in range(1, N+1): s += term(n) if s != closed(n): return ("mismatch at n =", n, s, closed(n)) return "ok" print(check(lambda i: i, lambda n: n*(n+1)//2)) print(check(lambda i: i*i, lambda n: n*(n+1)*(2*n+1)//6)) print(check(lambda i: i**3, lambda n: (n*(n+1)//2)**2)) print(check(lambda i: i**4, lambda n: n*(n+1)*(2*n+1)*(3*n*n+3*n-1)//30))Use integer division only where divisibility is guaranteed — n(n+1) is always even, n(n+1)(2n+1) is always divisible by 6 — otherwise
//silently truncates and the check "passes" on a wrong formula for some n. In a language with fixed-width integers this loop also overflows around n = 10⁵ for the cube formula (n⁴/4 ≈ 2.5·10¹⁹ exceeds 2&sup6;³ only past n ≈ 2·10⁴… but the intermediate (n(n+1)/2)² does not) — worth checking deliberately. - Program
Count exactly how many times the inner statement runs for
for i in 1..n: for j in 1..n step i. Compare the measured count to n Hₙ.Solution
def count(n): c = 0 for i in range(1, n+1): j = 1 while j <= n: c += 1; j += i return c import math for n in [10**3, 10**4, 10**5]: c = count(n) H = sum(1.0/i for i in range(1, n+1)) print(n, c, n*H, c / (n*H)) # ratio → 1The exact count is ∑ᵢ₋₁ⁿ ⌊n/i⌋, which equals n Hₙ - O(n) because each floor loses less than 1. The ratio approaches 1 from below. This same sum ∑⌊n/i⌋ is also the total number of divisors of all integers up to n, and it can be computed in O(√n) by the divisor-block trick — grouping equal values of ⌊n/i⌋, which is Chapter 19's technique.
- Program
Implement ∑ᵢ₋₁ⁿ ⌊n/i⌋ in O(√n) for n ≤ 10¹².
Solution
⌊n/i⌋ takes only O(√n) distinct values, and the i giving each value form a contiguous block.
def divisor_sum(n): total, i = 0, 1 while i <= n: q = n // i j = n // q # largest index with the same quotient total += q * (j - i + 1) i = j + 1 return totalWhy blocks: if q = ⌊n/i⌋ then every i' ≤ ⌊n/q⌋ gives the same quotient, and ⌊n/q⌋ is the exact right endpoint. There are at most 2√n blocks because for i ≤ √n there are √n indices, and for i > √n the quotient is below √n so there are at most √n distinct values. At n = 10¹² that is 2 million iterations — instant, versus a trillion for the naive loop.
- Program
Empirically demonstrate that repeated string concatenation is Θ(n²) while joining a list is Θ(n), and explain the summation behind each.
Solution
def quadratic(n): s = "" for i in range(n): s += "x" # copies the whole prefix each time return s def linear(n): parts = [] for i in range(n): parts.append("x") return "".join(parts) # one allocation, one passConcatenation: step i copies i characters, so the total is ∑ᵢ₋₁ⁿ i = Θ(n²). (CPython has an in-place optimisation that sometimes hides this for
str; the quadratic behaviour is immediate and unavoidable in Java withStringin a loop, which is whyStringBuilderexists.)Join: appends are amortised O(1) by array doubling — total copy work ∑ 2ᵢ < 2n — and the final join allocates once and copies n characters. Total Θ(n).
The two summations, ∑i versus ∑2ᵢ, are exactly the difference between the arithmetic and geometric regimes from the chapter's Principle box.
- Program
Given q queries each asking for ∑ᵢ₋₂₋ᵣ aᵢ on a static array, answer all in O(n+q). Extend to 2D and to a single point update.
Solution
// 1D prefix sums pre[0] = 0; for (int i = 0; i < n; i++) pre[i+1] = pre[i] + a[i]; auto query = [&](int l, int r){ return pre[r+1] - pre[l]; }; // O(1) // 2D: inclusion-exclusion over four corners P[i+1][j+1] = A[i][j] + P[i][j+1] + P[i+1][j] - P[i][j]; sum(r1,c1,r2,c2) = P[r2+1][c2+1] - P[r1][c2+1] - P[r2+1][c1] + P[r1][c1];With updates, prefix sums cost O(n) per update, so switch to a Fenwick tree: O(log n) both ways. The choice is a summation trade-off — prefix sums pay O(n) once for O(1) queries, Fenwick pays O(log n) always. With u updates and q queries, prefix sums win when u·n < (u+q)log n, i.e. essentially when updates are rare.
The 2D formula is inclusion–exclusion (Chapter 30) applied to rectangles, and the sign pattern is the same alternation.
- Program
Write a function computing ∑ᵢ₋₁ⁿ iᵏ mod p for n ≤ 10¹⁸ and k ≤ 100. State the method.
Solution
Sᵏ(n) = ∑ iᵏ is a polynomial in n of degree k+1 (Faulhaber). So evaluate it at k+2 points and use Lagrange interpolation on consecutive points, which is O(k) after O(k) precomputation.
def power_sum(n, k, p): m = k + 2 y, s = [], 0 for i in range(1, m+1): s = (s + pow(i, k, p)) % p; y.append(s) # S_k(1..m) if n <= m: return y[n-1] pre = [1]*(m+1); suf = [1]*(m+2) for i in range(1, m+1): pre[i] = pre[i-1] * ((n - i) % p) % p for i in range(m, 0, -1): suf[i] = suf[i+1] * ((n - i) % p) % p fact = [1]*(m+1) for i in range(1, m+1): fact[i] = fact[i-1]*i % p res = 0 for i in range(1, m+1): num = pre[i-1] * suf[i+1] % p den = fact[i-1] * fact[m-i] % p term = y[i-1] * num % p * pow(den, p-2, p) % p if (m - i) % 2: term = p - term res = (res + term) % p return resO(k log p) per query. The mathematical content is that a degree-(k+1) polynomial is determined by k+2 values — Chapter 52 develops the interpolation; here it is the tool that makes an n = 10¹⁸ summation instant.
- Program
Analyse and implement: a dynamic array that grows by a factor g > 1 when full. Empirically measure total copy work for g = 1.1, 1.5, 2, 4 and compare to the predicted n·g/(g-1).
Solution
def total_copies(n, g): cap, copies = 1, 0 for i in range(n): if i == cap: copies += cap # copy the whole old array cap = max(cap + 1, int(cap * g)) return copies for g in (1.1, 1.5, 2.0, 4.0): n = 10**6 print(g, total_copies(n, g), n * g / (g - 1))Derivation: the copies form a geometric series ending at n with ratio 1/g, so the total is n(1 + 1/g + 1/g² + …) = n·g/(g-1). Amortised cost per push is therefore g/(g-1) copies — 2 for g = 2, 3 for g = 1.5, 11 for g = 1.1, and 1.33 for g = 4.
The trade-off is memory: growth factor g wastes up to (g-1)/g of the allocation. That is why
std::vectoruses 2 (fast, up to 50% waste) while some allocators prefer 1.5 — a factor below 2 allows reusing previously freed blocks, since 1+g > g² only when g < φ ≈ 1.618. Both choices are defensible and the summation is what makes the trade-off explicit.
Solving recurrences: trees, master theorem, Akra-Bazzi
A recursive algorithm's cost is a recurrence, and there are exactly three ways to solve one: draw the tree, apply the master theorem, or guess and prove by induction. Knowing which regime you are in — leaf-heavy, root-heavy, or balanced — answers most cases in ten seconds.
A divide-and-conquer algorithm that makes a recursive calls on inputs of size n/b and does f(n) work outside the calls has cost
Everything follows from comparing the work at the top of the recursion tree to the work at the leaves. There are log_b n levels, level i has aᵢ nodes each doing f(n/bᵢ) work, and the leaf count is a^{log_b n} = n^{log_b a}. Write c = log_b a — the critical exponent.
The recursion tree, which is the real method
level 0 1 node × n = n
level 1 2 nodes × n/2 = n
level 2 4 nodes × n/4 = n
...
level log n n nodes × 1 = n
─────
log n levels × n = Θ(n log n)
Every level costs the same → BALANCED case, multiply by the depth.T(n) = 2T(n/2) + n² ROOT-HEAVY n² + n²/2 + n²/4 + ... = Θ(n²)
geometric, top dominates
T(n) = 4T(n/2) + n LEAF-HEAVY n + 2n + 4n + ... + n² = Θ(n²)
geometric, bottom dominates
(leaves = n^{log₂4} = n²)Those three pictures — flat, decreasing, increasing — are the entire content of the master theorem. Draw two levels and the leaf count, see which way the series runs, and you have the answer.
The master theorem
| Recurrence | c = log_b a | Case | Solution | Algorithm |
|---|---|---|---|---|
| 2T(n/2)+n | 1 | 2 | Θ(n log n) | mergesort |
| 2T(n/2)+1 | 1 | 1 | Θ(n) | tree traversal |
| T(n/2)+1 | 0 | 2 | Θ(log n) | binary search |
| T(n/2)+n | 0 | 3 | Θ(n) | quickselect (balanced) |
| 8T(n/2)+n² | 3 | 1 | Θ(n³) | naive matrix mult |
| 7T(n/2)+n² | 2.807 | 1 | Θ(n^{2.807}) | Strassen |
| 3T(n/2)+n | 1.585 | 1 | Θ(n^{1.585}) | Karatsuba |
| 2T(n/2)+n log n | 1 | 2 (k=1) | Θ(n log² n) | closest pair (naive merge) |
The gaps in the master theorem are real. T(n) = 2T(n/2) + n/log n falls between cases 1 and 2 — n/log n is not O(n^{1-ε}) and not Θ(n logᵏ n) — and the true answer is Θ(n log log n), obtainable only by summing the tree. Case 3 also requires the regularity condition, which real recurrences almost always satisfy but pathological ones do not.
Uneven splits and Akra–Bazzi
When the pieces have different sizes, the master theorem does not apply. Akra–Bazzi handles T(n) = ∑ aᵢT(n/bᵢ) + f(n): find p with ∑ aᵢ/bᵢ^p = 1, then T(n) = Θ(n^p (1 + ∫₁ⁿ f(u)/u^{p+1} du)).
For the median-of-medians selection recurrence T(n) = T(n/5) + T(7n/10) + n: since 1/5 + 7/10 = 9/10 < 1, we get p = 1 and the integral ∫ du/u² converges, giving T(n) = Θ(n). The intuition without machinery: the subproblem sizes shrink by a constant factor 9/10 per level, so the total is a geometric series n(1 + 0.9 + 0.81 + …) = 10n.
That intuition is worth more than the formula. If the child sizes sum to less than n, the recursion is root-heavy and T = Θ(f(n)). If they sum to exactly n, it is balanced and you pay a log. If they sum to more, it is leaf-heavy.
Substitution: guess and prove
For anything the theorems miss, guess a form and verify by induction — including the constants, which is where it goes wrong. To prove T(n) = 2T(⌊n/2⌋) + n is O(n log n), assume T(k) ≤ ck log k for k < n:
The classic failure is proving T(n) = O(n) for the same recurrence by writing T(n) ≤ 2c(n/2) + n = cn + n = O(n). That last step is wrong: you must reach ≤ cn exactly, not cn + n, because the induction requires the same constant at every level. Sloppiness about constants is how n log n gets "proved" linear.
When you propose a divide-and-conquer, state the recurrence before the complexity: "T(n) = 2T(n/2) + O(n) for the merge, so Θ(n log n)." It shows the analysis came from the structure rather than recall. If asked to improve it, the recurrence tells you where: reduce a (Strassen, Karatsuba), increase b, or reduce f.
- Theory
Solve by the master theorem: (a) T(n)=3T(n/2)+n², (b) T(n)=4T(n/2)+n², (c) T(n)=5T(n/2)+n².
Solution
All have b = 2 and f = n²; only c = log₂a changes.
- (a) c = log₂3 ≈ 1.585 < 2: root-heavy, Case 3, Θ(n²).
- (b) c = 2: balanced, Case 2 with k = 0, Θ(n² log n).
- (c) c = log₂5 ≈ 2.322 > 2: leaf-heavy, Case 1, Θ(n^{log₂5}).
The three cases from one parameter change — this triple is the cleanest illustration that the answer is decided by comparing f(n) with n^{log_b a} and nothing else.
- Theory
Solve T(n) = 2T(n/2) + n/log n by summing the recursion tree, and explain why the master theorem fails.
Solution
Level i has 2ᵢ nodes of size n/2ᵢ, each costing (n/2ᵢ)/log(n/2ᵢ). Level total: n/(log n - i). Summing i = 0..log n - 1:
∑ᵢ n/(log n - i) = n ∑₌₋₁^{log n} 1/j = n H_{log n} = Θ(n log log n).
The master theorem fails because n/log n is polynomially smaller than n^{c} = n by no fixed ε — for any ε > 0, n/log n is not O(n^{1-ε}) — so Case 1 does not apply, and it is not Θ(n logᵏ n) for k ≥ 0 so Case 2 does not either. This is the standard example of the gap.
- Theory
Solve T(n) = T(n-1) + n and T(n) = T(n-1) + 1/n and T(n) = 2T(n-1) + 1.
Solution
These are subtract-and-conquer, not divide-and-conquer, so the master theorem is inapplicable; unroll instead.
- T(n) = ∑ᵢ₋₁ⁿ i = Θ(n²) — insertion sort, quicksort worst case.
- T(n) = Hₙ = Θ(log n) — surprisingly slow-growing.
- T(n) = 2ⁿ - 1 = Θ(2ⁿ) — Towers of Hanoi. Unroll: T(n)+1 = 2(T(n-1)+1), so T(n)+1 = 2ⁿ(T(0)+1).
The general rule for T(n) = aT(n-1) + f(n): it is Θ(aⁿ) when a > 1, and Θ(∑ f) when a = 1.
- Theory
Prove by substitution that T(n) = 2T(⌊n/2⌋) + n is O(n log n), handling the floor rigorously.
Solution
Claim: T(n) ≤ cn log₂n for n ≥ 2 and suitable c.
Step: T(n) ≤ 2c⌊n/2⌋log₂⌊n/2⌋ + n ≤ 2c(n/2)log₂(n/2) + n = cn(log₂n - 1) + n = cn log₂n - cn + n, which is ≤ cn log₂n provided c ≥ 1.
Base: n = 2 gives T(2) = 2T(1)+2, a constant; choose c large enough that c·2·1 ≥ T(2). Note n = 1 cannot be the base because c·1·log 1 = 0 — a real subtlety, resolved by starting the induction at n = 2 and treating n = 1 as part of the constant. ∎
The floor was handled by ⌊n/2⌋ ≤ n/2 and monotonicity of x log x, which is the standard move; the Akra–Bazzi theorem's tolerance for ±O(n/log²n) perturbations formalises why floors never change the answer.
- Theory
Find the flaw: "T(n) = 2T(n/2) + n. Assume T(k) ≤ ck. Then T(n) ≤ 2c(n/2) + n = cn + n = O(n)."
Solution
The induction requires proving T(n) ≤ cn with the same c, and the derivation reaches cn + n, which exceeds cn for every n > 0 and every c. Writing "= O(n)" at the end hides the failure: the extra n per level accumulates over log n levels, which is exactly the log n factor the false proof lost.
The rule: in a substitution proof, the induction hypothesis must be re-established exactly, with no slack absorbed into asymptotic notation. If you need slack, strengthen the hypothesis (e.g. T(n) ≤ cn - d), which is Chapter 3's technique applied here.
- Theory
Analyse the median-of-medians recurrence T(n) = T(n/5) + T(7n/10) + O(n) and prove linearity by substitution.
Solution
Claim T(n) ≤ cn. Then T(n) ≤ c(n/5) + c(7n/10) + an = cn(1/5 + 7/10) + an = 0.9cn + an, and this is ≤ cn iff an ≤ 0.1cn, i.e. c ≥ 10a. ∎
The 7n/10 comes from the guarantee: the median of medians exceeds at least 3(n/10) elements and is exceeded by at least 3(n/10), so the recursive side has at most 7n/10. The reason group size 5 is chosen: with groups of 3 the recurrence becomes T(n/3)+T(2n/3)+O(n) whose child sizes sum to exactly n — balanced, hence Θ(n log n), not linear. The whole design hinges on 1/5 + 7/10 < 1.
- Theory
Karatsuba multiplies two n-digit numbers with 3 half-size multiplications. Write and solve the recurrence, and give the crossover reasoning against schoolbook.
Solution
T(n) = 3T(n/2) + O(n). Here c = log₂3 ≈ 1.585 > 1, so Case 1: T(n) = Θ(n^{1.585}).
The trick: with x = x₁B + x₀ and y = y₁B + y₀, compute z₂ = x₁y₁, z₀ = x₀y₀, and z₁ = (x₁+x₀)(y₁+y₀) - z₂ - z₀ — three multiplications instead of four, at the cost of extra additions.
Crossover: schoolbook is n² with a tiny constant; Karatsuba is ≈ 3n^{1.585} with recursion and allocation overhead. In practice GMP switches around 20–40 limbs (roughly 600–1200 bits), then to Toom-Cook, then to FFT-based multiplication above a few thousand limbs. Each transition is a crossover computed exactly as in Chapter 8 problem 15.
- Theory
Use Akra–Bazzi on T(n) = T(n/3) + T(2n/3) + n.
Solution
Solve (1/3)^p + (2/3)^p = 1. By inspection p = 1 works. Then
T(n) = Θ(n(1 + ∫₁ⁿ u/u² du)) = Θ(n(1 + ln n)) = Θ(n log n).
This matches the tree: the child sizes sum to exactly n, so every level costs n, and the depth is between log₃n and log_{3/2}n — both Θ(log n). It is the quicksort-with-a-guaranteed-1/3-split recurrence, and it explains why even a badly unbalanced but constant-ratio pivot still gives n log n.
- Program
Write a program that numerically solves an arbitrary recurrence T(n) = aT(n/b)+f(n) by memoised evaluation, then estimates the exponent by fitting log T vs log n. Verify against the master theorem on five examples.
Solution
from functools import lru_cache import math def solver(a, b, f): @lru_cache(None) def T(n): if n <= 1: return 1.0 return a * T(n // b) + f(n) return T def fit(T, lo=1<<10, hi=1<<22): xs, ys = [], [] n = lo while n <= hi: xs.append(math.log(n)); ys.append(math.log(T(n))); n *= 2 mx, my = sum(xs)/len(xs), sum(ys)/len(ys) return (sum((x-mx)*(y-my) for x,y in zip(xs,ys)) / sum((x-mx)**2 for x in xs)) print(fit(solver(2, 2, lambda n: n))) # ~1.06 (n log n) print(fit(solver(4, 2, lambda n: n))) # ~2.00 (n^2) print(fit(solver(7, 2, lambda n: n*n))) # ~2.81 (Strassen) print(fit(solver(3, 2, lambda n: n))) # ~1.58 (Karatsuba) print(fit(solver(1, 2, lambda n: 1))) # ~0.02 (log n)The fitted exponent matches log_b a in the leaf-heavy cases exactly; in balanced cases it comes out slightly above the polynomial exponent, because a log n factor looks like a small positive exponent over a bounded range — the same limitation as Chapter 8's empirical fit, and a good reminder that the theorem tells you what measurement cannot.
- Program
Implement Strassen's matrix multiplication and measure the crossover against the naive O(n³) algorithm.
Solution
def strassen(A, B, cutoff=64): n = len(A) if n <= cutoff: return naive(A, B) m = n // 2 a11,a12,a21,a22 = split(A, m); b11,b12,b21,b22 = split(B, m) M1 = strassen(add(a11,a22), add(b11,b22), cutoff) M2 = strassen(add(a21,a22), b11, cutoff) M3 = strassen(a11, sub(b12,b22), cutoff) M4 = strassen(a22, sub(b21,b11), cutoff) M5 = strassen(add(a11,a12), b22, cutoff) M6 = strassen(sub(a21,a11), add(b11,b12), cutoff) M7 = strassen(sub(a12,a22), add(b21,b22), cutoff) c11 = add(sub(add(M1,M4),M5), M7) c12 = add(M3, M5) c21 = add(M2, M4) c22 = add(sub(add(M1,M3),M2), M6) return join(c11,c12,c21,c22)7 multiplications instead of 8, at the cost of 18 additions instead of 4 — so T(n) = 7T(n/2) + Θ(n²) gives Θ(n^{2.807}). In practice the crossover is around n = 512–1024 for a careful implementation, because the naive version's inner loop is dense, contiguous and vectorisable while Strassen's additions destroy locality and allocate. It is also numerically less stable for floating point — the subtractions cancel — which is why BLAS libraries do not use it by default.
- Program
Implement quickselect and median-of-medians selection. Measure the constant factor difference and explain when the O(n) worst-case guarantee is worth paying for.
Solution
def quickselect(a, k): # expected O(n), worst O(n^2) import random while True: if len(a) == 1: return a[0] p = random.choice(a) lo = [x for x in a if x < p] eq = [x for x in a if x == p] hi = [x for x in a if x > p] if k < len(lo): a = lo elif k < len(lo)+len(eq): return p else: k -= len(lo)+len(eq); a = hiExpected comparisons for quickselect: ≈ 3.4n. Median-of-medians: ≈ 24n in a straightforward implementation — a 5–10× constant, which is why introselect (quickselect, falling back to median-of-medians only after too many bad pivots) is what
std::nth_elementactually does.Pay for the guarantee when the input is adversarial (a service exposed to untrusted data, where a chosen input triggering Θ(n²) is a DoS) or when a hard latency bound matters. Otherwise randomisation gives the same expectation with a far better constant, and the worst case has probability n^{-Θ(1)}.
- Program
Implement the closest-pair-of-points divide and conquer in O(n log n) and state precisely why the strip check is O(1) per point.
Solution
def closest(pts): P = sorted(pts) # by x def rec(P): n = len(P) if n <= 3: return brute(P), sorted(P, key=lambda p: p[1]) m = n // 2; mid_x = P[m][0] dl, Ly = rec(P[:m]); dr, Ry = rec(P[m:]) d = min(dl, dr) Y = merge_by_y(Ly, Ry) # keeps the whole thing O(n log n) strip = [p for p in Y if abs(p[0]-mid_x) < d] for i in range(len(strip)): for j in range(i+1, min(i+8, len(strip))): # at most 7 ahead d = min(d, dist(strip[i], strip[j])) return d, Y return rec(P)[0]Why 7 suffices: consider the d × 2d rectangle above a strip point. Each half of it (left and right of the dividing line) is d × d and contains points pairwise at distance ≥ d, so at most 4 points fit in each — 8 total including the point itself, hence at most 7 others to check. That geometric packing bound is what makes the merge step linear and the whole recurrence T(n) = 2T(n/2)+O(n) = Θ(n log n).
Sorting by y inside the recursion instead of merging gives T(n) = 2T(n/2)+O(n log n) = Θ(n log²n) — the Case 2 with k = 1 row of the chapter's table, and a good illustration that a log in f(n) costs you a log overall.
- Program
Write a tool that draws the recursion tree of a given recurrence: level, node count, per-node cost, level total. Use it to identify visually which master case applies.
Solution
def tree(a, b, f, n): print(f"{'lvl':>4} {'nodes':>12} {'size':>12} {'per-node':>14} {'total':>14}") lvl, nodes, size, grand = 0, 1, n, 0 while size >= 1: per = f(size); tot = nodes * per; grand += tot print(f"{lvl:>4} {nodes:>12} {size:>12} {per:>14.1f} {tot:>14.1f}") lvl += 1; nodes *= a; size //= b print("leaves:", nodes, " grand total:", grand) tree(2, 2, lambda s: s, 1<<10) # flat column → balanced → n log n tree(2, 2, lambda s: s*s, 1<<10) # decreasing → root-heavy → f(n) tree(4, 2, lambda s: s, 1<<10) # increasing → leaf-heavy → n^{log_b a}The "total" column is the whole diagnosis: constant means multiply by depth, geometrically decreasing means take the first row, geometrically increasing means take the last. Printing it for an unfamiliar recurrence is faster and more reliable than trying to recall which master case has which inequality direction.
- Program
Given T(n) = T(n-1) + T(n-2)-style tree recursion (naive Fibonacci), count the exact number of calls, then fix it with memoisation and with an iterative loop. Quantify all three.
Solution
calls = 0 def fib_naive(n): global calls; calls += 1 return n if n < 2 else fib_naive(n-1) + fib_naive(n-2)Exact call count: C(n) = 2Fₙ₊₁ - 1. So computing F₄₀ makes 331,160,281 calls — about Θ(φⁿ) with φ = 1.618. Note the base of the exponential is φ, not 2, because the tree is lopsided; this is the general solution of T(n)=T(n-1)+T(n-2)+O(1).
Memoisation makes it Θ(n) calls and Θ(n) space; the iterative two-variable loop is Θ(n) time and Θ(1) space; matrix exponentiation (Chapter 48) is Θ(log n). Four algorithms for one function, differing by φⁿ → n → log n — the clearest demonstration that a recurrence's shape, not its formula, determines cost.
- Program
Implement an iterative-deepening search whose cost is ∑d=0..D bᵈ and demonstrate that the repeated work is a constant factor, not a blow-up.
Solution
def ids(start, goal, succ, max_depth): for depth in range(max_depth + 1): found = dls(start, goal, succ, depth) # depth-limited DFS if found is not None: return found, depth return NoneCost: ∑d=0..D bᵈ = (b^{D+1}-1)/(b-1), versus bᵈ for a single depth-D search. The ratio is b/(b-1): 2× for b = 2, 1.5× for b = 3, 1.1× for b = 11. A geometric series dominated by its largest term, exactly the Chapter 9 principle.
So iterative deepening gets BFS's optimality and DFS's O(D) memory for a constant-factor time penalty that shrinks as the branching factor grows. That is why it is the standard choice for game-tree search, where b is large and memory is the binding constraint — and it is a case where an analysis that looks wasteful is refuted by summing the series properly.
Logarithms, harmonic numbers and Stirling's approximation
Three tools for turning products into sums, estimating factorials, and recognising the log that appears whenever something halves or when you sum reciprocals. They show up in nearly every analysis, and estimating them mentally to one significant figure is a genuinely useful skill.
Logarithms convert multiplication into addition, and that is why they appear wherever a quantity halves, doubles, or is measured in bits. The identities:
The last row is worth memorising outright: every factor of 1000 adds about 10 to log₂. That single fact lets you estimate any logarithmic cost in your head — a balanced tree over a billion keys is 30 levels deep, a binary search over 10¹⁸ takes 60 steps.
Inside Θ, the base of a logarithm is irrelevant (log_a n = log_b n / log_b a, a constant factor). Inside an exponent it is not: 2^{log₂n} = n but 2^{log₃n} = n^{0.63}. Confusing these is a common error, and the fix is always to rewrite a^{log_b c} as c^{log_b a} where the exponent is a plain constant.
Harmonic numbers
The harmonic series diverges — famously proved by the blocking argument of Chapter 9 — but so slowly that it is effectively a logarithm for every practical purpose. It appears in:
- Coupon collector: expected draws to see all n coupons is nHₙ ≈ n ln n.
- Quicksort: expected comparisons ≈ 2n ln n ≈ 1.39 n log₂n — the constant 2 ln 2 comes straight from Hₙ.
- Sieve loops: ∑ n/i = nHₙ.
- Random BST depth: expected depth ≈ 2 ln n ≈ 1.39 log₂n, only 39% worse than a perfectly balanced tree.
- Records in a random sequence: expected number of new maxima is exactly Hₙ.
Whenever an analysis produces "the i-th item has probability 1/i of something", the answer is Hₙ = Θ(log n) by linearity of expectation. Recognising that pattern — the 1/i term — is faster than deriving the sum, and it covers a surprising number of randomised-algorithm analyses.
Stirling's approximation
Stirling is what makes factorials tractable. Two immediate consequences:
The comparison-sort lower bound. A decision tree distinguishing n! permutations needs depth ≥ log₂(n!) = n log₂n - 1.44n + O(log n). So no comparison sort beats n log₂n comparisons, and the -1.44n term says the achievable constant is 1, not something smaller.
The central binomial coefficient. C(2n,n) ≈ 4ⁿ/√(πn). This is why a random walk returns to the origin with probability Θ(1/√n), why the number of balanced bracket sequences (Catalan, Chapter 32) is Θ(4ⁿ/n^{1.5}), and why the binomial distribution's peak has height Θ(1/√n) — the √n in all three is the same √(2πn) from Stirling.
# log-space: never compute n! directly for large n
import math
def log_fact(n): return math.lgamma(n + 1) # exact, no overflow
def log_binom(n, k): return log_fact(n) - log_fact(k) - log_fact(n-k)
def binom_float(n, k): return math.exp(log_binom(n, k))
# 100! has 158 digits; 1000! has 2568. Digit count:
def digits(n): return int(log_fact(n) / math.log(10)) + 1
# Stirling by hand, order of magnitude:
# log10(n!) ≈ n log10(n/e) + 0.5 log10(2πn)
# log10(70!) ≈ 70·log10(25.75) + 1.3 ≈ 70·1.411 + 1.3 ≈ 100 → 70! ≈ 10^100"How many comparisons does sorting need?" expects Θ(n log n); the strong answer adds "at least log₂(n!) ≈ n log₂n - 1.44n by the decision-tree argument, so mergesort's n log₂n is within a lower-order term of optimal — and radix sort beats it only because it is not a comparison sort." Naming why the lower bound does not apply to radix sort is the part that lands.
- Theory
Without a calculator, estimate log₂(10⁹), log₂(3&sup5;⁰) and the number of decimal digits in 2¹⁰⁰₀.
Solution
- log₂10⁹ = 9 log₂10 ≈ 9 × 3.322 ≈ 29.9, so about 30.
- log₂3⁵⁰ = 50 log₂3 ≈ 50 × 1.585 = 79.2.
- Digits of 2¹⁰²⁴: 1024 × log₁₀2 = 1024 × 0.30103 ≈ 308.25, so 309 digits. (The famous 2¹⁰²⁴ overflow threshold in
doubleis exactly this.)
The two constants to keep: log₂10 ≈ 3.32 and log₁₀2 ≈ 0.301. Everything else follows.
- Theory
Prove a^{log_b c} = c^{log_b a} and use it to simplify n^{log₂3} versus 3^{log₂n}.
Solution
Take log_b of both sides: LHS gives log_b c · log_b a, RHS gives log_b a · log_b c. Equal, and log_b is injective on positives. ∎
So 3^{log₂n} = n^{log₂3} = n^{1.585} — they are the same function, which surprises people. This identity is how you recognise Karatsuba's n^{log₂3} and a recursion producing 3^{depth} as identical growth rates.
- Theory
Prove Hₙ = ln n + γ + O(1/n) at least to the level of ln n + O(1), using the integral comparison.
Solution
Since 1/x is decreasing, for each i ≥ 2: ∫ᵢ^{i+1} dx/x ≤ 1/i ≤ ∫_{i-1}^{i} dx/x. Summing i = 1..n:
ln(n+1) = ∫₁^{n+1} dx/x ≤ Hₙ ≤ 1 + ∫₁ⁿ dx/x = 1 + ln n.
So ln n ≤ Hₙ ≤ 1 + ln n, i.e. Hₙ - ln n ∈ [0,1]. That difference is decreasing and bounded below, hence convergent — its limit is γ ≈ 0.5772. ∎ The O(1/n) refinement comes from the Euler–Maclaurin formula.
- Theory
Compute the expected number of "record" values (new running maxima) in a uniformly random permutation of n distinct elements.
Solution
Let Xᵢ = 1 if position i is a record. Position i is a record iff the maximum of the first i elements sits at position i, which by symmetry has probability 1/i. By linearity of expectation (no independence needed):
E[records] = ∑ᵢ₋₁ⁿ 1/i = Hₙ ≈ ln n + γ.
For n = 10⁶ that is about 14.4 records — strikingly few. This is exactly the expected number of times a "best so far" variable is updated in a linear scan, and the same computation gives the expected number of times a randomised incremental algorithm rebuilds.
- Theory
Use Stirling to show C(2n,n) = Θ(4ⁿ/√n) and give the constant.
Solution
C(2n,n) = (2n)!/(n!)². Substituting Stirling:
≈ [√(4πn)(2n/e)^{2n}] / [√(2πn)(n/e)ⁿ]² = [√(4πn) · 4ⁿ n^{2n}e^{-2n}] / [2πn · n^{2n}e^{-2n}] = 4ⁿ/√(πn).
So the constant is 1/√π ≈ 0.564. ∎ Check at n = 10: C(20,10) = 184{,}756 and 4¹⁰/√(10π) = 1048576/5.60 = 187{,}200 — 1.3% high, as expected from the 1+1/(12n) correction terms.
- Theory
Show that a comparison sort needs at least n log₂n - 1.443n comparisons in the worst case.
Solution
A deterministic comparison sort is a binary decision tree: each internal node is a comparison with two outcomes, each leaf is an output permutation. To sort correctly it must have at least n! distinct leaves. A binary tree with L leaves has depth ≥ log₂L, and depth is the worst-case comparison count. So the worst case is at least
log₂(n!) = n log₂n - n log₂e + Θ(log n) = n log₂n - 1.4427n + Θ(log n). ∎
The bound is essentially achieved: mergesort uses ≤ n log₂n - n + 1 comparisons, within 0.44n of optimal. Counting sort and radix sort evade the bound entirely because they read the keys' values, not just comparison outcomes — the decision tree has more than two children per node.
- Theory
Expected comparisons in randomised quicksort is 2(n+1)Hₙ - 4n. Derive the leading term and express the constant relative to n log₂n.
Solution
Derivation. Consider elements of rank i and j, i < j. They are compared iff the first pivot chosen from the range [i..j] is one of them (otherwise they are separated). That probability is 2/(j-i+1). By linearity,
E = ∑ᵢ₂₃ 2/(j-i+1) = ∑k=2..n (n-k+1)·2/k = 2(n+1)Hₙ - 4n.
Leading term: 2n ln n = 2 ln 2 · n log₂n ≈ 1.386 n log₂n. So randomised quicksort uses about 39% more comparisons than the information-theoretic minimum — and still beats mergesort in practice because its inner loop is a tight in-place partition with excellent locality and no auxiliary array.
- Program
Implement Hₙ three ways for n = 10⁷: forward summation, backward summation, and the asymptotic formula. Compare accuracy against an exact rational computation.
Solution
import math from fractions import Fraction def fwd(n): return sum(1.0/i for i in range(1, n+1)) def bwd(n): return sum(1.0/i for i in range(n, 0, -1)) def asym(n): g = 0.5772156649015329 return math.log(n) + g + 1/(2*n) - 1/(12*n*n) + 1/(120*n**4) def exact(n): return float(sum(Fraction(1, i) for i in range(1, n+1)))Backward summation is more accurate. Adding smallest-to-largest keeps the running sum comparable in magnitude to each addend, so fewer low-order bits are lost; forward summation loses precision once the partial sum dwarfs 1/i. At n = 10⁷ forward is off by roughly 10⁻⁹ relative, backward by about 10⁻¹², and the asymptotic formula is accurate to 10⁻¹⁶ — better than either sum, and 10⁷ times faster. This is the Chapter 66 lesson arriving early: the numerically best algorithm here is not to sum at all.
- Program
Compute the number of trailing zeros in n! for n ≤ 10¹⁸, and the number of decimal digits.
Solution
def trailing_zeros(n): # Legendre's formula for the exponent of 5 c = 0 while n: n //= 5; c += n return c def digits(n): # via Stirling in log10 import math if n < 2: return 1 return int(n*math.log10(n/math.e) + 0.5*math.log10(2*math.pi*n)) + 1Trailing zeros count factors of 10 = 2·5, and 5s are scarcer, so the answer is ∑i≥1 ⌊n/5ᵢ⌋ — Legendre's formula, O(log₅n). For n = 10¹⁸ that is about 2.5·10¹⁷ zeros.
The digit count uses Stirling in base 10. Beware precision: at n = 10¹⁸ the product n log₁₀(n/e) is about 1.7·10¹⁹, and a
doublehas only ~15–16 significant digits, so the answer's last digits are unreliable. Use exact integer arithmetic on a scaled logarithm, or accept the approximation and say so. - Program
Simulate the coupon collector for n = 1000 over 10,000 trials and compare the empirical mean and variance to nHₙ and n²π²/6.
Solution
import random, math def trial(n): seen, draws = set(), 0 while len(seen) < n: seen.add(random.randrange(n)); draws += 1 return draws n, T = 1000, 10000 xs = [trial(n) for _ in range(T)] mean = sum(xs)/T var = sum((x-mean)**2 for x in xs)/T H = sum(1.0/i for i in range(1, n+1)) print(mean, n*H) # ≈ 7485 both print(var, n*n*math.pi**2/6) # ≈ 1.64e6 bothWhy: the wait to go from i-1 to i distinct coupons is geometric with success probability (n-i+1)/n, so mean n/(n-i+1) and the total is n Hₙ by linearity. Variances add because the waits are independent, giving ∑ n²(1-p)/p² → n²π²/6 using ∑1/k² = π²/6.
The standard deviation is nπ/√6 ≈ 1283 — 17% of the mean, so individual runs vary a lot. That spread is why cache-warming and randomised-sampling estimates need many trials, and it is directly the π²/6 from Chapter 9's table.
- Program
Verify empirically that a random BST built from n random insertions has expected depth ≈ 4.31 ln n and average node depth ≈ 2 ln n.
Solution
import random, math def build_depths(n): root = None; depth = {} tree = {} for x in random.sample(range(n*10), n): d, cur = 0, root while cur is not None: d += 1 cur = tree[cur][0] if x < cur else tree[cur][1] ... # standard BST insert, recording d return maxdepth, avgdepthExpected results at n = 10⁵: average node depth ≈ 2 ln n ≈ 23, maximum depth ≈ 4.311 ln n ≈ 50. A perfectly balanced tree would have depth log₂n ≈ 17.
The average-depth result is the quicksort analysis in disguise: the comparisons made inserting all n keys are exactly the comparisons quicksort makes, because the BST's shape is quicksort's recursion tree with the same pivots. The maximum-depth constant 4.311 (the root of c ln(2e/c) = 1) is much harder and comes from a branching-process argument — worth knowing it is not 2 ln n, since the maximum and the average diverge by more than a factor of 2.
- Program
Write a function that computes C(n,k) mod p for n ≤ 10⁶ in O(1) after preprocessing, and separately a floating-point C(n,k) for n ≤ 10⁹ that does not overflow.
Solution
# exact mod p, O(n) preprocessing then O(1) per query MOD = 10**9 + 7; N = 10**6 + 1 f = [1]*N for i in range(1, N): f[i] = f[i-1]*i % MOD inv = [1]*N inv[N-1] = pow(f[N-1], MOD-2, MOD) # Fermat, Chapter 20 for i in range(N-1, 0, -1): inv[i-1] = inv[i]*i % MOD def C(n, k): return 0 if k<0 or k>n else f[n]*inv[k]%MOD*inv[n-k]%MOD # floating point, no overflow, for huge n import math def C_float(n, k): return math.exp(math.lgamma(n+1) - math.lgamma(k+1) - math.lgamma(n-k+1))The batched inverse trick — computing
inv[N-1]once with one modular exponentiation and then walking down withinv[i-1] = inv[i]*i— turns O(n log p) into O(n + log p), and is worth knowing verbatim.The float version works to n = 10⁹ and beyond because
lgammastays in log space; computing n! directly overflowsdoubleat n = 171. Accuracy is about 13 significant digits, adequate for probabilities, useless for exact counting — which is why both versions exist. - Program
Implement a fast integer ⌊log₂n⌋ and ⌈log₂n⌉ using bit tricks, and verify against a floating-point version for all n ≤ 10⁷. Find where the float version is wrong.
Solution
int floor_log2(uint64_t n){ return 63 - __builtin_clzll(n); } // n > 0 int ceil_log2 (uint64_t n){ return n <= 1 ? 0 : 64 - __builtin_clzll(n-1); } // the buggy float version: int bad(uint64_t n){ return (int)(log2((double)n)); }The float version fails at exact powers of two for large n:
log2(2^49)can evaluate to48.999999999999993, and truncation gives 48 instead of 49. It also fails for n > 2⁵³ where thedoublecannot represent n exactly. Testing all n ≤ 10⁷ may show no failures — the bug typically first appears around 2⁴⁵–2⁵⁰, which is precisely why it survives testing and breaks in production.Use the bit-count intrinsic (
__builtin_clzll, Java'sLong.numberOfLeadingZeros, Python'sint.bit_length()). It is exact, branchless and faster. - Program
Given n up to 10⁵, compute ∑ᵢ₋₁ⁿ ⌊log₂i⌋ in O(log n) and explain the connection to the total bits needed to store 1..n.
Solution
Exactly 2ᵏ integers in [1,n] have ⌊log₂i⌋ = k (namely 2ᵏ..2^{k+1}-1), truncated at n.
def sum_floor_log2(n): total, k, lo = 0, 0, 1 while lo <= n: hi = min(n, 2*lo - 1) total += k * (hi - lo + 1) k += 1; lo *= 2 return totalO(log n) iterations. The value is ≈ n log₂n - 2n, consistent with log₂(n!) ≈ n log₂n - 1.44n since ⌊log₂i⌋ undercounts log₂i by about 0.5 on average.
Interpretation: ⌊log₂i⌋+1 is the bit length of i, so this sum plus n is the total bits to store all integers from 1 to n — the size of a variable-length encoding, and the reason a delta-encoded sorted list of n values from a universe of size n compresses to O(n log(U/n)) bits rather than n log U.
Amortized analysis: aggregate, accounting and potential
Some operations are occasionally expensive but cheap on average over any sequence — and "on average" here means guaranteed, not probabilistic. Three methods formalise it, and the potential method is the one that generalises.
Amortized analysis bounds the total cost of a sequence of operations, then divides. It is not average-case analysis: there is no probability distribution and no randomness. The bound holds for every sequence, including the worst one an adversary can construct. That distinction is the thing to say out loud.
| Worst case per op | Amortized | Average case | |
|---|---|---|---|
| Guarantee | every op | every sequence | expectation over inputs |
| Adversary-proof | yes | yes | no |
| Example | balanced BST insert | vector push_back | hash table lookup |
Aggregate method
Bound the total cost of n operations directly, then divide by n. For a dynamic array that doubles when full, the copies happen at sizes 1, 2, 4, …, 2^{⌊log n⌋}, totalling < 2n. Plus n cheap appends, the total is < 3n, so the amortized cost per push is O(1). That is the entire argument, and it is a geometric series (Chapter 9) doing the work.
Accounting method
Charge each operation an amortized cost that may exceed its actual cost; the surplus is stored as credit on data-structure elements and spent on later expensive operations. The obligation is to prove the credit balance never goes negative.
Charge 3 units per push.
1 unit pays for writing the new element.
1 unit is stored on the new element, to pay for copying it later.
1 unit is stored on an element in the first half of the array,
to pay for copying that (uncredited) element.
When the array of size k doubles: the k/2 elements added since the last
resize each carry 1 credit for themselves and 1 for an older element,
so all k copies are paid for. Balance never goes negative.
Hence amortized O(1) per push. ∎The choice of "3" is not magic — you pick a constant and check the invariant holds. If it does not, increase it; if no constant works, the operation is not O(1) amortized.
Potential method
Define Φ mapping each state to a real number, with Φ(D₀) = 0 and Φ(Dᵢ) ≥ 0 always. The amortized cost of operation i is
The art is choosing Φ so that expensive operations release stored potential. For the dynamic array, take Φ = 2·(size) - (capacity). A cheap push: cost 1, potential rises by 2, so ĉ = 3. A push triggering a resize from capacity k to 2k with size going k → k+1: cost k+1, and ΔΦ = [2(k+1)-2k] - [2k-k] = 2 - k, so ĉ = (k+1)+(2-k) = 3. Constant either way. ∎
The potential function measures how far the structure is from its cheapest configuration — how much "mess" is stored up. Full array, unbalanced tree, long chain of same-rank nodes: all are high potential. If you can name what makes an operation expensive, that quantity is usually the potential.
The standard results
| Structure / operation | Amortized | Potential used |
|---|---|---|
| Dynamic array push | O(1) | 2·size - capacity |
| Binary counter increment | O(1) | number of 1 bits |
| Two-stack queue | O(1) | size of the input stack |
| Splay tree operations | O(log n) | ∑ log(subtree size) |
| Fibonacci heap decrease-key | O(1) | #trees + 2·#marked |
| Union-find with both heuristics | O(α(n)) | rank-based, intricate |
| Move-to-front list | 2× optimal | inversions vs. optimal order |
The binary counter is the cleanest example: incrementing flips a trailing run of 1s to 0 and one 0 to 1. A single increment can cost Θ(log n), but with Φ = #(1 bits), an increment that flips k ones costs k+1 actual and changes potential by 1-k, giving ĉ = 2. Over n increments the total bit flips are < 2n, which you can also see by noting bit i flips every 2ᵢ increments: ∑ n/2ᵢ < 2n.
Where amortization breaks
Amortized bounds are about totals, so they fail when you need per-operation guarantees. A real-time system cannot accept a 100 ms pause even if the average is 1 μs — which is why real-time garbage collectors and deamortized data structures exist, spreading the expensive work across many operations at a constant-factor cost.
They also fail under persistence: if an old version can be reused repeatedly, an adversary can trigger the expensive operation from the same high-potential state many times, and the credit was only ever saved once. Amortized bounds for functional data structures need the stronger machinery of debits and lazy evaluation.
"What is the complexity of ArrayList.add?" — O(1) amortized, O(n) worst case for a single call, and the total for n adds is Θ(n). Adding "so a latency-sensitive path should presize the list" converts a definition question into an engineering answer.
- Theory
Prove that n increments of a binary counter starting from 0 cost O(n) total bit flips, two different ways.
Solution
Aggregate: bit 0 flips every increment, bit 1 every 2, bit i every 2ᵢ. Total flips = ∑i≥0 ⌊n/2ᵢ⌋ < n ∑ 2^{-i} = 2n.
Potential: Φ = number of 1 bits, non-negative and 0 initially. An increment flipping k trailing 1s to 0 and one 0 to 1 costs k+1 with ΔΦ = 1-k, so ĉ = 2. Total amortized 2n ≥ total actual. ∎
Both give the constant 2, which is tight: the counter 2ᵏ-1 → 2ᵏ really does flip k+1 bits, it just happens rarely.
- Theory
A dynamic array doubles on overflow and halves when it drops below 1/2 full. Show this is not O(1) amortized, and give the standard fix.
Solution
Counterexample: fill to exactly capacity k. Then push (doubles to 2k, size k+1), pop (size k, now below half of 2k, halves back to k), push (doubles again)… Each operation costs Θ(k), so n operations cost Θ(nk) — no constant amortized bound. This is thrashing.
Fix: halve only when the array drops below 1/4 full, shrinking to half capacity. Then immediately after any resize the array is exactly half full, so Ω(k) operations must occur before the next resize in either direction. Formally, Φ = |2·size - capacity| works: it is 0 right after a resize and must reach capacity/2 before another triggers.
The general lesson: leave hysteresis between the grow and shrink thresholds. The same principle governs autoscaling policies and circuit breakers.
- Theory
Show that a queue implemented with two stacks has O(1) amortized enqueue and dequeue.
Solution
Enqueue pushes to stack
in. Dequeue pops fromout; ifoutis empty, move everything fromintooutfirst (reversing it), then pop.Take Φ = |in|. Enqueue: cost 1, ΔΦ = +1, so ĉ = 2. Dequeue without a transfer: cost 1, ΔΦ = 0, ĉ = 1. Dequeue with a transfer of k elements: cost k+1, ΔΦ = -k, so ĉ = 1. ∎
The intuition: every element is pushed to
inonce, moved once, and popped fromoutonce — exactly 3 stack operations over its lifetime, no matter the interleaving. - Theory
Explain why amortized is not average-case, with a concrete example where the two differ.
Solution
Hash table lookup is O(1) average-case and O(n) worst-case, and it has no useful amortized bound: an adversary who knows the hash function can make every key collide, so n lookups cost Θ(n²). The O(1) depends on a probabilistic assumption about the keys (or a randomly chosen hash function).
Dynamic array push is O(1) amortized with no assumptions whatsoever: for every sequence of n pushes, total cost is < 3n. There is nothing an adversary can do.
So amortized is the stronger guarantee. The confusion arises because both are stated as "O(1) per operation", and the difference only shows up under adversarial input — which is exactly when it matters, as the hash-flooding DoS attacks demonstrated.
- Theory
Give the potential function for a splay tree and state what it proves.
Solution
Let s(x) be the number of nodes in x's subtree and r(x) = log₂s(x) the rank. Then Φ = ∑x r(x).
The Access Lemma: splaying node x to the root has amortized cost ≤ 3(r(root) - r(x)) + 1 = O(log n). Summing over a sequence of m operations on an n-node tree gives O((m+n)log n) total.
Because Φ is defined purely by the tree shape and no balance information is stored, splay trees achieve O(log n) amortized with no bookkeeping fields — and additionally satisfy the static optimality theorem (within a constant of the best static BST for any access sequence) and the working-set property. Individual operations can still cost Θ(n), which is why splay trees are unsuitable for real-time systems.
- Theory
Prove that move-to-front is 2-competitive against any static list ordering.
Solution
Run MTF alongside a fixed optimal ordering OPT, and let Φ = the number of inversions between the two lists (pairs ordered differently).
Accessing an element at position k in MTF and k* in OPT: let x be the number of elements preceding it in both lists. Moving it to the front of MTF destroys k-1-x inversions and creates at most x, so ΔΦ ≤ x - (k-1-x) = 2x-k+1. Since those x elements precede the item in OPT too, x ≤ k*-1. Then
ĉ = k + ΔΦ ≤ k + 2x - k + 1 = 2x+1 ≤ 2k* - 1. ∎
Summing: MTF's total cost is at most twice OPT's. This is a competitive analysis result — amortized analysis against an adversary and a reference algorithm — and it is why LRU-style self-organising heuristics are defensible without any assumption about the access distribution.
- Theory
Why does a Fibonacci heap achieve O(1) amortized decrease-key, and what is the practical catch?
Solution
Φ = t + 2m where t is the number of trees in the root list and m the number of marked nodes. Decrease-key cuts the node from its parent (making a new tree, t{+}1) and cascades up while parents are marked — but each cascading cut unmarks a node, releasing 2 potential while costing 1, so the cascade pays for itself. Amortized O(1). The deferred consolidation happens in extract-min, which is O(log n) amortized.
This makes Dijkstra O(E + V log V) instead of O(E log V) — asymptotically better for dense graphs.
The catch: enormous constants and terrible cache behaviour. Each node needs parent, child, two sibling pointers, degree and a mark bit, and the structure is a pointer soup with no locality. In practice a binary heap or a pairing heap beats it at every realistic size, and Fibonacci heaps are essentially a theoretical result. Saying that is the mark of having actually used them.
- Program
Implement a dynamic array from scratch with an explicit copy counter. Empirically verify the amortized cost is under 3 for growth factor 2, and measure it for factors 1.5 and 4.
Solution
class Vec: def __init__(self, g=2.0): self.buf = [None]; self.n = 0; self.g = g; self.copies = 0 def push(self, x): if self.n == len(self.buf): new = [None] * max(self.n + 1, int(self.n * self.g)) for i in range(self.n): new[i] = self.buf[i] self.copies += self.n self.buf = new self.buf[self.n] = x; self.n += 1 for g in (1.5, 2.0, 4.0): v = Vec(g) for i in range(10**6): v.push(i) print(g, v.copies / 10**6, "predicted", 1/(g-1))Measured copies-per-push converge to 1/(g-1): 2.0 for g = 1.5, 1.0 for g = 2, 0.33 for g = 4. Total cost per push is that plus 1 for the write itself, so the "3" of the accounting argument is a comfortable over-estimate for g = 2 (the true figure is 2).
The trade-off is memory: worst-case waste is (g-1)/g of the buffer — 33% for g = 1.5, 50% for g = 2, 75% for g = 4. That is why growth factors above 2 are essentially never used.
- Program
Implement a queue from two stacks and a stack that supports
getMin()in O(1). Then combine them into a queue with O(1) amortizedgetMin().Solution
class MinStack: # each entry carries the min below it def __init__(self): self.s = [] def push(self, x): m = x if not self.s else min(x, self.s[-1][1]) self.s.append((x, m)) def pop(self): return self.s.pop()[0] def getmin(self): return self.s[-1][1] class MinQueue: # two min-stacks def __init__(self): self.i, self.o = MinStack(), MinStack() def push(self, x): self.i.push(x) def pop(self): if not self.o.s: while self.i.s: self.o.push(self.i.pop()) # note: reversal here is what makes the transferred order correct return self.o.pop() def getmin(self): if not self.i.s: return self.o.getmin() if not self.o.s: return self.i.getmin() return min(self.i.getmin(), self.o.getmin())Each element is pushed twice and popped twice over its lifetime, so all three operations are O(1) amortized;
getminis O(1) worst case. This structure solves the sliding-window-minimum problem in O(n), and is often easier to get right than the monotonic-deque version because there is no window bookkeeping. - Program
Implement a hash table with open addressing and resizing at load factor 0.75. Instrument it to record probe counts and demonstrate the amortized O(1) claim, then break it with adversarial keys.
Solution
class Table: def __init__(self): self.k = [None]*8; self.n = 0; self.probes = 0 def _idx(self, key): i = hash(key) & (len(self.k)-1) while self.k[i] is not None and self.k[i][0] != key: self.probes += 1; i = (i+1) & (len(self.k)-1) # linear probing return i def put(self, key, val): if (self.n+1) / len(self.k) > 0.75: self._grow() i = self._idx(key) if self.k[i] is None: self.n += 1 self.k[i] = (key, val)With random keys, the average probe count stabilises near (1)/(2)(1 + 1/(1-α)²) ≈ 8.5 at α = 0.75 for insertions — bounded, so amortized O(1) including rehashes (whose cost is a geometric series, exactly as for the dynamic array).
Breaking it: choose keys all hashing to the same bucket (easy for a known, unseeded hash —
hash(x) & 7collides for many small integers). Then insertion becomes Θ(n) each and the total is Θ(n²). The counted probes make the attack visible. The mitigations — a randomly seeded hash per process, or switching a long bucket to a tree as Java'sHashMapdoes at 8 entries — are worth naming. - Program
Implement union-find without path compression and without union by rank, and measure the degradation. Then add each heuristic separately.
Solution
def bench(n, use_rank, use_compress): p = list(range(n)); r = [0]*n; steps = 0 def find(x): nonlocal steps root = x while p[root] != root: root = p[root]; steps += 1 if use_compress: while p[x] != root: p[x], x = root, p[x] return root ...Typical asymptotics, all for m operations on n elements:
- Neither heuristic: O(mn) worst case — a path can degenerate to a chain of length n.
- Union by rank/size only: O(m log n) — trees have depth ≤ log₂n because a tree of height h has ≥ 2Ḥ nodes.
- Path compression only: O(m log n) amortized (in fact O(m log_{1+m/n} n)).
- Both: O(m α(n)) — inverse Ackermann, under 5 for any n that fits in the universe.
The measured path lengths make the difference concrete: at n = 10⁶ the naive version averages thousands of steps per find, both-heuristics averages under 2.
- Program
Write a "deamortized" dynamic array: instead of one big copy, migrate a few elements per operation so every push is O(1) worst case. State the cost.
Solution
class Deamortized: def __init__(self): self.cur = [None]*4; self.n = 0 self.old = None; self.moved = 0 def push(self, x): if self.old is None and self.n == len(self.cur): self.old, self.cur, self.moved = self.cur, [None]*(2*self.n), 0 if self.old is not None: # migrate a fixed quota for _ in range(3): if self.moved < len(self.old): self.cur[self.moved] = self.old[self.moved]; self.moved += 1 if self.moved == len(self.old): self.old = None self.cur[self.n] = x; self.n += 1Migrating 3 elements per push finishes the k-element copy within k/3 pushes, well before the new buffer of size 2k fills — so every push is O(1) worst case, not merely amortized. The costs are 2× memory during migration and a lookup that must check both buffers.
This is exactly what an incremental or concurrent garbage collector does, and what Redis does when rehashing (it keeps two tables and migrates a bucket per operation). Deamortization trades a constant factor and memory for the removal of latency spikes — the right trade whenever a p99 matters more than a mean.
- Program
Measure latency percentiles for
ArrayList-style pushes and show the amortized mean hides a heavy tail. Produce the p50, p99, p99.9 and max.Solution
import time lat = [] v = Vec(2.0) for i in range(10**6): t = time.perf_counter_ns(); v.push(i); lat.append(time.perf_counter_ns()-t) lat.sort() def pct(p): return lat[int(len(lat)*p)] print("p50", pct(.5), "p99", pct(.99), "p99.9", pct(.999), "max", lat[-1])Typical shape: p50 and p99 are both a few hundred nanoseconds, p99.9 is similar, and the max is 3–5 orders of magnitude larger — the final resize copying 10⁶ elements. Only about 20 operations out of a million are resizes, so they do not even reach p99.99.
The lesson for a service: an amortized bound says nothing about tail latency, and tail latency is what users experience. If a request can trigger the resize, one request in 10⁵ takes 10 ms. Presizing the collection, or deamortizing as in the previous problem, is the fix — and this measurement is how you justify it.
- Program
Implement a monotonic stack solving "next greater element" and prove its O(n) total cost by an amortized argument, then verify with an operation counter.
Solution
def next_greater(a): n = len(a); res = [-1]*n; st = []; ops = 0 for i in range(n): while st and a[st[-1]] < a[i]: res[st.pop()] = i; ops += 1 st.append(i); ops += 1 return res, opsAmortized argument: take Φ = |st|. Each iteration pushes exactly once (ΔΦ = +1, cost 1, ĉ = 2) and pops k times (cost k, ΔΦ = -k, ĉ = 0). Total amortized 2n, so total actual ≤ 2n = O(n). ∎ Equivalently: each index is pushed once and popped at most once.
The counter confirms
ops < 2nfor every input, including the worst-looking ones (a strictly decreasing array pushes n times and pops 0; a strictly increasing one pushes n and pops n-1). The innerwhileloop makes the code look quadratic, and this argument is precisely what tells you it is not — the single most common place amortized reasoning is needed in day-to-day algorithm work.
Lower bounds: decision trees, adversaries and information
Proving that no algorithm can do better is a different discipline from designing one. Three techniques — counting outputs, counting information, and playing adversary — cover almost every lower bound you will meet.
An upper bound is exhibited by an algorithm. A lower bound must quantify over all algorithms in a model, which is why it is harder and why the model must be stated precisely. "Sorting is Ω(n log n)" is false as stated — it is true for comparison-based sorting and false for radix sort. Every lower bound is a statement about a model.
Decision trees: counting outputs
Model the algorithm as a tree whose internal nodes are the queries it may make and whose leaves are the possible outputs. If there are L distinct outputs and each query has k outcomes, the tree has depth at least log_k L, and depth is the worst-case query count.
That last line is instructive: the counting bound gives log n, but the true answer is n-1 comparisons. Counting outputs is often too weak, and then you need an adversary.
Adversary arguments
Imagine an opponent who does not fix the input in advance but answers each query in whichever way keeps the most possibilities alive, subject only to remaining consistent with some valid input. If the adversary can survive T queries, no algorithm can finish in fewer.
Finding the minimum needs n-1 comparisons. Call an element "known non-minimal" once it has lost a comparison. Initially 0 elements are known non-minimal; the algorithm must finish with n-1 of them. Each comparison creates at most one new loser (comparing two already-known losers creates none). So at least n-1 comparisons are needed. ∎
Finding both min and max needs ⌈3n/2⌉ - 2. The adversary tracks each element's status: never compared (N), only won (W), only lost (L), both (B). A comparison of two N elements yields the most information — it converts two N's into a W and an L, and there are only ⌊n/2⌋ such comparisons available. Every other comparison advances the count by at most 1 toward the required total. Counting units of progress gives the bound, and it is achieved by pairing elements first.
The adversary is not choosing a hard input; it is refusing to commit. Any answer it gives must be consistent with at least one valid input, and the argument works because it maintains an invariant on the number of surviving possibilities. If you can define a progress measure that any single query can advance by at most c, you get a lower bound of (total progress needed)/c.
Information-theoretic bounds
An algorithm learning an answer from a set of N possibilities must extract log₂N bits. Each yes/no query supplies at most 1 bit; each comparison with three outcomes at most log₂3. This is the same argument as the decision tree, phrased in bits, and it generalises cleanly to randomised algorithms via Yao's principle: the expected cost of the best randomised algorithm on the worst input is at least the expected cost of the best deterministic algorithm on the worst input distribution. That converts a hard quantification over randomised algorithms into an easier one over distributions.
Ω(n log n) comparison sorting decision tree, n! outputs
Ω(n log n) element uniqueness algebraic decision tree
Ω(n) any problem reading all input trivial, but state it
Ω(log n) search in a sorted array decision tree, n+1 outputs
n−1 find the minimum adversary
⌈3n/2⌉−2 find min and max adversary
n + ⌈log n⌉−2 find the second largest adversary (tournament)
Ω(n²) matrix transpose, I/O bound external memory model
Ω(V + E) any graph traversal must touch each edge
Ω(2^{n/2}) subset sum (best known) no proof — this is a conjectureThe last line matters: for most interesting problems we have no super-linear lower bound. Nobody has proved that SAT requires more than linear time; P vs NP is exactly the statement that we cannot prove such things. Lower bounds are rare and precious, and the ones above are essentially the complete list you will use.
Reductions: borrowing a lower bound
If problem A reduces to problem B in O(f) time, then a lower bound of L on A gives a lower bound of L - O(f) on B. Element uniqueness is Ω(n log n), and it reduces in linear time to closest pair, to convex hull, and to computing the mode — so all three are Ω(n log n).
"Can you do better than O(n log n)?" is often a lower-bound question in disguise. The strong answer is: "Not with comparisons — the decision-tree bound is log₂(n!). But if the keys are integers in a bounded range, radix sort is O(n) because it reads the key bits directly and escapes the model." Naming the model and the escape hatch is the answer they want.
- Theory
Prove that any comparison-based algorithm to merge two sorted lists of length n each requires at least 2n-1 comparisons.
Solution
Adversary: interleave the two lists so the merged order alternates a₁ < b₁ < a₂ < b₂ < …. Every adjacent pair in the merged output comes from different lists, and the algorithm must determine the relative order of each of the 2n-1 adjacent pairs. If it skipped comparing some adjacent pair (x,y) directly, the adversary could swap them: no other comparison's outcome changes (they are adjacent, so nothing lies between), yet the correct output differs. So all 2n-1 adjacent pairs must be compared. ∎
The standard merge achieves exactly 2n-1 in the worst case, so the bound is tight — and it explains why merging cannot be sped up asymptotically, only constant-factor-wise.
- Theory
Prove that finding the second-largest element requires n + ⌈log₂n⌉ - 2 comparisons.
Solution
Upper bound: run a tournament. Finding the maximum takes n-1 comparisons. The second largest must have lost directly to the maximum, and the maximum played ⌈log₂n⌉ matches; the second largest is the best of those ⌈log₂n⌉ losers, found in ⌈log₂n⌉-1 more comparisons. Total n + ⌈log₂n⌉ - 2.
Lower bound (adversary): the adversary assigns each element a weight, starting at 1, and answers a comparison in favour of the heavier element (ties broken arbitrarily), then transfers the loser's weight to the winner. Weights only ever double at most, so after k comparisons the maximum's weight is at most 2ᵏ; to reach total weight n it needs k ≥ log₂n comparisons, so the maximum must win at least ⌈log₂n⌉ times. Every other element must lose at least once (n-1 comparisons), and the second largest must additionally be compared against the other ⌈log₂n⌉-1 losers-to-max. Combining gives the bound. ∎
- Theory
Show that any algorithm determining whether an n-element array contains a duplicate needs Ω(n log n) in the comparison model, and explain how hashing evades it.
Solution
Element uniqueness is Ω(n log n) in the algebraic decision tree model (Ben-Or's theorem), via a topological argument: the set of inputs with all distinct values has n! connected components, and a decision tree of depth d can distinguish at most 2^{O(d)} components — so d = Ω(log(n!)) = Ω(n log n).
Hashing evades it by leaving the model: it computes h(x) mod m, an operation not available to a comparison or algebraic decision tree, and it uses randomness. The result is O(n) expected — but with no worst-case guarantee, since an adversary who knows h can force collisions. There is no contradiction: the two algorithms live in different models with different guarantees, which is the point of always naming the model.
- Theory
What is the minimum number of comparisons to sort 5 elements? Justify both the lower and upper bound.
Solution
Lower: ⌈log₂(5!)⌉ = ⌈log₂120⌉ = ⌈6.907⌉ = 7.
Upper: 7 is achievable. Compare a{:}b and c{:}d (2), compare the winners (3) — say a < b, c < d, b < d, giving the chain a < b < d and c < d. Insert e into a < b < d by binary search (2 more, total 5), then insert c into the appropriate prefix, which has at most 3 candidates positions, by binary search (2 more, total 7). ∎
Note that the information-theoretic bound is achieved here, but it is not always: for n = 12 the bound gives 29 and the true minimum is 30. Merge-insertion (Ford–Johnson) is optimal for n ≤ 15, and the general problem of the exact minimum is open — a reminder that log₂(n!) is a bound, not a formula.
- Theory
State Yao's minimax principle and use it to sketch a randomised lower bound.
Solution
Yao's principle: for any distribution D over inputs, the expected cost of the best deterministic algorithm on D lower-bounds the worst-case expected cost of any randomised algorithm. It follows from the minimax theorem applied to the zero-sum game between algorithm designer and input chooser.
Sketch application — searching an unsorted array of n bits for a 1, promised exactly one exists. Take D = uniform over the n positions. Any deterministic algorithm probes positions in some fixed order and finds the 1 after (n+1)/2 probes in expectation. Hence every randomised algorithm has worst-case expected cost ≥ (n+1)/2 = Ω(n). ∎
The power of the principle is that it removes the quantifier over randomised algorithms — the hardest part — leaving only the choice of a hard distribution, which is a design task you can reason about.
- Theory
Prove that any data structure supporting insert and find-min must have insert + delete-min = Ω(log n) amortized, by reduction from sorting.
Solution
Given such a structure with insert cost I and delete-min cost D, sort n elements by inserting all and extracting all: total n(I+D). Since comparison sorting is Ω(n log n), we need n(I+D) = Ω(n log n), i.e. I + D = Ω(log n). ∎
This is why no structure achieves O(1) for both — and it explains the design space precisely: a binary heap splits the cost (O(log n), O(log n)), a Fibonacci heap moves it all to one side (O(1), O(log n)), and an unsorted array does (O(1), O(n)). The sum is the invariant. It also shows the bound does not apply to integer keys: van Emde Boas trees achieve O(log log U) for both, again by leaving the comparison model.
- Theory
Give a lower bound for the number of I/Os needed to sort N elements in the external-memory model with block size B and memory M.
Solution
The bound is Θ((N/B) log_{M/B}(N/B)) I/Os, achieved by (M/B)-way mergesort.
Intuition for the lower bound: reading one block of B elements into memory reveals at most log₂C(M,B) ≈ B log₂(M/B) bits of new ordering information (the interleaving of the block with what is already in memory). Sorting requires log₂(N!) ≈ N log₂N bits, so the number of I/Os is at least N log N / (B log(M/B)) = (N/B) log_{M/B} N.
Concretely with N = 10¹², B = 10⁶ (1 MB blocks), M = 10⁹ (1 GB): log_{1000}(10⁶) = 2, so two passes. This is why real external sorts are 2–3 passes and why the number is so insensitive to N — the base of the logarithm is huge.
- Program
Write a program that computes the exact minimum number of comparisons to sort n elements for n ≤ 5 by exhaustive search over decision trees, and compare to ⌈log₂(n!)⌉.
Solution
from itertools import permutations from functools import lru_cache def solve(n): perms = frozenset(permutations(range(n))) @lru_cache(None) def best(S): # min depth to distinguish the set S if len(S) <= 1: return 0 r = float('inf') for i in range(n): for j in range(i+1, n): A = frozenset(p for p in S if p.index(i) < p.index(j)) B = S - A if not A or not B: continue # useless comparison r = min(r, 1 + max(best(A), best(B))) return r return best(perms) for n in range(1, 6): import math print(n, solve(n), math.ceil(math.log2(math.factorial(n))))Results: n = 1,2,3,4,5 give 0, 1, 3, 5, 7 — matching ⌈log₂(n!)⌉ = 0, 1, 3, 5, 7 exactly. The first divergence is at n = 12 (30 vs 29), far beyond this search's reach, since the state space is the power set of n! permutations.
- Program
Implement an adversary for the "guess the number" game that answers higher/lower to maximise the number of remaining candidates, and verify that it forces ⌈log₂n⌉ guesses.
Solution
class Adversary: def __init__(self, n): self.lo, self.hi = 1, n # never commits to a value def ask(self, g): left = g - self.lo # candidates below g right = self.hi - g # candidates above g if g < self.lo or g > self.hi: return "wrong" if left >= right and left > 0: self.hi = g - 1; return "lower" if right > 0: self.lo = g + 1; return "higher" return "correct" # forced: only one leftThe adversary always keeps the larger side, so the candidate count drops by at most half per guess: from n it takes at least ⌈log₂n⌉ guesses to reach 1. Binary search achieves exactly that, so the bound is tight.
This is the cleanest demonstration of the technique: the adversary never picks a number, and yet every answer it has given is consistent with at least one number — so the player genuinely cannot have finished earlier.
- Program
Implement min-and-max in ⌈3n/2⌉-2 comparisons and verify the count empirically against the naive 2n-2.
Solution
def minmax(a): n = len(a); cmps = 0 if n == 0: return None i = 0 if n % 2: mn = mx = a[0]; i = 1 else: mn, mx = (a[0], a[1]) if a[0] < a[1] else (a[1], a[0]); cmps += 1; i = 2 while i + 1 < n: x, y = (a[i], a[i+1]) if a[i] < a[i+1] else (a[i+1], a[i]); cmps += 1 if x < mn: mn = x cmps += 1 if y > mx: mx = y cmps += 1 i += 2 return mn, mx, cmpsEach pair costs 3 comparisons (one within the pair, one against the running min, one against the running max) and handles 2 elements, giving 3n/2 versus the naive 2 per element. At n = 10⁶ that is 1.5M versus 2M comparisons — a 25% saving, and it exactly meets the adversary lower bound, so no further improvement is possible in this model.
- Program
Demonstrate that radix sort beats the comparison lower bound: sort 10⁷ 32-bit integers with LSD radix sort and with
std::sort, and explain why there is no contradiction.Solution
void radix(vector<uint32_t>& a){ vector<uint32_t> buf(a.size()); for (int shift = 0; shift < 32; shift += 8){ size_t cnt[257] = {0}; for (uint32_t v : a) cnt[((v >> shift) & 255) + 1]++; for (int i = 0; i < 256; i++) cnt[i+1] += cnt[i]; for (uint32_t v : a) buf[cnt[(v >> shift) & 255]++] = v; a.swap(buf); } }Four passes of counting sort, each O(n): total O(4n) versus O(n log₂n) = O(23n) comparisons for n = 10⁷. Radix typically wins by 2–4× in wall time.
No contradiction: radix sort never compares two elements. It reads the bits of each key and uses them as array indices, an operation outside the comparison model. Its cost is O(n · w/b) for w-bit keys and b-bit digits, which is linear in n but depends on the key width — for arbitrary-precision keys, w = Ω(log n) and the two bounds coincide.
- Program
Write a brute-force verifier that, for a small problem, enumerates all decision trees up to depth d and reports whether any solves it — i.e. computationally proves a lower bound.
Solution
def solvable_within(states, queries, d, answer): """states: set of possible inputs; queries: list of functions state -> outcome; answer: state -> required output.""" def rec(S, depth): if len({answer(s) for s in S}) <= 1: return True # decided if depth == 0: return False for q in queries: parts = {} for s in S: parts.setdefault(q(s), set()).add(s) if len(parts) < 2: continue # useless query if all(rec(P, depth-1) for P in parts.values()): return True return False return rec(frozenset(states), d) # Then: d* = min d such that solvable_within(..., d) is True # and "not solvable_within(..., d*-1)" IS the lower-bound proof.This is a genuine machine-checked lower bound for the finite instance: it exhausts every decision tree of depth d^*-1 and finds none that works. The limitation is combinatorial explosion — the recursion branches over every query at every node — so it is practical only for tiny instances (sorting 5 elements, weighing 12 coins). For those it is decisive, and it is how the exact comparison counts in the literature were originally established.
- Program
The 12-coin problem: one of 12 coins is counterfeit (heavier or lighter). Prove 3 weighings suffice and that 2 do not, and implement the solution.
Solution
Lower bound: there are 12 × 2 = 24 possible answers (which coin, and which direction). Each weighing has 3 outcomes, so 3ᵏ ≥ 24 requires k ≥ 3 since 3² = 9 < 24. ∎
Upper bound: 3 weighings give 3³ = 27 ≥ 24 outcomes, and a non-adaptive assignment achieves it. Label coins with distinct sequences from {L, R, off}³ chosen so that no sequence is the reverse of another and each weighing has 4 coins per side:
weighing 1: L: 1,2,3,4 R: 5,6,7,8 weighing 2: L: 1,2,5,9 R: 3,6,10,11 weighing 3: L: 2,4,6,9 R: 1,7,8,12 # the triple of outcomes (<, =, >)^3 maps uniquely to (coin, heavy/light)Note 3³ = 27 against 24 needs — only 3 slack outcomes, which is why the coin-to-pattern assignment is delicate and why 13 coins (26 answers) is the true maximum for 3 weighings when you may not use a known-good reference coin.
- Program
Empirically estimate the constant in the Ω(n log n) sorting bound: count comparisons for mergesort, quicksort and heapsort on random data, and compare each to log₂(n!).
Solution
import math, random def ratio(sorter, n, trials=20): tot = 0 for _ in range(trials): a = [random.random() for _ in range(n)] tot += sorter(a) # returns comparison count return tot/trials / math.log2(math.factorial(n)) n = 100000 print("mergesort", ratio(merge_count, n)) # ≈ 1.02 print("quicksort", ratio(quick_count, n)) # ≈ 1.39 print("heapsort", ratio(heap_count, n)) # ≈ 2.0Mergesort is within 2% of the information-theoretic minimum — essentially optimal in comparisons. Quicksort uses 39% more (the 2 ln 2 constant of Chapter 11) but wins in wall time on locality. Heapsort uses about twice the minimum and has poor cache behaviour, which is why it is used as introsort's fallback rather than its main algorithm.
The experiment makes the lower bound tangible: it is not an abstraction but a floor that real algorithms sit within a small constant of, and the differences between them are entirely in that constant.
Divisibility, gcd and the Euclidean algorithm
The gcd is computed by a five-line algorithm from 300 BC that is still asymptotically optimal, and it is the foundation for modular inverses, fractions, Diophantine equations and RSA. Everything in this part starts here.
a divides b (written a | b) when b = ak for some integer k. Basic consequences, all one-line proofs from the definition: divisibility is transitive; a|b and a|c imply a|(bx+cy) for any integers x,y; and a|b with b ≠ 0 implies |a| ≤ |b|.
The division algorithm: for any integers a and b > 0 there are unique q and r with a = qb + r and 0 ≤ r < b. Uniqueness matters — it is what makes "the remainder" well-defined, and it is the reason the next paragraph's recursion is deterministic.
In C, C++, Java and Rust, % truncates toward zero, so -7 % 3 == -1, not the mathematical 2. Python and Ruby floor instead, giving 2. Any modular code that can see negative inputs needs ((a % m) + m) % m. This single discrepancy causes more competitive-programming wrong answers than any other language detail.
The Euclidean algorithm
The key lemma: gcd(a, b) = gcd(b, a mod b). Proof: if d | a and d | b then d | (a - qb) = a mod b, so every common divisor of (a,b) divides (b, a mod b); conversely if d | b and d | (a-qb) then d | a. The two pairs have identical common-divisor sets, hence identical greatest ones. ∎
int gcd(int a, int b){ return b ? gcd(b, a % b) : a; } // recursive
int gcd(int a, int b){ while (b) { int t = a % b; a = b; b = t; } return a; }
// binary GCD (Stein): no division, only shifts and subtraction
uint64_t bgcd(uint64_t a, uint64_t b){
if (!a) return b; if (!b) return a;
int s = __builtin_ctzll(a | b); // common factors of 2
a >>= __builtin_ctzll(a);
do {
b >>= __builtin_ctzll(b);
if (a > b) { uint64_t t = a; a = b; b = t; }
b -= a;
} while (b);
return a << s;
}Complexity. Each two steps at least halve the larger argument, since a mod b < a/2 whenever b ≤ a (if b ≤ a/2 then a mod b < b ≤ a/2; otherwise a mod b = a - b < a/2). So the number of iterations is O(log min(a,b)). Lamé's theorem sharpens this: the worst case is consecutive Fibonacci numbers, and the number of steps is at most ≈ 4.785 log₁₀(min(a,b)) + 1.67. Two 64-bit numbers therefore take under 90 iterations.
Identities worth knowing
Compute lcm as a / gcd(a,b) * b, never a * b / gcd(a,b) — the latter overflows for inputs whose product exceeds the word size even when the answer fits comfortably.
Bézout's identity
For any a, b there exist integers x, y with ax + by = gcd(a,b), and gcd(a,b) is the smallest positive integer expressible as such a combination.
Proof of the second claim: let d be the smallest positive value of ax+by. Dividing, a = qd + r with 0 ≤ r < d; but r = a - qd = a(1-qx) + b(-qy) is also a combination, and it is smaller than d, so r = 0 and d | a. Similarly d | b, so d is a common divisor; and every common divisor divides ax+by = d, so d = gcd(a,b). ∎
Euclid's lemma follows immediately and is the cornerstone of unique factorisation: if p is prime and p | ab then p | a or p | b. Indeed if p ∤ a then gcd(p,a) = 1, so px + ay = 1; multiply by b to get pbx + aby = b, and p divides both terms on the left, hence p | b. ∎
"Reduce a fraction", "find if a rectangle tiles by squares", "when do two periodic events coincide" and "can I measure c litres with jugs of a and b" are all gcd questions. Recognising the gcd behind a word problem is worth more than knowing the algorithm — everyone knows the algorithm.
- Theory
Prove gcd(a,b) · lcm(a,b) = |ab|.
Solution
Write the prime factorisations a = ∏pᵢ^{αᵢ}, b = ∏pᵢ^{βᵢ}. Then gcd = ∏pᵢ^{min(αᵢ,βᵢ)} and lcm = ∏pᵢ^{max(αᵢ,βᵢ)}. Since min(x,y)+max(x,y) = x+y for every prime, the product is ∏pᵢ^{αᵢ+βᵢ} = |ab|. ∎
Note the identity fails for three or more numbers: gcd(2,3,4)·lcm(2,3,4) = 1·12 = 12 ≠ 24, because min+max does not equal the sum when there are three exponents. The correct three-variable statement needs inclusion–exclusion over the pairwise gcds.
- Theory
Prove that the Euclidean algorithm's worst case is consecutive Fibonacci numbers.
Solution
Claim: if the algorithm on (a,b) with a > b > 0 takes n steps, then a ≥ Fₙ₊₂ and b ≥ Fₙ₊₁. Induction on n. For n = 1, b ≥ 1 = F₂ and a ≥ 2 = F₃. For the step, after one iteration the pair is (b, a mod b) taking n-1 steps, so b ≥ Fₙ₊₁ and a mod b ≥ Fₙ. Since a ≥ b + (a mod b) ≥ Fₙ₊₁ + Fₙ = Fₙ₊₂. ∎
Contrapositive: if a < Fₙ₊₂ then fewer than n steps. Since Fₖ ≈ φᵏ/√5, the step count is ≤ log_φ(√5 a) ≈ 4.785 log₁₀a + 1.67 — Lamé's theorem, historically the first-ever complexity analysis of an algorithm.
- Theory
Find integers x, y with 240x + 46y = 2 by hand, showing the extended Euclid table.
Solution
240 = 5·46 + 10 46 = 4·10 + 6 10 = 1·6 + 4 6 = 1·4 + 2 4 = 2·2 + 0 ⟹ gcd = 2 back-substitute: 2 = 6 − 1·4 = 6 − (10 − 6) = 2·6 − 10 = 2(46 − 4·10) − 10 = 2·46 − 9·10 = 2·46 − 9(240 − 5·46) = 47·46 − 9·240So x = -9, y = 47. Check: 240(-9) + 46(47) = -2160 + 2162 = 2. ✓ The general solution is x = -9 + 23t, y = 47 - 120t, adding b/g = 23 and subtracting a/g = 120.
- Theory
Prove gcd(aᵐ - 1, aₙ - 1) = a^{gcd(m,n)} - 1 for a ≥ 2.
Solution
Key step: aᵐ - 1 mod (aₙ - 1) = a^{m mod n} - 1 for m ≥ n. Indeed aᵐ - 1 = a^{m-n}(aₙ-1) + (a^{m-n} - 1), so subtracting multiples of aₙ-1 reduces the exponent by n each time — exactly the Euclidean step on the exponents.
So running Euclid on (aᵐ-1, aₙ-1) mirrors running it on (m,n), terminating at a^{gcd(m,n)} - 1. ∎
Corollary: 2ᵐ-1 and 2ₙ-1 are coprime iff m and n are — the fact behind Mersenne-prime exponents needing to be prime, and behind the structure of the multiplicative group used in some hash designs.
- Theory
Show that gcd(Fᵐ, Fₙ) = F_{gcd(m,n)}.
Solution
Two lemmas. (i) Fᵐ₊ₙ = FᵐFₙ₊₁ + Fᵐ₋₁Fₙ (provable by induction on n). (ii) gcd(Fₙ, Fₙ₊₁) = 1 (consecutive Fibonacci numbers are coprime, by induction using the recurrence).
From (i) with m = qn + r, repeated application gives gcd(Fᵐ, Fₙ) = gcd(Fₕ, Fₙ), mirroring Euclid on the indices. The recursion ends at gcd(F₀, F_g) = gcd(0, F_g) = F_g with g = gcd(m,n). ∎
Concretely gcd(F₁₂, F₁₈) = F₆ = 8: gcd(144, 2584) = 8. ✓ This makes the Fibonacci sequence a strong divisibility sequence, a property shared by aₙ-1 from the previous problem.
- Theory
Given gcd(a,b) = 1, prove that gcd(a+b, ab) = 1 and that gcd(a+b, a-b) ∈ {1,2}.
Solution
First: let p be a prime dividing both a+b and ab. From p | ab and Euclid's lemma, p | a or p | b; say p|a. Then from p | a+b we get p | b, contradicting gcd(a,b)=1. So no prime divides both. ∎
Second: let d = gcd(a+b, a-b). Then d | (a+b)+(a-b) = 2a and d | (a+b)-(a-b) = 2b, so d | gcd(2a,2b) = 2gcd(a,b) = 2. ∎ It is 2 exactly when a and b are both odd, and 1 when they have different parities.
- Theory
Prove that √2 is irrational using Bézout / Euclid's lemma rather than the parity argument.
Solution
Suppose √2 = p/q with gcd(p,q)=1. Then p² = 2q², so 2 | p². By Euclid's lemma (2 is prime), 2 | p. Write p = 2r: then 4r² = 2q², so q² = 2r², and again 2 | q. But then 2 | gcd(p,q) = 1, contradiction. ∎
The value of this phrasing: it generalises verbatim. For any prime p and any n not a perfect p-th power, the same argument shows n^{1/p} is irrational — Euclid's lemma is doing all the work, and it is exactly what fails for a composite radicand.
- Program
Implement gcd for arbitrary-precision integers and compare Euclid, binary GCD and the language built-in on 1000-bit numbers.
Solution
import math, random, time def euclid(a, b): while b: a, b = b, a % b return a def binary(a, b): if not a: return b if not b: return a s = ((a | b) & -(a | b)).bit_length() - 1 a >>= (a & -a).bit_length() - 1 while b: b >>= (b & -b).bit_length() - 1 if a > b: a, b = b, a b -= a return a << sFor big integers, Euclid wins because each
%is a single optimised bignum division, whereas binary GCD does O(bits) subtractions each costing O(bits/64) words — quadratic. Binary GCD's advantage is only on fixed-width machine integers where division is 20–40 cycles and shifts are 1.math.gcdbeats both, using Lehmer's algorithm: it runs several Euclid steps on the leading digits alone, then applies the accumulated 2×2 transformation to the full numbers, doing far fewer full-precision divisions. - Program
Given an array of up to 10⁵ integers, answer 10⁵ range-gcd queries. Give an O(n log n + q log n) solution and an O(n log n + q) one.
Solution
// sparse table: gcd is idempotent, so overlapping blocks are fine int sp[LOG][N]; for (int i = 0; i < n; i++) sp[0][i] = a[i]; for (int k = 1; (1<<k) <= n; k++) for (int i = 0; i + (1<<k) <= n; i++) sp[k][i] = gcd(sp[k-1][i], sp[k-1][i + (1<<(k-1))]); int query(int l, int r){ // O(1) per query int k = 31 - __builtin_clz(r - l + 1); return gcd(sp[k][l], sp[k][r - (1<<k) + 1]); }The sparse table gives O(1) queries (times the O(log) cost of one gcd) because gcd is idempotent — gcd(x,x) = x — so the two overlapping halves may double-count. A segment tree gives O(log n) queries but supports updates.
A useful extra fact: for a fixed left endpoint, the gcd of a[l..r] takes only O(log A) distinct values as r grows, since each change divides the previous by at least 2. That gives an O(n log A) algorithm for counting all subarrays with a given gcd.
- Program
Implement exact rational arithmetic (add, multiply, compare) with automatic reduction, and explain where overflow can still occur.
Solution
struct Frac { long long n, d; // invariant: d > 0, gcd=1 Frac(long long a = 0, long long b = 1){ if (b < 0) a = -a, b = -b; long long g = std::gcd(llabs(a), b); if (!g) g = 1; n = a/g; d = b/g; } Frac operator+(const Frac& o) const { long long g = std::gcd(d, o.d); // reduce BEFORE multiplying return Frac(n*(o.d/g) + o.n*(d/g), d/g*o.d); } Frac operator*(const Frac& o) const { long long g1 = std::gcd(llabs(n), o.d), g2 = std::gcd(llabs(o.n), d); return Frac((n/g1)*(o.n/g2), (d/g2)*(o.d/g1)); } bool operator<(const Frac& o) const { // avoid n*o.d overflow return (__int128)n * o.d < (__int128)o.n * d; } };Cross-reducing before multiplying keeps intermediates as small as possible, but overflow remains possible: adding 1/p for many distinct primes p makes the denominator their product, growing exponentially. There is no fix within fixed-width integers — the only real answers are big integers, or a problem-specific bound proving denominators stay small. Always ask which one you have before shipping rationals.
- Program
Given n ≤ 10⁵ integers, count pairs (i,j) with gcd(aᵢ,a₃) = 1, values up to 10⁶.
Solution
Count multiples, then Möbius-invert. Let cᵈ be the number of elements divisible by d; the pairs with gcd divisible by d number C(cᵈ,2).
MAX = 10**6 + 1 cnt = [0]*MAX for v in a: cnt[v] += 1 mult = [0]*MAX for d in range(1, MAX): for m in range(d, MAX, d): mult[d] += cnt[m] # O(MAX log MAX) mu = mobius_sieve(MAX) ans = sum(mu[d] * (mult[d]*(mult[d]-1)//2) for d in range(1, MAX))By Möbius inversion (Chapter 25), ∑_d μ(d)·#{pairs both divisible by d} counts exactly the coprime pairs, because ∑_{d | g} μ(d) = [g = 1]. Total cost O(A log A + n) — about 1.4·10⁷ operations, comfortably fast.
- Program
Implement the Stern–Brocot tree / continued-fraction search to find the fraction with the smallest denominator in a given real interval (x, y).
Solution
from fractions import Fraction def simplest(x, y): # smallest-denominator fraction in (x, y) # both bounds as Fractions, x < y fx = x.numerator // x.denominator if fx + 1 < y or (fx + 1 == y and False): pass if x < 0: return -simplest(-y, -x) i = x.numerator // x.denominator if i + 1 <= y and i + 1 > x: return Fraction(i + 1) if i > x and i < y: return Fraction(i) # recurse on the fractional parts, inverted (continued-fraction step) inner = simplest(1/(y - i), 1/(x - i)) return i + 1/innerThis is the continued-fraction algorithm: strip the integer part, invert (which swaps and reverses the interval), recurse. It runs in O(log) steps — the same recursion as Euclid, which is not a coincidence: the continued-fraction expansion of a/b is precisely the sequence of quotients produced by the Euclidean algorithm.
Applications: finding the best rational approximation with a bounded denominator (the convergents are provably optimal), recovering an exact fraction from a floating-point value, and reconstructing a rational from its value modulo a large prime.
- Program
Two gears with a and b teeth mesh. After how many rotations of the first do both return to their initial positions? Handle a,b ≤ 10¹⁸.
Solution
from math import gcd def rotations(a, b): L = a // gcd(a, b) * b # lcm, computed without overflow return L // a # rotations of the first gearBoth realign after lcm(a,b) teeth have passed, which is lcm/a rotations of the first gear and lcm/b of the second. Note
a // gcd * brather thana * b // gcd: with a,b ≈ 10¹⁸ the product is 10³⁶ and overflows any 64-bit type, while the answer itself may be small.The same computation answers "when do two cron schedules coincide", "what is the period of two interleaved cycles", and "how long until two rotating replicas realign".
- Program
Compute gcd of a large set incrementally and exploit early termination. Show why the loop usually stops after a handful of elements.
Solution
def array_gcd(a): g = 0 for v in a: g = math.gcd(g, v) if g == 1: return 1 # cannot decrease further return gEach step either leaves g unchanged or divides it by at least 2, so g strictly decreases at most log₂(a₀) times — after that the gcd is stable and the loop is just confirming it. Better, for random inputs the gcd hits 1 almost immediately: the probability that k random integers are coprime is 1/ζ(k), which is 0.608 for k = 2, 0.832 for k = 3, and 0.924 for k = 4. So the early exit fires within about 3 elements on average.
The 1/ζ(k) result itself is a nice Möbius computation: P = ∑_d μ(d)/dᵏ = 1/ζ(k), previewing Chapter 25.
- Program
Write a stress test comparing your gcd to a brute-force divisor search on random small inputs, including edge cases (zeros, negatives, equal values, maximum values).
Solution
def brute(a, b): if a == 0 and b == 0: return 0 return max(d for d in range(1, max(abs(a), abs(b)) + 1) if (a % d == 0) and (b % d == 0)) import random cases = [(0,0), (0,5), (5,0), (-12,18), (12,-18), (-12,-18), (7,7), (1,1)] cases += [(random.randint(-50,50), random.randint(-50,50)) for _ in range(5000)] for a, b in cases: assert my_gcd(a, b) == (0 if a==0==b else brute(a,b)), (a, b)The edge cases are where implementations actually break:
gcd(0,0)(convention: 0), negatives (should the result be positive? — yes, by convention gcd is non-negative), and gcd(INT_MIN, x) in C++ whereabs(INT_MIN)is undefined behaviour. Handling these explicitly, and testing them, is the difference between a snippet and a library function.
Extended Euclid and linear Diophantine equations
The extended Euclidean algorithm is a constructive proof of Bézout's identity, and it is the single most reused routine in number-theoretic code: it computes modular inverses, solves linear congruences, and drives CRT and RSA key generation.
Ordinary Euclid returns g = gcd(a,b). Extended Euclid additionally returns x and y with ax + by = g. It is the same recursion carrying two extra coefficients back up.
// returns g, and sets x,y with a*x + b*y = g
long long extgcd(long long a, long long b, long long &x, long long &y){
if (!b) { x = 1; y = 0; return a; }
long long x1, y1;
long long g = extgcd(b, a % b, x1, y1);
x = y1;
y = x1 - (a / b) * y1; // because a = (a/b)*b + a%b
return g;
}
// iterative: track the row (x, y) alongside the remainder
long long extgcd_it(long long a, long long b, long long &x, long long &y){
long long x0 = 1, y0 = 0, x1 = 0, y1 = 1;
while (b) {
long long q = a / b;
a -= q*b; std::swap(a, b);
x0 -= q*x1; std::swap(x0, x1);
y0 -= q*y1; std::swap(y0, y1);
}
x = x0; y = y0; return a;
}The back-substitution step is worth understanding rather than memorising. If the recursive call gives b·x₁ + (a mod b)·y₁ = g, substitute a mod b = a - ⌊a/b⌋b:
Size of the coefficients. The x, y produced satisfy |x| ≤ b/(2g) and |y| ≤ a/(2g), so they never overflow if a and b fit. That bound is worth knowing, because a naive worry about growth is the usual reason people avoid the algorithm.
Linear Diophantine equations
The equation ax + by = c has integer solutions iff gcd(a,b) | c. Necessity: g divides the left side always. Sufficiency: Bézout gives ax₀+by₀ = g, so scaling by c/g gives a solution. The full solution set is then a lattice line:
The step sizes are b/g and a/g, not b and a — a very common bug. Once you have the parametric form, questions like "the smallest positive x" or "how many solutions with 0 ≤ x ≤ N" become interval arithmetic on t.
To find the solution with smallest non-negative x: compute m = b/g, then x = ((x₀(c/g)) mod m + m) mod m, and recover y from the equation. Doing the reduction in the t parameter rather than by looping is what makes this O(log) rather than O(answer).
Linear congruences
ax ≡ b (mod n) is the Diophantine equation ax + ny = b in disguise. So it is solvable iff g = gcd(a,n) divides b, and then there are exactly g solutions modulo n, spaced n/g apart.
vector<long long> solve(long long a, long long b, long long n){
long long x, y, g = extgcd(a, n, x, y);
if (b % g) return {}; // no solution
long long x0 = (__int128)x % n * (b / g) % n;
x0 = ((x0 % (n/g)) + n/g) % (n/g); // canonical smallest
vector<long long> r;
for (long long i = 0; i < g; i++) r.push_back(x0 + i * (n/g));
return r; // exactly g solutions mod n
}The special case g = 1 gives a unique solution, and with b = 1 that solution is the modular inverse of a — the subject of Chapter 17, and the reason extended Euclid appears in every modular-arithmetic library.
"Given jugs of 3 and 5 litres, can you measure 4?" is 3x + 5y = 4, solvable since gcd = 1 | 4, with x = 3, y = -1 — fill the 3-litre jug three times and empty the 5-litre once. Framing the puzzle as a Diophantine equation gives both the yes/no answer and the procedure, which is a much stronger response than a search.
- Theory
Solve 1215x + 1000y = 5 completely, showing the back-substitution.
Solution
FORWARD (Euclid) BACKWARD (substitute upward) 1215 = 1·1000 + 215 5 = 65 − 6·10 1000 = 4· 215 + 140 = 65 − 6(75 − 65) = 7·65 − 6·75 215 = 1· 140 + 75 = 7(140 − 75) − 6·75 = 7·140 − 13·75 140 = 1· 75 + 65 = 7·140 − 13(215 − 140) = 20·140 − 13·215 75 = 1· 65 + 10 = 20(1000 − 4·215) − 13·215 65 = 6· 10 + 5 = 20·1000 − 93·215 10 = 2· 5 + 0 = 20·1000 − 93(1215 − 1000) gcd = 5 = 113·1000 − 93·1215So x₀ = -93, y₀ = 113. Check: 1215(-93) + 1000(113) = -112{,}995 + 113{,}000 = 5. ✓
General solution, stepping by b/g = 200 and a/g = 243:
x = -93 + 200t, y = 113 - 243t.
Smallest non-negative x is at t = 1: (x,y) = (107, -130), and indeed 1215·107 - 1000·130 = 130{,}005 - 130{,}000 = 5. ✓
Hand back-substitution is error-prone — each line rewrites one remainder in terms of the two above it, and a single sign slip propagates to the end. Always verify by substituting into the original equation; in code, assert
a*x + b*y == gbefore returning. - Theory
Prove that ax + by = c has an integer solution iff gcd(a,b) | c.
Solution
(⟹) Let g = gcd(a,b). Then g | a and g | b, so g | ax+by = c.
(⟸) By Bézout there are x₀,y₀ with ax₀+by₀ = g. If c = kg, then a(kx₀) + b(ky₀) = kg = c. ∎
For the solution set: if (x,y) and (x',y') both work, subtracting gives a(x-x') = -b(y-y'), so (a/g)(x-x') = -(b/g)(y-y'). Since gcd(a/g, b/g) = 1, (b/g) | (x-x'), giving the parametrisation with step b/g.
- Theory
Prove the bound |x| ≤ b/(2g) on the coefficient extended Euclid returns.
Solution
Track the sequences xᵢ and yᵢ generated by the iterative algorithm. Two facts, both by induction: the signs of consecutive xᵢ alternate, and |xᵢ| is non-decreasing with |xᵢ₊₁| = |xᵢ₋₁| + qᵢ|xᵢ|. One also shows the invariant |xᵢ| ≤ bᵢ₊₁/(2g) where bᵢ is the remainder sequence. At termination this gives |x| ≤ b/(2g). ∎
Practically: for 64-bit inputs the coefficients stay within 64 bits, so no overflow occurs inside extgcd. Overflow appears only when you afterwards multiply x by c/g — which is why the code in the chapter casts to
__int128at exactly that step. - Theory
How many solutions does 6x ≡ 4 (mod 10) have modulo 10? List them.
Solution
g = gcd(6,10) = 2, and 2 | 4, so there are exactly 2 solutions mod 10, spaced 10/2 = 5 apart.
Reduce: 3x ≡ 2 (mod 5). Since 3·2 = 6 ≡ 1, the inverse of 3 mod 5 is 2, giving x ≡ 4 (mod 5). So x ∈ {4, 9} mod 10. Check: 6·4 = 24 ≡ 4 ✓, 6·9 = 54 ≡ 4 ✓.
The general recipe — divide the whole congruence by g including the modulus, solve the now-invertible congruence, then expand back to g solutions — is the one to internalise.
- Theory
The Frobenius / Chicken McNugget problem: for coprime a,b > 1, prove the largest integer not representable as ax+by with x,y ≥ 0 is ab-a-b.
Solution
Not representable: suppose ab-a-b = ax+by with x,y ≥ 0. Then ab = a(x+1) + b(y+1), so a | b(y+1); as gcd(a,b)=1, a | y+1, so y+1 ≥ a. Similarly x+1 ≥ b. Then a(x+1)+b(y+1) ≥ 2ab > ab, contradiction. ∎
Everything larger is representable: for n > ab-a-b, choose x ≡ n b^{-1} (mod a) with 0 ≤ x < a. Then n - bx ≡ 0 (mod a) and n - bx > ab-a-b - b(a-1) = -a, so n-bx ≥ 0 and divisible by a. ∎
Bonus: exactly (a-1)(b-1)/2 non-negative integers are unrepresentable — a nice symmetry, since n is representable iff ab-a-b-n is not.
- Theory
Show that ax ≡ b (mod n) has a unique solution for every b iff gcd(a,n) = 1.
Solution
(⟸) If gcd(a,n)=1, extended Euclid gives a^{-1}, and x ≡ a^{-1}b is a solution; it is unique because ax ≡ ax' implies a(x-x') ≡ 0, and multiplying by a^{-1} gives x ≡ x'.
(⟹) If g = gcd(a,n) > 1, take b = 1: g | ax + ny always, but g ∤ 1, so there is no solution at all — failing existence, hence failing uniqueness-for-every-b. ∎
Restated: x ↦ ax is a bijection on ℤ/n exactly when a is a unit. This is why the "multiply by an odd constant" step in hash functions mixing modulo 2ᵏ is invertible — odd numbers are precisely the units mod a power of 2.
- Theory
Given ax+by = c with a solution, count the solutions with 0 ≤ x ≤ X and 0 ≤ y ≤ Y.
Solution
Parametrise: x = x₀+ (b/g)t, y = y₀ - (a/g)t. Each constraint becomes an interval on t:
0 ≤ x₀+(b/g)t ≤ X gives t ∈ [⌈ -x₀g/b ⌉, ⌊ (X-x₀)g/b ⌋], and 0 ≤ y₀-(a/g)t ≤ Y gives another interval (with reversed direction since the coefficient is negative).
The answer is the length of the intersection, max(0, hi - lo + 1). ∎ The whole problem is one gcd plus careful floor/ceiling arithmetic on negatives — which is where the bugs live, so use
floordiv/ceildivhelpers rather than inline division. - Program
Implement extended Euclid iteratively and verify ax+by=g plus the coefficient bounds on 10⁶ random 62-bit pairs.
Solution
def extgcd(a, b): old_r, r = a, b old_s, s = 1, 0 old_t, t = 0, 1 while r: q = old_r // r old_r, r = r, old_r - q*r old_s, s = s, old_s - q*s old_t, t = t, old_t - q*t return old_r, old_s, old_t # g, x, y import random for _ in range(10**6): a = random.randrange(1, 1 << 62); b = random.randrange(1, 1 << 62) g, x, y = extgcd(a, b) assert a*x + b*y == g assert g == math.gcd(a, b) assert abs(x) <= b // (2*g) + 1 and abs(y) <= a // (2*g) + 1The
+1slack in the assertion covers the degenerate cases a = b and b | a, where the bound is attained exactly. Testing with random and structured inputs (equal values, one dividing the other, consecutive Fibonacci numbers) is what catches the sign errors that random testing misses. - Program
Solve ax+by=c for |a|,|b|,|c| ≤ 10¹⁸, printing the solution with the smallest non-negative x, or reporting impossibility.
Solution
def solve(a, b, c): g, x, y = extgcd(abs(a), abs(b)) if c % g: return None x *= c // g; y *= c // g if a < 0: x = -x if b < 0: y = -y m = abs(b) // g # period of x k = (-x) // m + (1 if (-x) % m else 0) if x < 0 else -(x // m) x += k * m if x < 0: x += m y = (c - a*x) // b return x, yTwo traps. First,
x *= c // gcan overflow 64-bit even though the final reduced answer is small — do the multiplication in 128-bit, or reduce x modulo b/g before scaling and again after. Second, Python's floor division makes the shift arithmetic easy, while C++'s truncating division needs an explicitfloordiv; mixing the two conventions is the standard source of off-by-one here. - Program
Implement modular inverse via extended Euclid, and separately via Fermat when the modulus is prime. Benchmark and state when each is correct.
Solution
def inv_egcd(a, m): # works for ANY m with gcd(a,m)=1 g, x, _ = extgcd(a % m, m) if g != 1: raise ValueError("not invertible") return x % m def inv_fermat(a, p): # requires p PRIME and p ∤ a return pow(a, p-2, p)Extended Euclid does O(log m) divisions; Fermat does O(log p) modular multiplications, roughly 1.5 log₂p of them. In practice extgcd is 3–10× faster because division, though slower than multiplication, is needed far fewer times.
Correctness domains differ and this matters: Fermat silently returns a wrong answer for composite moduli rather than failing —
pow(2, 8, 10) = 6, and 2·6 = 12 ≡ 2 ≠ 1 (mod 10). Since 2 is not invertible mod 10 at all, the correct behaviour is an error. Always use extgcd unless the modulus is a known prime. - Program
Compute inverses of 1..n modulo a prime p in O(n) total.
Solution
inv = [0]*(n+1) inv[1] = 1 for i in range(2, n+1): inv[i] = (p - (p//i) * inv[p % i] % p) % pDerivation: write p = qi + r with q = ⌊p/i⌋ and r = p mod i. Then qi + r ≡ 0 (mod p), so qi ≡ -r. Multiply both sides by i^{-1}r^{-1}: q r^{-1} ≡ -i^{-1}, giving i^{-1} ≡ -q · r^{-1} (mod p). Since r < i, the value is already computed. ∎
One multiplication and one subtraction per element — vastly faster than n separate
powcalls (O(n log p)) and simpler than the prefix-product method. It requires p prime and n < p. - Program
Given a coin system with two coprime denominations a and b, compute the Frobenius number and the count of unrepresentable amounts. Then handle three denominations by DP.
Solution
# two coins: closed form frob = a*b - a - b unrep = (a-1)*(b-1)//2 # three or more: no closed form; DP over residues mod the smallest coin def frobenius(coins): a = min(coins) INF = float('inf') dist = [INF]*a; dist[0] = 0 # dist[r] = smallest reachable ≡ r mod a import heapq pq = [(0, 0)] while pq: d, r = heapq.heappop(pq) if d > dist[r]: continue for c in coins: nd = d + c; nr = nd % a if nd < dist[nr]: dist[nr] = nd; heapq.heappush(pq, (nd, nr)) return max(dist) - a # largest unrepresentableThis is the "coin problem" / Round Robin algorithm: a shortest-path computation on the residue graph mod the smallest coin, O(a · k log a). There is no closed form for three coins — computing the Frobenius number for a general coin set is NP-hard when the number of coins is part of the input, which is exactly why the two-coin formula feels like such a gift.
- Program
Implement the "water jugs" solver that outputs an actual sequence of pours reaching a target c, using the Diophantine solution rather than BFS.
Solution
def pours(a, b, c): g = math.gcd(a, b) if c % g or c > max(a, b): return None # Strategy: repeatedly fill A, pour into B, empty B when full. # This realises c = a*x - b*y for the x, y from extended Euclid. x = y = 0; steps = [] while x != c and y != c: if x == 0: x = a; steps.append(f"fill A -> ({x},{y})") elif y == b: y = 0; steps.append(f"empty B -> ({x},{y})") else: t = min(x, b - y); x -= t; y += t steps.append(f"pour A->B -> ({x},{y})") return stepsThe loop terminates because the sequence of states (x,y) cycles through all multiples of g modulo b, hitting c by Bézout. The number of steps is O(a+b), and the reverse strategy (fill B, pour into A) gives the other Bézout coefficient and may be shorter — checking both and picking the shorter is the complete answer.
The point of the chapter is that existence is settled in O(log) by a gcd, with no search at all; the search is only needed to print a witness.
- Program
Write a solver for systems of two linear Diophantine equations in three unknowns, reducing to the one-equation case.
Solution
# a1*x + b1*y + c1*z = d1 # a2*x + b2*y + c2*z = d2 # Eliminate one variable by taking an integer combination that clears it. def solve2(a1,b1,c1,d1, a2,b2,c2,d2): g, u, v = extgcd(c1, c2) # u*c1 + v*c2 = g if (c1 and c2) and (d1*c2 - d2*c1) % g: pass # Row-reduce over Z using the Hermite normal form of the 2x3 matrix, # keeping the unimodular transform to map solutions back. ...The systematic method is the Hermite normal form: apply unimodular row operations (add an integer multiple of one row to another; swap; negate) to reduce the coefficient matrix to an upper-triangular form, tracking the transform. Solvability is then read off the triangular system, and the solution set is a coset of the integer kernel lattice.
Equivalently, for two equations in three unknowns: eliminate z using extended Euclid on (c₁,c₂) to get one equation in x and y, solve that as in the chapter, then back-substitute for z — and check that the substitution divides exactly. The general theory is the Smith normal form, which handles any number of equations and unknowns and gives the structure of the solution lattice.
Modular arithmetic as a ring
Working modulo n means working in a finite ring where addition and multiplication behave normally but division does not. Knowing exactly which rules survive is what keeps contest code and cryptographic code correct.
a ≡ b (mod n) means n | (a-b). This is an equivalence relation (Chapter 5), and its classes — the residue classes — form the set ℤ/nℤ = {0, 1, …, n-1}.
The essential fact is that addition and multiplication are well defined on classes: if a ≡ a' and b ≡ b' then a+b ≡ a'+b' and ab ≡ a'b'. Proof of the second: ab - a'b' = ab - a'b + a'b - a'b' = b(a-a') + a'(b-b'), and n divides both terms. ∎
That is exactly what licenses reducing at every step, which is the single most important practical consequence: you never need to compute a huge intermediate value, only its residue.
ℤ/n is a commutative ring with unity: closed, associative, commutative, distributive, with 0 and 1. It is a field — every non-zero element invertible — precisely when n is prime. For composite n there are zero divisors: 2 · 3 ≡ 0 (mod 6) even though neither factor is 0, which is why cancellation fails.
ac ≡ bc (mod n) does not imply a ≡ b (mod n). The correct cancellation law is a ≡ b (mod n/gcd(c,n)). Example: 2·4 ≡ 2·1 (mod 6) but 4 ¬equiv; 1 (mod 6) — only 4 ≡ 1 (mod 3). Cancelling blindly is the most common modular-arithmetic bug in proofs.
Implementation, where it actually goes wrong
const long long MOD = 1'000'000'007;
long long add(long long a, long long b){ return (a + b) % MOD; }
long long sub(long long a, long long b){ return (a - b + MOD) % MOD; } // !
long long mul(long long a, long long b){ return a * b % MOD; } // ok: MOD² < 2⁶³
// If MOD can be up to 10¹⁸, a*b overflows. Use __int128:
long long mul_big(long long a, long long b, long long m){
return (__int128)a * b % m;
}
// Normalising an arbitrary (possibly negative) value:
long long norm(long long a, long long m){ return ((a % m) + m) % m; }Three rules that prevent nearly all bugs: subtract by adding MOD first; check that MOD² fits in your integer type before multiplying; and normalise any value that could be negative before using it as an index or comparing it.
Choice of modulus
Contest problems use 10⁹+7 or 998{,}244{,}353. Both are prime, so every non-zero residue is invertible. The second is chosen for NTT (Chapter 51): 998244353 = 119 · 2²³ + 1, so it has 2²³-th roots of unity, and 3 is a primitive root. 10⁹+7 has no such structure — 10⁹+6 = 2 · 500000003 — so convolution modulo it needs three NTTs and CRT.
Both are just under 2³⁰, so a product of two residues fits in 60 bits and never overflows a signed 64-bit integer. That is not a coincidence; it is why those specific primes were chosen.
Divisibility tests, explained
Every schoolroom divisibility rule is a statement about the base modulo n. Since 10 ≡ 1 (mod 9), every power of 10 is ≡ 1, so a number is congruent to its digit sum mod 9. Since 10 ≡ -1 (mod 11), powers alternate +1, -1, giving the alternating-sum rule. Since 1000 ≡ -1 (mod 7·11·13), grouping digits in threes with alternating signs tests all three at once.
"Compute aⁿ mod m for huge n" is a two-part answer: binary exponentiation for the algorithm, and — if the exponent is itself huge — reduce it modulo φ(m) by Euler's theorem, but only when gcd(a,m)=1. The second half is what distinguishes someone who has used this from someone who has read about it.
- Theory
Prove the cancellation law: ac ≡ bc (mod n) implies a ≡ b (mod n/gcd(c,n)).
Solution
Let g = gcd(c,n), c = gc', n = gn' with gcd(c',n')=1. From n | c(a-b) we get gn' | gc'(a-b), so n' | c'(a-b). Since gcd(c',n')=1, Euclid's lemma gives n' | (a-b), i.e. a ≡ b (mod n/g). ∎
When gcd(c,n)=1 this reduces to ordinary cancellation. The general form is what you need whenever the multiplier shares a factor with the modulus — for instance dividing a congruence through by 2 when the modulus is even.
- Theory
Find the last two digits of 7²⁰²⁵.
Solution
Work mod 100. gcd(7,100)=1 and φ(100) = 100(1-1/2)(1-1/5) = 40. So 7⁴⁰ ≡ 1, and 2025 = 40·50 + 25, giving 7²⁰²⁵ ≡ 7²⁵.
Now 7² = 49, 7⁴ = 2401 ≡ 1 (mod 100). So the order of 7 divides 4, and 7²⁵ = 7^{4·6+1} ≡ 7.
Answer: 07. The shortcut — noticing 7⁴ ≡ 1 directly — beats Euler here, and looking for a small order before invoking the theorem is generally worthwhile.
- Theory
Prove the divisibility rules for 9, 11 and 7·11·13 = 1001 from modular arithmetic.
Solution
Write N = ∑ dᵢ 10ᵢ.
- 9: 10 ≡ 1, so 10ᵢ ≡ 1 and N ≡ ∑ dᵢ — the digit sum. Same for 3.
- 11: 10 ≡ -1, so 10ᵢ ≡ (-1)ᵢ and N ≡ d₀ - d₁ + d₂ - … — the alternating sum.
- 1001: group digits in threes, N = ∑ gᵢ 1000ᵢ with 1000 ≡ -1 (mod 1001), so N ≡ g₀ - g₁ + g₂ - …. Since 1001 = 7·11·13, this single alternating three-digit-group sum tests divisibility by all three primes.
The general principle: a divisibility rule in base b for modulus m is easy exactly when bᵏ ≡ ±1 (mod m) for a small k — i.e. when the multiplicative order of b mod m is small.
- Theory
Show that ℤ/n is a field iff n is prime.
Solution
(⟸) If n = p is prime and 1 ≤ a < p, then gcd(a,p) = 1 (p's only divisors are 1 and p), so Bézout gives ax + py = 1, i.e. ax ≡ 1. Every non-zero element is invertible.
(⟹) If n is composite, write n = ab with 1 < a,b < n. Then a ≠ 0 in ℤ/n, but if a had an inverse a^{-1}, multiplying ab ≡ 0 by it gives b ≡ 0 — false. So a is a zero divisor and not invertible; ℤ/n is not a field. ∎ (And n = 1 gives the zero ring, which is excluded by convention.)
- Theory
How many solutions does x² ≡ 1 (mod n) have for n = 8, 15, 105? Find the pattern.
Solution
- n = 8: x = 1,3,5,7 all square to 1 mod 8 — 4 solutions.
- n = 15 = 3·5: solutions mod 3 are ±1 (2 of them), mod 5 likewise (2), so by CRT 4: x = 1,4,11,14.
- n = 105 = 3·5·7: 2³ = 8 solutions.
Pattern: the count is 2^k where k is the number of odd prime factors, multiplied by 1, 1, 2 or 4 for the power of 2 dividing n (respectively 2⁰/2¹, 2², 2ᵏ with k≥3). The n = 8 case having four square roots of 1 is why RSA-style square-root attacks work modulo composites, and why Rabin's cryptosystem produces four decryptions.
- Theory
Explain why 998244353 is the preferred NTT modulus and what makes a prime "NTT-friendly".
Solution
An NTT of length 2ᵏ needs a primitive 2ᵏ-th root of unity in ℤ/p, which exists iff 2ᵏ | p-1. So an NTT-friendly prime has the form p = c · 2ᵏ + 1 with k large.
998244353 = 119 · 2²³ + 1, supporting transforms up to length 2²³ = 8{,}388{,}608, with 3 as a primitive root (so g = 3^{(p-1)/2ᵏ} is the required root). It is also just under 2³⁰ so products fit in 64 bits.
By contrast 10⁹+7 - 1 = 2 · 500000003, divisible by only 2¹, so no NTT of length beyond 2 exists modulo it. Convolving modulo 10⁹+7 requires either three NTTs modulo different friendly primes plus CRT, or splitting each coefficient into high and low halves and using floating-point FFT.
- Theory
Prove that n⁵ ≡ n (mod 30) for every integer n.
Solution
30 = 2·3·5, so by CRT it suffices to check each prime separately, and Fermat's little theorem gives each immediately:
- mod 2: n² ≡ n, so n⁵ ≡ n.
- mod 3: n³ ≡ n, so n⁵ = n³·n² ≡ n·n² = n³ ≡ n.
- mod 5: n⁵ ≡ n directly.
Agreeing modulo 2, 3 and 5, they agree mod 30. ∎
Generalisation: nᵏ ≡ n (mod m) for all n exactly when m is squarefree and (p-1) | (k-1) for every prime p | m — the Korselt criterion, which also characterises Carmichael numbers (Chapter 23).
- Program
Implement a modular arithmetic struct with add, sub, mul, pow and inverse, templated on the modulus, and stress-test it against Python's arbitrary precision.
Solution
template<long long MOD> struct Mint { long long v; Mint(long long x = 0){ v = x % MOD; if (v < 0) v += MOD; } Mint operator+(Mint o) const { long long r = v + o.v; return Mint(r >= MOD ? r-MOD : r); } Mint operator-(Mint o) const { long long r = v - o.v; return Mint(r < 0 ? r+MOD : r); } Mint operator*(Mint o) const { return Mint((__int128)v * o.v % MOD); } Mint pow(long long e) const { Mint r = 1, b = *this; while (e){ if (e & 1) r = r*b; b = b*b; e >>= 1; } return r; } Mint inv() const { return pow(MOD - 2); } // MOD must be prime };Using conditional subtraction rather than
%for add and sub is a measurable win (a branch or acmovagainst a 20–40 cycle division). The__int128cast inmulis what makes it safe for any MOD < 2⁶³; without it, correctness silently depends on MOD < 2³¹.Stress test by generating random operation sequences and replaying them with Python integers reduced only at the end — that catches exactly the overflow and normalisation bugs this struct exists to prevent.
- Program
Compute a · b mod m for m up to 10¹⁸ without 128-bit integers, three ways.
Solution
// 1. Russian peasant: O(log b), always correct long long mulmod(long long a, long long b, long long m){ long long r = 0; a %= m; while (b){ if (b & 1) r = (r + a) % m; a = (a + a) % m; b >>= 1; } return r; } // 2. long double trick: O(1), correct for m < 2^62 on x86 (80-bit extended) long long mulmod_fast(long long a, long long b, long long m){ long long q = (long long)((long double)a * b / m); long long r = a*b - q*m; // both sides wrap identically return r < 0 ? r + m : (r >= m ? r - m : r); } // 3. Barrett / Montgomery reduction: precompute m⁻¹ mod 2⁶⁴, replace // division by multiplication and shifts — the fastest when m is fixed.Method 1 is 64× slower but portable and obviously correct. Method 2 relies on
long doublehaving 64 bits of mantissa (true on x86, false on ARM where it is 53 bits or 113) — a genuine portability trap. Method 3 is what real libraries use, amortising a per-modulus setup over many operations. - Program
Given a string of up to 10⁶ digits, compute it modulo m, and answer queries for the value of any substring modulo m.
Solution
# prefix values: val[i] = int(s[:i]) mod m val = [0]*(n+1) for i, ch in enumerate(s): val[i+1] = (val[i]*10 + int(ch)) % m pw = [1]*(n+1) for i in range(1, n+1): pw[i] = pw[i-1]*10 % m def substring(l, r): # s[l:r] as a number, mod m return (val[r] - val[l]*pw[r-l]) % mEach query is O(1) after O(n) preprocessing. The formula is the same "remove the prefix, shifted" identity that underlies polynomial (Rabin–Karp) hashing — a substring hash is a substring value in a base, modulo a prime.
The Python
%returns a non-negative result so the subtraction is safe; in C++ the same line needs(val[r] - val[l]*pw[r-l] % m + m) % mwith the multiplication reduced first. - Program
Implement Rabin–Karp string matching with a modular rolling hash, and demonstrate a collision attack against a fixed base and modulus.
Solution
def rolling(text, pat, B=131, M=(1<<61)-1): n, m = len(text), len(pat) if m > n: return [] hp = 0 for c in pat: hp = (hp*B + ord(c)) % M h, Bm = 0, pow(B, m, M) out = [] for i, c in enumerate(text): h = (h*B + ord(c)) % M if i >= m: h = (h - ord(text[i-m])*Bm) % M if i >= m-1 and h == hp and text[i-m+1:i+1] == pat: # verify! out.append(i-m+1) return outThe attack: with a fixed B and M (a common shortcut), an attacker solves for two strings with equal hash. For M ≈ 2&sup6;&sup4;, the Thue–Morse sequence construction produces colliding pairs of length O(log M) for any base; more simply, a birthday search over 2³² random strings finds a collision in minutes. Codeforces "anti-hash" tests exploit exactly this.
Mitigations: pick B randomly at run time from [2, M-2]; use M = 2⁶¹-1 (a Mersenne prime, with fast reduction) rather than 2⁶⁴; use a double hash with two independent moduli; and always verify a hash match with a direct comparison, as the code above does — which makes the algorithm correct regardless, and only the running time probabilistic.
- Program
Compute n! mod p for n ≥ p, and explain the trap.
Solution
def fact_mod(n, p): if n >= p: return 0 # the product contains the factor p r = 1 for i in range(2, n+1): r = r*i % p return rThe trap: for n ≥ p the answer is trivially 0, which makes C(n,k) mod p a 0/0 form if you compute it as n!·(k!)^{-1}·((n-k)!)^{-1} — and 0!^{-1} does not exist. The naive formula silently produces garbage.
The fixes are Lucas' theorem (Chapter 29) for prime p — express n and k in base p and multiply the digit-wise binomials — or, when you need n! mod p with the factors of p stripped out, Wilson's theorem plus the generalised factorial (n!)_p. Recognising that n ≥ p changes the problem entirely is the whole point.
- Program
Given 10⁵ queries of the form "is aⁿ ≡ b (mod m)" with huge n given as a decimal string, answer each in O(|n|).
Solution
def pow_bigexp(a, n_str, m): # reduce the exponent modulo φ(m) when gcd(a,m)=1 (Euler); # otherwise use the lifting-the-exponent form of Euler's theorem. phi = totient(m) e = 0 for ch in n_str: e = (e*10 + int(ch)) % phi if math.gcd(a, m) == 1: return pow(a, e, m) # generalised: if n >= log2(m), a^n ≡ a^{(n mod φ) + φ} (mod m) big = len(n_str) > 7 or int(n_str) >= m.bit_length() return pow(a, e + (phi if big else 0), m) if big else pow(a, int(n_str), m)The coprime case is straight Euler. The non-coprime case needs the generalised Euler theorem: for n ≥ log₂m, aⁿ ≡ a^{(n mod φ(m)) + φ(m)} (mod m). Applying the plain reduction when gcd(a,m) > 1 gives wrong answers — for instance 2⁵ = 32 ≡ 8 (mod 12) while φ(12) = 4 and 2^{5 mod 4} = 2 ¬equiv; 8. The +φ(m) is what repairs it.
- Program
Write a checker that detects modular-arithmetic bugs in a codebase: scan for subtraction without normalisation and for multiplication that can overflow.
Solution
import re, sys PATTERNS = [ (r'\(\s*\w+\s*-\s*\w+\s*\)\s*%\s*MOD', "subtraction mod without + MOD: may yield a negative residue"), (r'(?<!__int128\)\s)\b(?:int|long)\s+\w+\s*=\s*\w+\s*\*\s*\w+\s*%\s*MOD', "32-bit multiplication before mod: overflow"), (r'pow\(\s*\w+\s*,\s*MOD\s*-\s*2', "Fermat inverse: verify MOD is prime and the base is nonzero"), (r'%\s*MOD\s*\]', # using a residue as an index "residue used as an index: ensure it is normalised non-negative"), ] for path in sys.argv[1:]: for i, line in enumerate(open(path), 1): for pat, msg in PATTERNS: if re.search(pat, line): print(f"{path}:{i}: {msg}\n {line.strip()}")Crude, but it catches the four highest-frequency modular bugs, and a linter that flags
(a - b) % MODpays for itself. The deeper check — proving no intermediate exceeds the type's range — is a job for a static analyser with interval arithmetic, or for switching to aMinttype where the invariant "always in [0, MOD)" is maintained by construction and cannot be violated by a caller.
Modular inverses and division
Division does not exist modulo n; multiplication by an inverse does, and only for elements coprime to n. Four algorithms compute inverses, each with a different precondition and cost, and picking the wrong one is a silent correctness bug rather than a crash.
The modular inverse of a modulo n is the element a^{-1} with a · a^{-1} ≡ 1 (mod n). It exists iff gcd(a,n) = 1, and then it is unique modulo n.
Existence follows from Bézout: ax + ny = 1 means ax ≡ 1. Uniqueness: if ab ≡ ac ≡ 1 then b ≡ b(ac) = (ba)c ≡ c. Non-existence when g = gcd(a,n) > 1: any ax is divisible by g, so it cannot be ≡ 1. ∎
The invertible elements form the unit group (ℤ/n)^×, of size φ(n). When n is prime this is everything non-zero, which is why prime moduli are so much more pleasant.
Four algorithms
| Method | Precondition | Cost | When to use |
|---|---|---|---|
| Extended Euclid | gcd(a,n)=1 | O(log n) divisions | the general default |
| Fermat: a^{p-2} | p prime, p ∤ a | O(log p) mults | prime modulus, one-liner |
| Euler: a^{φ(n)-1} | gcd(a,n)=1 | O(log n) + factoring | rarely — needs φ(n) |
| Batch prefix-product | all coprime to n | O(k) + one inverse | many inverses at once |
// 1. Extended Euclid — works for any modulus
long long inv(long long a, long long m){
long long x, y, g = extgcd(((a % m) + m) % m, m, x, y);
if (g != 1) return -1; // NOT invertible — do not ignore
return ((x % m) + m) % m;
}
// 2. Fermat — p must be PRIME
long long inv_p(long long a, long long p){ return power(a, p - 2, p); }
// 4. Batch: invert k elements with ONE modular exponentiation
vector<long long> batch_inv(vector<long long> a, long long m){
int k = a.size();
vector<long long> pre(k + 1); pre[0] = 1;
for (int i = 0; i < k; i++) pre[i+1] = pre[i] * a[i] % m;
long long cur = inv(pre[k], m); // one inverse for all of them
vector<long long> r(k);
for (int i = k - 1; i >= 0; i--){
r[i] = pre[i] * cur % m;
cur = cur * a[i] % m;
}
return r;
}The batch trick is worth internalising: aᵢ^{-1} = (a₀…aᵢ₋₁) · (a₀…aᵢ)^{-1}, so one inverse plus 3k multiplications inverts the whole array. It turns O(k log m) into O(k + log m), and it is used everywhere in elliptic-curve implementations, where inversion is the dominant cost.
Never write pow(a, MOD-2, MOD) without knowing MOD is prime and a ¬equiv; 0. If MOD is composite the result is silently wrong; if a ≡ 0 the result is 0, which is not an inverse of anything. Extended Euclid at least detects non-invertibility, which is why it is the safer default even when Fermat applies.
Fractions modulo a prime
Contest answers are often "a fraction p/q modulo 10⁹+7". This means p · q^{-1} mod (10⁹+7) — well defined as long as q is not a multiple of the modulus. It is not an approximation; the map from rationals with small denominators to residues is injective, and the original fraction can be recovered by continued fractions (Chapter 14, problem 12).
The practical consequence: probability and expectation problems can be computed entirely in ℤ/p with exact arithmetic, no floating point, no accumulated error. An expectation E = ∑ pᵢxᵢ becomes a sum of products of residues, and the answer is exact.
Inverses when the modulus is not prime
Two situations arise. If gcd(a,n) = 1 but n is composite, use extended Euclid — it does not care. If gcd(a,n) = g > 1, no inverse exists, and the usual repair is to work with the equation ax ≡ b (mod n) directly (Chapter 15): divide a, b and n all by g, provided g | b.
A third situation, common in binomial computations: you need a/b mod p^k where b is divisible by p. Then track the power of p separately — write b = p^e b' with p ∤ b', invert b', and handle p^e by cancellation against the numerator. That is exactly the technique behind computing binomials modulo prime powers.
"How would you compute C(n,k) mod 10⁹+7 for n ≤ 10⁶ and 10⁵ queries?" — precompute factorials in O(n), precompute inverse factorials in O(n) using one modular exponentiation and the backward recurrence invfact[i-1] = invfact[i] · i, then each query is three multiplications. Mentioning that the backward recurrence avoids n separate exponentiations is the detail that shows practice.
- Theory
Find 17^{-1} mod 43 by hand.
Solution
Euclid: 43 = 2·17 + 9, 17 = 1·9 + 8, 9 = 1·8 + 1.
Back: 1 = 9 - 8 = 9 - (17 - 9) = 2·9 - 17 = 2(43 - 2·17) - 17 = 2·43 - 5·17.
So -5·17 ≡ 1 (mod 43), giving 17^{-1} ≡ -5 ≡ 38. Check: 17·38 = 646 = 15·43 + 1. ✓
- Theory
Prove that the modular inverse, when it exists, is unique modulo n.
Solution
Suppose ab ≡ 1 and ac ≡ 1 (mod n). Then
b ≡ b · 1 ≡ b(ac) = (ba)c ≡ 1 · c ≡ c (mod n). ∎
The proof uses only associativity and the two inverse relations — it is the standard group-theoretic uniqueness-of-inverse argument, valid in any monoid. Note it does not need commutativity, which is why the same statement holds for matrix inverses.
- Theory
For which a in {1,…,11} does a^{-1} mod 12 exist? Compute all of them and identify the group structure.
Solution
Invertible iff coprime to 12: a ∈ {1,5,7,11}, so |(ℤ/12)^×| = φ(12) = 4.
Each is its own inverse: 5² = 25 ≡ 1, 7² = 49 ≡ 1, 11² = 121 ≡ 1. So every element has order 1 or 2, and the group is ℤ₂ × ℤ₂ (the Klein four-group), not cyclic.
Consequence: 12 has no primitive root. The moduli with primitive roots are exactly 1, 2, 4, pᵏ, 2pᵏ for odd primes p — a fact that matters when a problem needs a generator (Chapter 24).
- Theory
Show that if p is prime, a^{-1} ≡ a^{p-2} (mod p), and explain what goes wrong if p is composite.
Solution
By Fermat's little theorem, a^{p-1} ≡ 1 for p ∤ a. So a · a^{p-2} = a^{p-1} ≡ 1, making a^{p-2} the inverse. ∎
Composite failure: take n = 15, a = 2. Then 2^{13} = 8192 = 546·15 + 2 ≡ 2 (mod 15), and 2·2 = 4 ≠ 1. The true inverse is 8 (2·8 = 16 ≡ 1). Fermat's exponent n-2 is simply the wrong exponent; the correct one is φ(n)-1 = 7, and 2⁷ = 128 ≡ 8 ✓.
The danger is that it fails silently — no exception, just a wrong number propagating through the computation.
- Theory
Derive the recurrence inv[i] = -(p/i) · inv[p mod i] mod p for computing all inverses mod a prime p in linear time.
Solution
Write p = qi + r with q = ⌊p/i⌋, r = p mod i, so 0 ≤ r < i. Reducing mod p: qi + r ≡ 0, hence qi ≡ -r.
Multiply both sides by i^{-1}r^{-1}: q r^{-1} ≡ -i^{-1}, i.e. i^{-1} ≡ -q · r^{-1} = -⌊ p/i ⌋ · (p mod i)^{-1} (mod p). ∎
Since r < i, the needed value is already known, so a single forward pass computes all of 1^{-1},…,n^{-1} in O(n). Requires p prime and n < p (so that no i is ≡ 0).
- Theory
Explain how to recover the rational p/q from its residue r = p q^{-1} mod M, given |p|, q < √(M/2).
Solution
This is rational reconstruction. Run the extended Euclidean algorithm on (M, r), recording the remainder sequence rᵢ and the coefficient sequence tᵢ with rᵢ ≡ tᵢ · r (mod M). Stop at the first i with rᵢ < √(M/2). Then p = rᵢ and q = tᵢ (up to sign), and this pair is unique under the size bound.
Uniqueness: if two pairs (p₁,q₁) and (p₂,q₂) both satisfied the congruence with all parts below √(M/2), then p₁q₂ - p₂q₁ ≡ 0 (mod M) while |p₁q₂ - p₂q₁| < M, forcing it to be 0 — so the fractions are equal. ∎
This is how computer algebra systems do exact linear algebra: work modulo a large prime (fast, no coefficient growth), then reconstruct the rationals at the end.
- Theory
Given n and its factorisation, count how many a ∈ [1,n] are self-inverse, i.e. a² ≡ 1 (mod n).
Solution
By CRT the condition splits over prime powers. For odd pᵏ, x² ≡ 1 has exactly 2 solutions (±1), since pᵏ | (x-1)(x+1) and p cannot divide both factors. For 2ᵏ: 1 solution when k = 1, 2 when k = 2, and 4 when k ≥ 3 (namely ±1, 2^{k-1}±1).
So the count is 2^{ω_{odd}(n)} · c₂ where ω_{odd} counts distinct odd primes and c₂ ∈ {1,1,2,4} by the power of 2.
Application: the product of all units mod n is -1 when there is exactly one self-inverse element other than 1 (odd prime powers and twice them), and +1 otherwise — the generalisation of Wilson's theorem, and the reason Wilson's (p-1)! ≡ -1 works: all units pair off with their distinct inverses except ±1.
- Program
Implement all four inverse algorithms and verify they agree on every invertible residue for all n ≤ 200.
Solution
import math def inv_eg(a, m): g, x, _ = extgcd(a % m, m) return None if g != 1 else x % m def inv_fermat(a, p): return pow(a, p-2, p) def inv_euler(a, m): return pow(a, totient(m)-1, m) def inv_brute(a, m): for x in range(m): if a*x % m == 1: return x return None for m in range(2, 201): for a in range(m): e = inv_eg(a, m); b = inv_brute(a, m) assert e == b, (a, m, e, b) if math.gcd(a, m) == 1: assert inv_euler(a, m) == b if is_prime(m) and a: assert inv_fermat(a, m) == bThe test that matters is the non-invertible case:
inv_egmust returnNone, whileinv_fermatandinv_eulerreturn numbers that are not inverses. Asserting that they agree only when the precondition holds is what documents the precondition in executable form. - Program
Precompute factorials and inverse factorials mod 10⁹+7 up to 10⁷, then answer 10⁶ binomial queries. Give both the O(n log p) and O(n + log p) preprocessing.
Solution
MOD = 10**9 + 7; N = 10**7 + 1 fact = [1]*N for i in range(1, N): fact[i] = fact[i-1] * i % MOD # O(n + log p): ONE exponentiation, then walk down ifact = [1]*N ifact[N-1] = pow(fact[N-1], MOD-2, MOD) for i in range(N-1, 0, -1): ifact[i-1] = ifact[i] * i % MOD def C(n, k): if k < 0 or k > n: return 0 return fact[n] * ifact[k] % MOD * ifact[n-k] % MODThe naive version calls
powonce per index: 10⁷ × 30 multiplications ≈ 3·10⁸, several seconds. The backward recurrence uses 10⁷ multiplications plus one exponentiation — about 30× faster.Why it works: (i-1)!^{-1} = (i!)^{-1} · i, since i! = (i-1)! · i. It is the batch-inverse trick specialised to a sequence where each element is a prefix product of the one before.
- Program
Implement batch inversion for 10⁶ arbitrary residues and measure the speed-up over per-element Fermat.
Solution
def batch_inv(a, m): k = len(a) pre = [1]*(k+1) for i in range(k): pre[i+1] = pre[i] * a[i] % m cur = pow(pre[k], m-2, m) # one exponentiation total out = [0]*k for i in range(k-1, -1, -1): out[i] = pre[i] * cur % m cur = cur * a[i] % m return outCost: 3k multiplications plus one exponentiation (≈ 1.5 log₂m ≈ 45 multiplications). Per-element Fermat costs 45k. At k = 10⁶ that is 3M versus 45M multiplications — a measured 10–15× speed-up.
Caveat: every element must be invertible, since a single zero makes the whole prefix product zero and corrupts all outputs. Guard with a filter, or substitute 1 for the zeros and mask the results afterwards.
- Program
Compute the expected value of a random process exactly as a residue mod 10⁹+7, and verify it against a floating-point simulation.
Solution
MOD = 10**9 + 7 def inv(x): return pow(x, MOD-2, MOD) # Example: expected number of rolls of a fair d-sided die to see all faces # (coupon collector) = d * H_d, exactly, as a residue. def expected_collect(d): r = 0 for i in range(1, d+1): r = (r + inv(i)) % MOD # H_d as a residue return r * d % MOD # verify: reconstruct the rational and compare to the float value from fractions import Fraction def check(d): exact = d * sum(Fraction(1, i) for i in range(1, d+1)) assert expected_collect(d) == exact.numerator % MOD * inv(exact.denominator) % MODThe residue carries the exact rational, not an approximation — so no error accumulates across a long DP, which is the whole reason contest problems ask for the answer in this form. To sanity-check during development, compute the same DP in
Fractionfor small inputs and compare, since a residue is unreadable to a human and a sign or inverse error looks like any other 9-digit number. - Program
Given a, b, n with n composite, solve ax ≡ b (mod n) and return all solutions, using the gcd reduction rather than an inverse.
Solution
def solve_lin(a, b, n): g = math.gcd(a, n) if b % g: return [] # no solution a1, b1, n1 = a//g, b//g, n//g x0 = b1 * inv_eg(a1 % n1, n1) % n1 # now gcd(a1,n1)=1 return [(x0 + i*n1) % n for i in range(g)] # exactly g solutions mod nTest cases that matter:
solve_lin(6,4,10)→[4,9];solve_lin(6,3,10)→[]since gcd(6,10)=2 ∤ 3;solve_lin(0,0,n)→ every residue.The point of the chapter: reaching for an inverse first is the wrong instinct when the modulus is composite, because the inverse may not exist even though the equation is perfectly solvable. Reduce by the gcd, then invert.
- Program
Implement Montgomery multiplication (multiplication mod n using only shifts, multiplications and additions) and benchmark it against
%.Solution
// Precompute for odd n: R = 2^64, n' = -n^{-1} mod R, R2 = R^2 mod n struct Mont { uint64_t n, n_inv, r2; Mont(uint64_t n_): n(n_) { n_inv = 1; for (int i = 0; i < 6; i++) n_inv *= 2 - n * n_inv; // Newton, mod 2^64 n_inv = -n_inv; r2 = -(__uint128_t)n % n; } uint64_t reduce(__uint128_t t) const { uint64_t m = (uint64_t)t * n_inv; uint64_t y = (t + (__uint128_t)m * n) >> 64; return y >= n ? y - n : y; } uint64_t mul(uint64_t a, uint64_t b) const { return reduce((__uint128_t)a*b); } };The Newton iteration computes n^{-1} mod 2⁶⁴ in 6 steps because each doubles the number of correct bits (1 → 2 → 4 → … → 64). Requires n odd, which is why Montgomery is used for prime moduli and RSA.
Measured: 2–4× faster than
%for repeated multiplications with a fixed modulus, because 64-bit division is 20–40 cycles while Montgomery reduction is two multiplications and a shift. Values live in "Montgomery form" aR mod n, so convert in once (mul(a, r2)) and out at the end (reduce(a)) — the conversion only pays off across many operations, which is exactly the modular-exponentiation workload. - Program
Write a fuzz test that finds inputs where a
pow(a, MOD-2, MOD)-based inverse silently produces a wrong answer.Solution
import random, math bad = [] for _ in range(200000): m = random.randrange(3, 1000) a = random.randrange(1, m) r = pow(a, m-2, m) if a * r % m != 1: bad.append((a, m, r, math.gcd(a, m), is_prime(m))) # group the failures from collections import Counter print(Counter((g > 1, prime) for _,_,_,g,prime in bad))Two failure classes appear. (a) composite m with gcd(a,m)=1: an inverse exists but Fermat's exponent is wrong — e.g. a=2, m=15 gives 2, not 8. (b) gcd(a,m) > 1: no inverse exists at all, yet
powhappily returns a number.Notably there are no failures for prime m with a ≠ 0 — the precondition is exactly right. That is the value of this fuzz test: it converts "MOD must be prime" from a comment into a demonstrated fact, and it is a good regression test to keep, because someone will eventually change MOD to a non-prime.
Primes and the sieve of Eratosthenes
The sieve is the fastest way to know everything about every number up to n, and its variants — linear, segmented, smallest-prime-factor — cover almost every "for each number up to a million" problem you will meet.
A prime is an integer > 1 whose only positive divisors are 1 and itself. The fundamental theorem of arithmetic says every integer > 1 factors into primes uniquely up to order — existence by strong induction, uniqueness by Euclid's lemma.
Two facts about density govern all the algorithms:
The second line is why RSA key generation is fast: primes are dense enough that random search works, and each candidate is rejected in microseconds by trial division against small primes before an expensive Miller–Rabin test is run.
The classical sieve
vector<bool> sieve(int n){
vector<bool> comp(n+1, false);
for (int i = 2; (long long)i*i <= n; i++)
if (!comp[i])
for (int j = i*i; j <= n; j += i) // start at i², not 2i
comp[j] = true;
return comp; // comp[x] == false ⟺ x prime
}Two optimisations carry all the weight. Starting the inner loop at i² is correct because any smaller multiple ki with k < i was already crossed off when the smaller prime factor of k was processed. Stopping the outer loop at √n is correct because a composite ≤ n has a prime factor ≤ √n.
Complexity is ∑p ≤ n n/p = n ln ln n + O(n) by Mertens' theorem — effectively linear, with ln ln 10⁸ ≈ 2.9.
Three variants worth knowing
Smallest prime factor sieve. Store spf[x] instead of a boolean, and factorisation of any x ≤ n becomes O(log x) with no division loop. This is the single most useful sieve variant for contest work — one array answers "factorise", "is prime", "count divisors", and "is squarefree".
Linear sieve. Each composite is crossed off exactly once, by its smallest prime factor, giving Θ(n) and — more importantly — a natural place to compute any multiplicative function (Chapter 19) alongside.
vector<int> primes; vector<int> spf(n+1, 0);
for (int i = 2; i <= n; i++){
if (!spf[i]) { spf[i] = i; primes.push_back(i); }
for (int p : primes){
if (p > spf[i] || (long long)p*i > n) break; // ← the crucial guard
spf[p*i] = p;
}
}The p > spf[i] break is what makes it linear: it stops before p exceeds i's smallest prime factor, so each composite p·i is written exactly once, at the moment p is its smallest prime factor.
Segmented sieve. To find primes in [L, R] with R up to 10¹² and R-L ≤ 10⁶: sieve base primes up to √R, then cross their multiples off a window of size R-L+1. Memory is O(√R + (R-L)) rather than O(R).
Trial division to √n tests one number in O(√n). A sieve tests every number up to n in O(n log log n) — that is O(log log n) amortised each. So: one query, trial-divide; many queries over a bounded range, sieve; one query with n astronomically large, Miller–Rabin (Chapter 23).
"Count primes below n" is a sieve question, and the follow-up is memory: a vector<bool> for n = 10⁹ is 125 MB, which fits, but sieving only odd numbers halves it and a wheel on {2,3,5} cuts it to 27%. If asked for n = 10¹², the answer is a segmented sieve, or the Meissel–Lehmer method which counts primes without enumerating them in about O(n^{2/3}).
- Theory
Prove that it suffices to test divisors up to √n when checking primality by trial division.
Solution
If n = ab with 1 < a ≤ b < n, then a² ≤ ab = n, so a ≤ √n. Hence any composite n has a divisor in (1, √n]. Contrapositive: if no divisor ≤ √n exists, n is prime. ∎
Implementation note: write the loop condition as
i*i <= n, noti <= sqrt(n). The latter calls a floating-point function whose result can be off by one ulp near perfect squares, producing a wrong answer for specific large n — andi*imust itself be computed in a wide enough type. - Theory
Prove the sieve's inner loop may start at i².
Solution
Consider a multiple ki with 2 ≤ k < i. Let q be the smallest prime factor of k; then q ≤ k < i, and q was processed by the outer loop earlier. At that time ki was crossed off, since ki is a multiple of q and ki ≥ q² (because ki ≥ 2i > 2q and… more directly, ki/q ≥ i > q so ki ≥ q·i > q²). So no multiple below i² needs crossing again. ∎
The saving is real: without it, the work is ∑p≤nn/p ≈ n ln ln n anyway asymptotically, but the constant roughly halves and, more importantly, the write traffic to memory halves — which dominates for large n.
- Theory
Estimate π(10¹⁸) using the prime number theorem and state the accuracy of the estimate.
Solution
π(n) ≈ n/ln n: 10¹⁸/(18 ln 10) = 10¹⁸/41.45 ≈ 2.41 × 10¹⁶.
The better estimate is the logarithmic integral li(n) = ∫₂ⁿ dt/ln t ≈ 2.4739 × 10¹⁶, and the true value is 24{,}739{,}954{,}287{,}740{,}860. So n/ln n is about 2.6% low while li(n) is accurate to 9 significant figures.
The error term is where the Riemann hypothesis lives: RH is equivalent to |π(n) - li(n)| = O(√n log n). Unconditionally we only know a weaker bound, which is why the difference between the two estimates is not a detail.
- Theory
Prove the linear sieve crosses off each composite exactly once.
Solution
Every composite m has a unique smallest prime factor p, so it is uniquely written as m = p · i with i = m/p and p ≤ spf(i).
The algorithm, at iteration i, loops over primes p in increasing order and breaks when p > spf(i). So it writes p·i exactly for the primes p ≤ spf(i) — which is precisely the set of pairs (p,i) in the unique representation above. Each composite is therefore produced by exactly one (p,i) pair. ∎
Total work is O(n), but the classical sieve is often faster in wall time despite doing n ln ln n work, because its writes are sequential strides while the linear sieve accesses
primes[]andspf[]irregularly. Use the linear sieve when you need the multiplicative-function hook, not for speed alone. - Theory
Show that there are arbitrarily large gaps between consecutive primes.
Solution
For any n, the n-1 consecutive integers n!+2, n!+3, …, n!+n are all composite: k | n! and k | k, so k | (n!+k) for 2 ≤ k ≤ n. ∎
So gaps of any length exist. The construction is wasteful — it places the gap near n!, an astronomically large number — whereas the maximal gap below 10¹⁸ is only 1,476. Cramér's conjecture predicts gaps are O(log² p); the best proved bound is far weaker.
Contrast with the Bertrand–Chebyshev theorem: there is always a prime between n and 2n. So gaps grow, but never as fast as the numbers themselves.
- Theory
How much memory does a bitset sieve to 10⁹ need? Design a wheel-based layout reducing it further.
Solution
- Naive
vector<bool>: 10⁹ bits = 125 MB. - Odd numbers only: 62.5 MB (drop all evens; index x at bit (x-3)/2).
- Wheel mod 30: only 8 residues coprime to 30 (1,7,11,13,17,19,23,29), so 8/30 = 26.7% → 33 MB, one byte per 30 integers.
- Wheel mod 210 (adding 7): 48/210 = 22.9% → 28.6 MB, with more complex indexing.
The mod-30 wheel is the sweet spot: one byte per block of 30 makes the indexing a table lookup, and it is what primesieve and similar libraries use. Beyond that, a segmented wheel sieve keeps the working set inside L2 cache, which matters more than total memory — cache misses, not bytes, are the binding constraint above 10⁸.
- Naive
- Theory
Prove that if 2ₙ-1 is prime then n is prime, and that the converse fails.
Solution
If n = ab with 1 < a,b < n, then 2ᵃ-1 divides 2ₙ-1, since x-1 | xᵇ-1 with x = 2ᵃ. As 1 < 2ᵃ-1 < 2ₙ-1, the latter is composite. Contrapositive gives the claim. ∎
Converse fails: n = 11 is prime but 2¹¹-1 = 2047 = 23 × 89. Only 52 Mersenne primes are known, and searching for them (GIMPS) uses the Lucas–Lehmer test, which is O(n²)-ish per candidate and specific to this form — dramatically faster than general primality testing, which is why the largest known primes are all Mersenne.
- Program
Implement the sieve, the odd-only sieve and a mod-30 wheel sieve to 10⁹, and compare time and memory.
Solution
// odd-only: bit i represents the number 2i+3 vector<uint64_t> bits((n/2)/64 + 2, 0); auto isc = [&](int x){ return (bits[x>>6] >> (x&63)) & 1; }; auto set = [&](int x){ bits[x>>6] |= 1ull << (x&63); }; for (int i = 0; (2*i+3)*(long long)(2*i+3) <= n; i++) if (!isc(i)){ int p = 2*i+3; for (long long j = (long long)p*p; j <= n; j += 2*p) // odd multiples only set((j-3)/2); }Typical timings at n = 10⁹ on a modern CPU: naive ~6 s / 125 MB, odd-only ~3 s / 62 MB, mod-30 wheel ~1.5 s / 33 MB, segmented wheel ~0.8 s / a few MB. The segmented version wins not by doing less work but by keeping the active window in L2 cache — the same Chapter 8 lesson that the RAM model omits memory hierarchy.
Note
j += 2*p: since p is odd, every second multiple is odd, so stepping by 2p skips the evens that are not represented at all. - Program
Build a smallest-prime-factor table to 10⁷ and use it to factorise 10⁶ queries in O(log x) each.
Solution
N = 10**7 spf = list(range(N+1)) i = 2 while i*i <= N: if spf[i] == i: for j in range(i*i, N+1, i): if spf[j] == j: spf[j] = i # only if not yet assigned i += 1 def factorise(x): f = {} while x > 1: p = spf[x]; e = 0 while x % p == 0: x //= p; e += 1 f[p] = e return fEach division removes at least a factor of 2, so factorisation is O(log x) — at most 23 steps for x ≤ 10⁷. Compare trial division at O(√x) = 3163 steps: a 100× difference per query, which at 10⁶ queries is the difference between 0.2 s and 30 s.
Memory is the cost: 10⁷ 32-bit ints = 40 MB. Storing only the smallest prime factor as a 32-bit value is already minimal; below 2¹⁶ a 16-bit array halves it.
- Program
Implement a segmented sieve listing all primes in [L, R] with R ≤ 10¹² and R - L ≤ 10⁶.
Solution
def segmented(L, R): lim = int(R**0.5) + 1 base = simple_sieve(lim) # primes up to sqrt(R) mark = bytearray(R - L + 1) # 0 = prime candidate for p in base: start = max(p*p, ((L + p - 1) // p) * p) # first multiple of p ≥ L for j in range(start, R+1, p): mark[j - L] = 1 if L == 1: mark[0] = 1 return [L + i for i, m in enumerate(mark) if not m]Memory is O(√R + (R-L)) — about 1 MB of base primes plus the window. The subtle line is
start: beginning at p² when p² ≥ L avoids marking p itself as composite, and the ceiling division finds the first multiple ≥ L otherwise.Watch the edge cases: L = 1 (1 is not prime), L = 0, and windows lying entirely below √R. Each has bitten every implementation of this at least once.
- Program
Implement the linear sieve computing, in the same pass, φ(i), μ(i) and the divisor count d(i) for all i ≤ n.
Solution
phi = [0]*(N+1); mu = [0]*(N+1); d = [0]*(N+1); cnt = [0]*(N+1) phi[1] = mu[1] = d[1] = 1 primes = []; spf = [0]*(N+1) for i in range(2, N+1): if not spf[i]: spf[i] = i; primes.append(i) phi[i] = i-1; mu[i] = -1; d[i] = 2; cnt[i] = 1 for p in primes: if p > spf[i] or p*i > N: break j = p*i; spf[j] = p if p == spf[i]: # p divides i: same prime again phi[j] = phi[i]*p mu[j] = 0 # square factor cnt[j] = cnt[i] + 1 d[j] = d[i] // (cnt[i]+1) * (cnt[j]+1) else: # p is new: multiplicativity applies phi[j] = phi[i]*(p-1) mu[j] = -mu[i] cnt[j] = 1 d[j] = d[i]*2The two branches are exactly the two cases of a multiplicative function: when p is a new prime the values multiply, and when p already divides i you must use the prime-power rule. Tracking
cnt[i](the exponent of the smallest prime) is what makes the divisor-count update possible in O(1).This one pass, O(n) total, replaces three separate sieves and is the standard preprocessing for any problem summing a multiplicative function over a range.
- Program
Count primes below 10¹² without enumerating them, using the Meissel–Lehmer / Lucy_Hedgehog method.
Solution
def prime_count(n): r = int(n**0.5) V = [n//i for i in range(1, r+1)] V += list(range(V[-1]-1, 0, -1)) S = {v: v-1 for v in V} # S[v] = count of 2..v, refined for p in range(2, r+1): if S[p] > S[p-1]: # p is prime sp, p2 = S[p-1], p*p for v in V: if v < p2: break S[v] -= S[v//p] - sp # remove multiples of p return S[n]O(n^{3/4}) time and O(√n) memory — 10¹² in a couple of seconds, where a sieve would need 125 GB.
The idea is a dynamic programming over "the count of integers in [2,v] with no prime factor below p", evaluated only at the O(√n) distinct values of ⌊n/i⌋ — the same divisor-block structure as Chapter 9's ∑⌊n/i⌋. The same recurrence, with
v-1replaced by a sum, computes ∑p≤np and other prime sums. - Program
Generate a random 1024-bit probable prime, as an RSA key generator would. Report how many candidates were tested.
Solution
import random SMALL = [p for p in range(3, 1000) if all(p % q for q in range(2, int(p**.5)+1))] def random_prime(bits): tried = 0 while True: tried += 1 c = random.getrandbits(bits) | (1 << (bits-1)) | 1 # top and bottom bit set if any(c % p == 0 for p in SMALL): continue # cheap rejection if miller_rabin(c, rounds=40): return c, triedBy the prime number theorem the density of primes near 2¹⁰²⁴ is 1/ln(2¹⁰²⁴) = 1/710; restricting to odd numbers doubles it to 1/355, and the small-prime pre-filter removes about 80% of the remaining composites at negligible cost. Expect roughly 355 candidates, of which only ~70 reach Miller–Rabin.
Setting the top bit guarantees the modulus n = pq has the full 2048 bits; setting the bottom bit skips evens. Both are what real implementations do, and omitting the top bit is a genuine (if benign) key-generation bug.
- Program
For each k from 1 to 20, find the smallest prime gap of length ≥ k below 10⁸, and compare to the log²p heuristic.
Solution
ps = sieve_list(10**8) best = {} for a, b in zip(ps, ps[1:]): g = b - a if g not in best: best[g] = a for g in sorted(best): print(g, best[g])Results: the first gap of 10 is after 139; of 20, after 887; of 50, after 19,609; of 100, after 370,261; of 148, after 2,010,733; the maximal gap below 10⁸ is 220, after 47,326,693.
Cramér's heuristic predicts the maximal gap near p is about ln²p: at p = 5×10⁷, ln²p ≈ 313 — the right order of magnitude, and the observed maxima track 0.7 ln²p closely across the whole range. The record-gap sequence is one of the cleanest examples of a heuristic with no proof matching data over many orders of magnitude.
- Program
Given q ≤ 10⁵ queries "how many primes in [l, r]" with r ≤ 10⁷, answer each in O(1).
Solution
N = 10**7 comp = sieve(N) pre = [0]*(N+2) for i in range(2, N+1): pre[i+1] = pre[i] + (0 if comp[i] else 1) def count(l, r): return pre[r+1] - pre[l]Prefix sums over the sieve: O(N log log N) preprocessing, O(1) per query, O(N) memory (40 MB as 32-bit ints, or 20 MB as 32-bit with N = 10⁷ — fine).
If memory is tight, store the prefix count only every 64th index and recompute the remainder from the bitset with a popcount — 64× less memory at the cost of one popcount per query, which is still effectively O(1). That trade is worth knowing whenever a prefix-sum array over a bitmask blows the memory budget.
Factorisation, divisors and multiplicative functions
Once you have the prime factorisation, the number of divisors, their sum, the totient and a dozen other quantities are one formula each. Multiplicativity is the structural reason why, and it is what makes them all sievable in linear time.
Write n = p₁^{a₁} p₂^{a₂} … pₖ^{aₖ}. Every divisor is obtained by choosing an exponent 0 ≤ eᵢ ≤ aᵢ independently for each prime. That single observation gives all of the following:
Sizes to keep in mind: d(n) is on average ln n but its maximum is far larger — 240 for n ≤ 10⁶, 6,720 for n ≤ 10¹², 103,680 for n ≤ 10¹⁸. These are the highly composite numbers, and knowing the bound tells you whether "iterate over all divisors" is affordable.
Multiplicative functions
f is multiplicative if f(1)=1 and f(mn) = f(m)f(n) whenever gcd(m,n)=1; completely multiplicative if the coprimality condition is dropped. Multiplicativity means f is determined by its values on prime powers, which is exactly why every formula above is a product over primes.
| f | Multiplicative? | f(pᵃ) |
|---|---|---|
| d(n) | yes | a+1 |
| σ(n) | yes | 1+p+…+pᵃ |
| φ(n) | yes | pᵃ - p^{a-1} |
| μ(n) | yes | -1 if a=1, else 0 |
| nᵏ | completely | p^{ak} |
| Ω(n) | no (additive) | a |
Dirichlet convolution (f*g)(n) = ∑d|n f(d)g(n/d) preserves multiplicativity, and the classical identities are all convolutions:
∑d|n φ(d) = n has a beautiful bijective proof: classify each of 1..n by g = gcd(k,n). The values with gcd(k,n) = d are exactly d · m with gcd(m, n/d) = 1, and there are φ(n/d) of them. Summing over d partitions 1..n. This is the model for how almost every divisor-sum identity is proved.
Enumerating divisors
// 1. Trial division: O(√n), unsorted output
vector<long long> divisors(long long n){
vector<long long> d;
for (long long i = 1; i*i <= n; i++)
if (n % i == 0){ d.push_back(i); if (i != n/i) d.push_back(n/i); }
return d;
}
// 2. From the factorisation: O(d(n)) after factoring
void gen(int idx, long long cur, ...){
if (idx == (int)f.size()){ out.push_back(cur); return; }
for (int e = 0; e <= f[idx].second; e++){
gen(idx+1, cur, ...);
cur *= f[idx].first;
}
}
// 3. Divisor sieve: ALL divisors of ALL numbers up to n, in O(n log n)
for (int i = 1; i <= n; i++)
for (int j = i; j <= n; j += i)
div[j].push_back(i); // total size is Σ n/i ≈ n ln nThe third is the one people forget: the total number of (divisor, multiple) pairs up to n is ∑ ⌊n/i⌋ ≈ n ln n, so building the entire divisor structure for n = 10⁶ costs about 1.4 × 10⁷ operations. Any problem of the form "for each i, look at its divisors" is solvable this way in O(n log n) regardless of how hard individual factorisation is.
Divisor blocks
⌊n/i⌋ takes only O(√n) distinct values, and the indices giving each value are contiguous. This turns ∑ᵢ₋₁ⁿ f(⌊n/i⌋) from O(n) into O(√n) — the technique behind computing ∑ d(i), ∑ σ(i) and ∑ φ(i) for astronomically large n.
"Find all divisors of n" expects O(√n) trial division and the pairing observation that divisors come in pairs (d, n/d) straddling √n. The follow-up — "now do it for a million values" — wants the divisor sieve, because 10⁶ × √(10⁶) = 10⁹ is too slow while n ln n = 1.4×10⁷ is not.
- Theory
Derive d(n), σ(n) and ∏_{d|n} d from the factorisation, and compute all three for n = 360.
Solution
360 = 2³ · 3² · 5.
- d(360) = 4·3·2 = 24.
- σ(360) = (1+2+4+8)(1+3+9)(1+5) = 15 · 13 · 6 = 1170.
- ∏ d = 360^{24/2} = 360¹². This follows by pairing d with n/d: each pair multiplies to n, and there are d(n)/2 pairs. When d(n) is odd (n a perfect square) the middle divisor √n is its own partner and the formula still holds.
- Theory
Prove ∑d|n φ(d) = n.
Solution
Partition {1,…,n} by the value g = gcd(k,n), which is always a divisor of n. For a fixed divisor d, the set {k : gcd(k,n) = d} consists of k = dm with 1 ≤ m ≤ n/d and gcd(m, n/d) = 1 — exactly φ(n/d) values.
Summing over all divisors: n = ∑d|n φ(n/d) = ∑d|n φ(d), since d ↦ n/d permutes the divisors. ∎
In convolution language this is φ * 1 = id, and Möbius inversion turns it around into φ = μ * id, i.e. φ(n) = ∑d|n μ(d)·n/d — the formula behind the coprime-counting algorithms of Chapter 25.
- Theory
Prove that the Dirichlet convolution of two multiplicative functions is multiplicative.
Solution
Let gcd(m,n)=1. Every divisor of mn factors uniquely as d = d₁d₂ with d₁|m, d₂|n, and gcd(d₁,d₂)=1 (unique factorisation plus coprimality). So
(f*g)(mn) = ∑d₁|m∑d₂|n f(d₁d₂)g((mn)/(d₁d₂)) = ∑d₁|m∑d₂|n f(d₁)f(d₂)g(m/d₁)g(n/d₂)
= (∑d₁|m f(d₁)g(m/d₁))(∑d₂|n f(d₂)g(n/d₂)) = (f*g)(m)(f*g)(n). ∎
The step splitting f(d₁d₂) uses gcd(d₁,d₂)=1, which is where coprimality of m and n is consumed. This closure property is why the multiplicative functions form a group under convolution, with identity ε and inverses computable prime-power by prime-power.
- Theory
Show that n is a perfect square iff d(n) is odd, and use it to solve the "100 lockers" puzzle.
Solution
Divisors pair as (d, n/d), and a divisor is its own partner exactly when d = n/d, i.e. d² = n. So the pairing leaves an unmatched element iff n is a perfect square, making d(n) odd precisely then. ∎ Equivalently, d(n) = ∏(aᵢ+1) is odd iff every aᵢ is even.
Lockers: 100 lockers, initially closed; person k toggles every k-th locker. Locker n is toggled once per divisor, so it ends open iff d(n) is odd, i.e. iff n is a perfect square. The open lockers are 1, 4, 9, …, 100 — exactly 10 of them.
- Theory
What is the maximum of d(n) for n ≤ 10¹⁸, and what does the extremal n look like?
Solution
The maximum is 103,680, attained by n = 897{,}612{,}484{,}786{,}617{,}600.
Highly composite numbers have the shape 2^{a₁}3^{a₂}5^{a₃}… with non-increasing exponents over consecutive primes starting at 2 — if a larger prime had a bigger exponent, swapping them would give a smaller number with the same divisor count. Below 10¹⁸ such a number uses primes up to about 47.
Practical use: this bounds the cost of "iterate over all divisors". At 10¹⁸ that is 10⁵ divisors — feasible once, not per query. At 10⁶ it is 240, trivially cheap.
- Theory
Prove ∑ᵢ₋₁ⁿ d(i) = ∑ᵢ₋₁ⁿ ⌊n/i⌋ and estimate the sum.
Solution
Count the pairs (i, j) with j | i ≤ n in two ways. Grouping by i gives ∑ d(i). Grouping by the divisor j gives, for each j, the number of multiples of j up to n, which is ⌊n/j⌋. ∎ (This is the same double-counting exchange as Chapter 9 problem 6.)
Estimate: ∑⌊n/i⌋ = n Hₙ + O(n) = n ln n + (2γ-1)n + O(√n) — the Dirichlet divisor problem. The error term is famously open: the truth is conjectured to be O(n^{1/4+ε}), and the current record is about O(n^{0.3149}).
The practical takeaway: the average number of divisors of a number near n is about ln n — around 14 at n = 10⁶, versus a maximum of 240.
- Theory
A number is perfect if σ(n) = 2n. Prove that every even perfect number has the form 2^{p-1}(2ᴵ-1) with 2ᴵ-1 prime.
Solution
Let n = 2^{k}m with m odd, k ≥ 1. Multiplicativity gives σ(n) = (2^{k+1}-1)σ(m), and perfection requires (2^{k+1}-1)σ(m) = 2^{k+1}m. Since 2^{k+1}-1 is odd, it divides m: write m = (2^{k+1}-1)M. Substituting, σ(m) = 2^{k+1}M = m + M.
But m and M are both divisors of m (as M = m/(2^{k+1}-1)), so σ(m) ≥ m + M with equality only if those are the only divisors — forcing M = 1 and m prime. Hence m = 2^{k+1}-1 is a Mersenne prime, and n = 2^k(2^{k+1}-1). Writing p = k+1 gives the stated form. ∎ (Euclid proved sufficiency; Euler proved this necessity.)
Whether any odd perfect number exists is open — it is known any such number exceeds 10¹⁵₀.
- Program
Compute d(i), σ(i) and φ(i) for all i ≤ 10⁷ in one pass, two ways: harmonic sieve and linear sieve.
Solution
# harmonic sieve: simple, O(n log n), no factorisation needed d = [0]*(N+1); sig = [0]*(N+1) for i in range(1, N+1): for j in range(i, N+1, i): d[j] += 1 sig[j] += i # phi by the "sieve of Euler" pattern: O(n log log n) phi = list(range(N+1)) for p in range(2, N+1): if phi[p] == p: # p is prime for j in range(p, N+1, p): phi[j] -= phi[j] // p # multiply by (1 - 1/p)The
phisieve is elegant: initialisephi[i] = i, and for each prime p multiply every multiple by (1-1/p) — implemented as a subtraction to stay in integers. The testphi[p] == pdetects primality because no smaller prime has touched p.At N = 10⁷ the harmonic sieve does 1.6×10⁸ operations (a few seconds in C++, far too slow in Python); the linear sieve of Chapter 18 problem 11 computes all three in 10⁷ steps and is the right choice at this size.
- Program
Factorise 10⁵ numbers up to 10¹², choosing the right algorithm and justifying it.
Solution
# Trial division by primes up to 10^6 handles all factors ≤ 10^6. # What remains is 1, a prime, or a product of exactly two primes > 10^6. base = sieve_list(10**6) def factor(n): f = {} for p in base: if p*p > n: break while n % p == 0: n //= p; f[p] = f.get(p,0)+1 if n > 1: f[n] = f.get(n,0)+1 # n is prime here, since n < 10^12 return fO(π(10⁶)) = 78{,}498 divisions worst case per number — 7.8×10⁹ total, too slow for 10⁵ queries.
Better: since √(10¹²) = 10⁶, the loop usually breaks far earlier, but the worst case (n prime) is the full sweep. For 10⁵ queries use Pollard's rho (Chapter 23) at O(n^{1/4}) = 1000 operations per number, giving 10⁸ total. The crossover between trial division and Pollard is around 10⁸–10¹⁰; below it, trial division by a sieved prime list wins on constant factors.
- Program
Compute ∑ᵢ₋₁ⁿ d(i) for n ≤ 10¹² in O(√n) using the divisor-block / hyperbola method.
Solution
def sum_d(n): s = int(n**0.5) total = 0 for i in range(1, s+1): total += n // i return 2*total - s*s # hyperbola method: count under xy ≤ nWhy: ∑ d(i) counts lattice points under the hyperbola xy ≤ n. By symmetry, count those with x ≤ √n (giving ∑i≤√n⌊n/i⌋), double it to cover y ≤ √n, then subtract the ⌊√n⌋² points counted twice in the corner square. ∎
O(√n) = 10⁶ operations at n = 10¹² — instant. Use
math.isqrtfor s, neverint(n**0.5), which is off by one for some n near perfect squares and silently corrupts the corner correction. - Program
Given n ≤ 10⁵ numbers up to 10⁶, for each element count how many others divide it.
Solution
MAX = 10**6 + 1 cnt = [0]*MAX for v in a: cnt[v] += 1 # divisor-count sieve: for each d, add cnt[d] to all multiples res = [0]*MAX for d in range(1, MAX): if cnt[d]: for m in range(d, MAX, d): res[m] += cnt[d] answer = [res[v] - 1 for v in a] # exclude the element itselfO(A log A) = 1.4×10⁷ — fast. The
if cnt[d]guard skips absent values, often a large constant-factor win when the input is sparse.Contrast with the naive approach: for each element, enumerate its divisors in O(√v) and look each up — 10⁵ × 1000 = 10⁸, also feasible but slower and it does not extend to "how many are multiples of me", which the sieve direction handles by reversing the loop.
- Program
Find the smallest number with exactly k divisors, for k ≤ 1000.
Solution
PRIMES = [2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53] best = float('inf') def rec(idx, cur, rem, max_exp): global best if rem == 1: best = min(best, cur); return for e in range(1, max_exp+1): if rem % (e+1): continue # exponent must divide the target nxt = cur * PRIMES[idx]**e if nxt >= best: break rec(idx+1, nxt, rem//(e+1), e) # exponents non-increasing rec(0, 1, k, 64)Two prunings make this fast. Non-increasing exponents: swapping a larger exponent onto a smaller prime always yields a smaller number with the same divisor count, so the optimum has a₁ ≥ a₂ ≥ …. Divisibility: since d(n) = ∏(aᵢ+1) = k, each (aᵢ+1) must divide the remaining target.
Answers: k=12 → 60, k=100 → 45{,}360, k=1000 → 810{,}810{,}000. Note the answer is not always of the form "highly composite" — for prime k the answer is 2^{k-1}, which is enormous.
- Program
Implement Dirichlet convolution of two arrays of length n in O(n log n) and verify the identities 1*1 = d, φ*1 = id, μ*1 = ε.
Solution
def dirichlet(f, g, n): h = [0]*(n+1) for d in range(1, n+1): if f[d] == 0: continue for m in range(d, n+1, d): h[m] += f[d] * g[m//d] return h one = [0] + [1]*n idn = list(range(n+1)) eps = [0]*(n+1); eps[1] = 1 assert dirichlet(one, one, n)[1:] == d_table[1:] assert dirichlet(phi, one, n)[1:] == idn[1:] assert dirichlet(mu, one, n)[1:] == eps[1:]Cost is ∑ n/d = n Hₙ = O(n log n) — the same harmonic sum as the divisor sieve, which is not a coincidence: the divisor sieve is convolution with the constant function 1.
Having convolution as a primitive makes the whole theory executable: you can discover identities empirically (convolve candidates, compare to known tables) before proving them, which is a genuinely useful workflow for this material.
- Program
Compute ∑ᵢ₋₁ⁿ φ(i) for n ≤ 10¹² using a Dirichlet-hyperbola recurrence with memoisation.
Solution
from functools import lru_cache SMALL = 10**7 phi_pre = linear_sieve_phi(SMALL) pre = [0]*(SMALL+1) for i in range(1, SMALL+1): pre[i] = pre[i-1] + phi_pre[i] @lru_cache(None) def Phi(n): if n <= SMALL: return pre[n] r = n*(n+1)//2 # = Σ_{i≤n} i, since φ*1 = id i = 2 while i <= n: q = n // i; j = n // q # divisor block r -= (j - i + 1) * Phi(q) i = j + 1 return rDerivation: from ∑d|mφ(d) = m, summing over m ≤ n gives ∑k=1..n Φ(⌊ n/k ⌋) = n(n+1)/2. Isolating the k=1 term yields the recurrence. ∎
Complexity is O(n^{2/3}) when the small-value cutoff is tuned to n^{2/3} — about 10⁸ operations at n = 10¹², a second or two. The same "Dirichlet hyperbola + memoised divisor blocks" template computes ∑μ, ∑d and ∑σ, and it is the standard tool for large-n multiplicative sums.
Fermat, Euler, the totient and multiplicative order
Exponents modulo n live in a smaller world than the numbers do: they reduce modulo φ(n), not modulo n. That one fact powers RSA, fast exponentiation of huge towers, and half of contest number theory.
Fermat's little theorem: for prime p and p ∤ a, a^{p-1} ≡ 1 (mod p). Equivalently aᴵ ≡ a for all a.
Euler's theorem: for gcd(a,n)=1, a^{φ(n)} ≡ 1 (mod n), where φ(n) counts the integers in [1,n] coprime to n.
Both are the same theorem, and the proof is one idea. Let S = {s₁,…,s_{φ(n)}} be the units mod n. Multiplication by a permutes S (it is injective because a is invertible, and it maps units to units). So ∏(a sᵢ) = ∏ sᵢ, i.e. a^{φ(n)} ∏sᵢ ≡ ∏sᵢ. Cancel the product (it is a unit) to get a^{φ(n)} ≡ 1. ∎
Stated structurally: the units form a group of order φ(n), and by Lagrange's theorem every element's order divides the group order. Fermat and Euler are Lagrange's theorem in disguise.
Computing the totient
long long phi(long long n){ // O(√n)
long long r = n;
for (long long p = 2; p*p <= n; p++)
if (n % p == 0){
while (n % p == 0) n /= p;
r -= r / p; // multiply by (1 - 1/p)
}
if (n > 1) r -= r / n; // the remaining prime factor
return r;
}Note r -= r/p rather than r = r*(p-1)/p: the former stays in integers with no intermediate overflow, and it is exact because at that moment p | r.
Multiplicative order
The order ord_n(a) is the smallest k > 0 with aᵏ ≡ 1 (mod n). Key facts:
- aᵐ ≡ 1 iff ord_n(a) | m. So the order divides φ(n).
- To find the order: start from φ(n) and, for each prime q dividing it, divide out q while a^{k/q} ≡ 1 still holds.
- ord(aᵏ) = ord(a)/gcd(k, ord(a)).
To reduce a huge exponent: aⁿ ≡ a^{n mod φ(m)} (mod m) — but only when gcd(a,m)=1. When they share a factor, use the generalised form: for n ≥ log₂ m, aⁿ ≡ a^{(n mod φ(m)) + φ(m)} (mod m). Forgetting the +φ(m) is the classic wrong answer on power-tower problems.
Where this is used
RSA. Choose primes p, q, set n = pq and φ(n) = (p-1)(q-1). Pick e coprime to φ(n) and compute d = e^{-1} mod φ(n). Then (m^e)^d = m^{ed} = m^{1 + kφ(n)} ≡ m (mod n) by Euler. Security rests on φ(n) being computable only if you can factor n.
Cycle lengths. The decimal expansion of 1/n for gcd(n,10)=1 has period exactly ord_n(10). 1/7 has period 6 because ord_7(10) = 6; 1/11 has period 2 because 10² = 100 ≡ 1 (mod 11).
Fast inverses. a^{-1} = a^{p-2} mod p, the one-liner of Chapter 17.
"Compute 2^{2^{100}} mod 10⁹+7" is a two-level reduction: the exponent 2^{100} reduces mod φ(10⁹+7) = 10⁹+6, and to compute that you reduce 100 in the exponent mod φ(10⁹+6). Recognising that towers require recursion down the totient chain — and that the chain reaches 1 in O(log m) steps — is the whole answer.
- Theory
Prove Fermat's little theorem by the permutation argument, then by induction on a.
Solution
Permutation: the map x ↦ ax on {1,…,p-1} is a bijection (injective since a is invertible mod p; the set is finite). Hence ∏(ai) ≡ ∏ i, i.e. a^{p-1}(p-1)! ≡ (p-1)!, and cancelling (p-1)! (coprime to p) gives a^{p-1} ≡ 1. ∎
Induction on a for the form aᴵ ≡ a: base a=0 trivial. Step: (a+1)ᴵ = ∑_k C(p,k)aᵏ. For 0 < k < p, p | C(p,k) because C(p,k) = p!/(k!(p-k)!) has a p in the numerator and none in the denominator. So (a+1)ᴵ ≡ aᴵ + 1 ≡ a + 1 by the hypothesis. ∎
The second proof is the "freshman's dream" (x+y)ᴵ ≡ xᴵ+yᴵ (mod p) — which is genuinely true in characteristic p, and is the Frobenius endomorphism.
- Theory
Compute φ(1000), φ(2^{10}), φ(p q) for distinct primes, and φ(n) for n = 2^a.
Solution
- φ(1000) = 1000(1-1/2)(1-1/5) = 1000 · 1/2 · 4/5 = 400.
- φ(2^{10}) = 2^{10} - 2⁹ = 512.
- φ(pq) = (p-1)(q-1) — the RSA quantity.
- φ(2^a) = 2^{a-1}, i.e. exactly the odd residues.
The last one has a practical reading: modulo a power of two, the units are exactly the odd numbers, which is why multiplying by an odd constant is an invertible mixing step in hash functions on 64-bit words, and why an even multiplier destroys information.
- Theory
Find ord_{13}(2) and ord_{13}(3). Which is a primitive root?
Solution
φ(13) = 12, so orders divide 12: possible values 1,2,3,4,6,12.
2¹=2, 2²=4, 2³=8, 2⁴=3, 2⁵=6, 2⁶=12, 2⁷=11, 2⁸=9, 2⁹=5, 2^{10}=10, 2^{11}=7, 2^{12}=1. Order 12 — so 2 is a primitive root mod 13.
3¹=3, 3²=9, 3³=1. Order 3 — not a primitive root.
Efficient method: test only a^{12/q} for each prime q | 12, i.e. q ∈ {2,3}. For 2: 2⁶ = 12 ≠ 1 and 2⁴ = 3 ≠ 1, so the order is not a proper divisor — it is 12. Two exponentiations instead of twelve.
- Theory
Compute 7^{100} mod 13 and 2^{1000000} mod 1000.
Solution
First: φ(13)=12, 100 = 8·12+4, so 7^{100} ≡ 7⁴. 7²=49≡10, 7⁴≡100≡9 (mod 13). Answer 9.
Second: gcd(2,1000) = 8 ≠ 1, so plain Euler is invalid. Split by CRT: mod 8, 2^{10⁶} ≡ 0. Mod 125, φ(125)=100 and gcd(2,125)=1, so 2^{10⁶} ≡ 2^{0} = 1. Solve x ≡ 0 (mod 8), x ≡ 1 (mod 125): x = 376. Answer 376.
Applying φ(1000)=400 naively would give 2^{0}=1 — wrong, precisely because gcd(2,1000) ≠ 1. This is the trap the chapter's Principle box warns about.
- Theory
Prove that aᵐ ≡ 1 (mod n) iff ord_n(a) | m.
Solution
Let k = ord_n(a). (⟸) If m = kt then aᵐ = (aᵏ)ᵗ ≡ 1.
(⟹) Write m = qk + r with 0 ≤ r < k. Then 1 ≡ aᵐ = (aᵏ)ⁿaₕ ≡ aₕ. Minimality of k forces r = 0. ∎
Corollary: ord_n(a) | φ(n), since a^{φ(n)} ≡ 1 by Euler. This is why order-finding only needs to test divisors of φ(n) — and, in Shor's algorithm, why finding the order of a random a modulo n leads directly to a factorisation of n.
- Theory
Show that the decimal expansion of 1/n (for gcd(n,10)=1) is purely periodic with period ord_n(10).
Solution
The expansion repeats with period k iff 10ᵏ · 1/n - 1/n is an integer, i.e. n | 10ᵏ-1, i.e. 10ᵏ ≡ 1 (mod n). The smallest such k is by definition ord_n(10). ∎
Examples: 1/7 = 0.142857 with ord_7(10)=6 = φ(7), so 7 is a "full reptend" prime (10 is a primitive root mod 7). 1/11 = 0.09, period 2. 1/3 = 0.3, period 1.
When gcd(n,10) ≠ 1 the expansion has a non-repeating prefix of length max(v₂(n), v₅(n)) followed by a period of ord_{n'}(10) where n' is n with its 2s and 5s removed — e.g. 1/6 = 0.16.
- Theory
Explain why RSA decryption works, including the case gcd(m,n) > 1.
Solution
With n = pq, ed ≡ 1 (mod φ(n)), so ed = 1 + k(p-1)(q-1).
Coprime case: m^{ed} = m · (m^{φ(n)})^k ≡ m (mod n) by Euler.
Non-coprime case (say p | m): then m^{ed} ≡ 0 ≡ m (mod p) trivially. Mod q, gcd(m,q)=1 so Fermat gives m^{ed} = m(m^{q-1})^{k(p-1)} ≡ m (mod q). Agreeing mod both primes, CRT gives m^{ed} ≡ m (mod n). ∎
So RSA is correct for every message, not just coprime ones — a detail often skipped, and one that matters because m = 0 and multiples of p do occur. (Encountering such an m would also factor n, so it never happens in practice for random messages.)
- Program
Implement φ(n) for a single n ≤ 10¹⁸ using Pollard's rho, and for all n ≤ 10⁷ by sieve. Compare.
Solution
def phi_single(n): # O(n^{1/4}) with Pollard r = n for p in set(pollard_factorise(n)): r -= r // p return r def phi_sieve(N): # O(N log log N) phi = list(range(N+1)) for p in range(2, N+1): if phi[p] == p: for j in range(p, N+1, p): phi[j] -= phi[j]//p return phiOne value at n = 10¹⁸: microseconds with Pollard, impossible by sieve (would need 10¹⁸ memory). All values to 10⁷: 0.1 s by sieve, 10⁷ Pollard calls would take minutes.
This is the general shape of number-theoretic algorithm choice: one huge input means factorisation-based, many small inputs means sieve. The crossover is when the range fits in memory.
- Program
Compute the multiplicative order of a mod n efficiently, given the factorisation of φ(n).
Solution
def order(a, n): from math import gcd assert gcd(a, n) == 1 k = phi(n) for p, e in factorise(k).items(): while k % p == 0 and pow(a, k//p, n) == 1: k //= p return kStart with φ(n), which the order divides, and remove each prime factor as long as the smaller exponent still gives 1. The result is the minimal such divisor. Cost: O(log² n) modular exponentiations at worst — vastly better than trying every divisor of φ(n), which could be tens of thousands.
Correctness: the loop maintains the invariant "aᵏ ≡ 1 and the true order divides k". Removing p when a^{k/p} ≡ 1 preserves it; when no removal applies, k is minimal because any proper divisor of k omits at least one prime power, and the loop already checked that direction.
- Program
Compute a power tower a₁^{a₂^{···^{aₖ}}} mod m correctly, handling non-coprime bases.
Solution
def tower(a, i, m): """value of a[i]^a[i+1]^... mod m, using the generalised Euler theorem""" if m == 1: return 0 if i == len(a) - 1: return a[i] % m ph = phi(m) e = tower(a, i+1, ph) # generalised Euler: if the true exponent ≥ log2(m), add phi(m) if is_big(a, i+1, ph): # true exponent ≥ log2(m)? e += ph return pow(a[i], e, m)The essential subtlety: aⁿ ≡ a^{n mod φ(m)} (mod m) requires gcd(a,m)=1. Without it, use aⁿ ≡ a^{(n mod φ(m)) + φ(m)}, valid whenever n ≥ log₂m. Since a tower of height ≥ 2 with bases ≥ 2 already exceeds log₂m for any practical m, the +φ(m) is applied unconditionally in most implementations — the exception being tiny towers, which must be handled separately.
Termination: the chain m, φ(m), φ(φ(m)), … reaches 1 in O(log m) steps (each step at least halves the value once it is even), so recursion depth is small even for a tower of height 10⁵.
- Program
Implement RSA end to end: key generation, encryption, decryption, and CRT-accelerated decryption. Measure the speed-up.
Solution
def keygen(bits=1024): p = random_prime(bits); q = random_prime(bits) n = p*q; ph = (p-1)*(q-1) e = 65537 d = pow(e, -1, ph) dp, dq = d % (p-1), d % (q-1) qinv = pow(q, -1, p) return (n, e), (n, d, p, q, dp, dq, qinv) def decrypt_crt(c, priv): n, d, p, q, dp, dq, qinv = priv m1 = pow(c % p, dp, p) m2 = pow(c % q, dq, q) h = (qinv * (m1 - m2)) % p return (m2 + h*q) % nSpeed-up ≈ 4×: two exponentiations with half-size moduli and half-size exponents. Modular exponentiation is O(log e · M(log n)) and multiplication is roughly quadratic, so halving both the exponent and the modulus gives 2 × 1/2 × 1/4 = 1/4 of the work.
Security note: CRT-RSA is vulnerable to the Bellcore fault attack — if a hardware glitch corrupts m₁ but not m₂, then gcd(m^e - c, n) = q reveals the factorisation from a single faulty signature. Real implementations verify the result by re-encrypting before releasing it.
- Program
Find all n for which φ(n) = k, for a given k ≤ 10⁶ (the inverse totient problem).
Solution
def inverse_totient(k): # any prime p in a solution has (p-1) | k, so enumerate candidate primes cands = [d+1 for d in divisors(k) if is_prime(d+1)] out = [] def rec(idx, rem, cur): if rem == 1: out.append(cur); return for i in range(idx, len(cands)): p = cands[i] if (p-1) > rem: continue if rem % (p-1): continue r, v = rem // (p-1), cur * p rec(i+1, r, v) # p to the first power while r % p == 0: # higher powers of p r //= p; v *= p rec(i+1, r, v) rec(0, k, 1) return sorted(out)Key facts making this tractable: every prime p dividing a solution satisfies (p-1) | k, so the candidate set has size at most d(k); and the exponents are bounded because p^{a-1} | k.
Not every k is a totient value — φ(n) is even for n > 2, so every odd k > 1 has no preimage, and Carmichael's conjecture (open) says no k has exactly one preimage. Testing that conjecture computationally for k ≤ 10⁶ is a good use of this code.
- Program
Given n ≤ 10⁵, compute ∑_{i=1}^{n} gcd(i, n) in O(√n).
Solution
def gcd_sum(n): total = 0 for d in divisors(n): # d ranges over divisors of n total += d * phi(n // d) # exactly φ(n/d) values of i with gcd = d return totalDerivation: group the terms by g = gcd(i,n). As shown in Chapter 19, exactly φ(n/g) values of i ∈ [1,n] have gcd(i,n)=g. So ∑ gcd(i,n) = ∑_{d|n} d · φ(n/d) = (id * φ)(n). ∎
Since it is a convolution of two multiplicative functions it is itself multiplicative, with f(pᵃ) = (a+1)pᵃ - a p^{a-1} — so it can also be computed directly from the factorisation in O(√n) or, for many queries, sieved in O(N log N).
- Program
Write a test that finds Carmichael numbers below 10⁶ — composites n with a^{n-1} ≡ 1 (mod n) for all a coprime to n.
Solution
def is_carmichael(n): if is_prime(n) or n % 2 == 0: return False # Korselt's criterion: n squarefree and (p-1) | (n-1) for every prime p | n f = factorise(n) if any(e > 1 for e in f.values()): return False return all((p-1) % 1 == 0 and (n-1) % (p-1) == 0 for p in f) carms = [n for n in range(3, 10**6, 2) if is_carmichael(n)] print(len(carms), carms[:5]) # 43 of them; 561, 1105, 1729, 2465, 2821There are 43 Carmichael numbers below 10⁶, the smallest being 561 = 3 · 11 · 17. Every one has at least three prime factors (a squarefree n=pq cannot satisfy Korselt).
Why they matter: they defeat the Fermat primality test entirely — a Carmichael number passes for every coprime base, so no amount of random base choice helps. Alford, Granville and Pomerance proved there are infinitely many. This is precisely why Miller–Rabin (Chapter 23) uses the stronger square-root condition rather than plain Fermat.
Binary exponentiation and its generalisations
Squaring repeatedly computes a^n in log n steps, and the same skeleton works for any associative operation — matrices, permutations, min-plus semirings, geometric transforms. Recognising "this is exponentiation over a different monoid" is the single most transferable trick in this part.
Computing aⁿ by n-1 multiplications is unnecessary. Write n in binary and square:
long long power(long long a, long long e, long long m){
long long r = 1; a %= m;
while (e > 0){
if (e & 1) r = (__int128)r * a % m; // this bit is set
a = (__int128)a * a % m; // square for the next bit
e >>= 1;
}
return r;
}Cost: exactly ⌊log₂n⌋ squarings and popcount(n) - 1 multiplications, so between log₂n and 2log₂n operations. For a 64-bit exponent that is at most 127 multiplications — instead of 1.8 × 10¹⁹.
Correctness by the loop invariant: at every point, r · a^{e} = a₀^{e₀} where a₀, e₀ are the original values. The body preserves it: if the low bit is set, moving one factor of a into r and halving e keeps the product; squaring a compensates for the halving.
The generalisation that matters
Nothing in the algorithm uses commutativity, division, or numbers. It needs only an associative binary operation with an identity — a monoid. So the same six lines compute:
| Monoid | Operation | Identity | Computes |
|---|---|---|---|
| integers mod m | a · b mod m | 1 | aⁿ |
| k×k matrices | matrix product | I | linear recurrences (Ch. 48) |
| permutations | composition | identity perm | σⁿ, cycle structure |
| (min, +) matrices | cᵢ₃ = min_k(aᵢₖ+bₖ₃) | ∞ off-diagonal | shortest paths with exactly n edges |
| strings | concatenation | ε | repeated string |
| 2×2 affine maps | composition | identity | iterated linear map |
| polynomials mod f | mult. mod f | 1 | xⁿ mod f, Kitamasa |
Whenever a problem says "apply this operation 10¹⁸ times", ask whether the operation is associative. If it is, binary exponentiation over that monoid turns O(n) into O(log n) multiplications. If the state is a fixed-size vector and the step is linear, the monoid is matrices and the answer is Chapter 48.
Variants and refinements
Right-to-left vs left-to-right. The code above is right-to-left (LSB first) and needs a running a. Left-to-right scans bits from the top and needs only r, which is preferable when the base is huge (fewer large intermediates) and is what gmp uses.
Windowing. Precompute a¹, a³, a⁵, …, a^{2ᵏ-1} and process k exponent bits at a time. This reduces multiplications from log n / 2 to about log n / (k+1) at the cost of 2^{k-1} precomputed values — a 20–30% saving for cryptographic sizes.
Constant time. The branch if (e & 1) leaks the exponent bits through timing and power analysis. Cryptographic implementations use a Montgomery ladder, which performs the same operations regardless of the bit:
// invariant: (r0, r1) = (a^k, a^{k+1}) for the prefix k read so far
r0 = 1; r1 = a;
for (int i = bits - 1; i >= 0; i--){
if ((e >> i) & 1) { r0 = r0*r1 % m; r1 = r1*r1 % m; }
else { r1 = r0*r1 % m; r0 = r0*r0 % m; }
}
return r0; // same operation count on every path"Compute xⁿ" is a warm-up; the real question is the follow-up. Handle n < 0 (return 1/x^{|n|}, or a modular inverse), handle n = 0 (return 1, including 0⁰ by convention), and mention overflow: n = INT_MIN makes -n overflow, which is a real bug in many published solutions.
- Theory
Count the exact number of multiplications binary exponentiation uses for n = 1000, and find the n below 1000 needing the most.
Solution
1000 = 1111101000₂: 10 bits, so 9 squarings, and popcount = 6 so 5 extra multiplications — 14 operations.
The worst case for a given bit length is all-ones. Below 1000 that is 511 = 111111111₂: 8 squarings + 8 multiplications = 16, and 1023 would need 18.
Note binary exponentiation is not always optimal. a^{15} takes 6 operations by this method but only 5 via a³ → a⁶ → a^{12} → a^{15}. The minimal count is the addition-chain problem, NP-hard in general, and binary exponentiation is within a factor of 2 of optimal.
- Theory
Prove the loop invariant of the iterative algorithm.
Solution
Invariant: r · a^{e} = a₀^{e₀}, where a, e, r are the current values.
Initialisation: r = 1, a = a₀, e = e₀, so 1 · a₀^{e₀} = a₀^{e₀}. ✓
Maintenance: if e is even, the body sets a' = a², e' = e/2, and r' = r; then r'a'^{e'} = r(a²)^{e/2} = raᵉ. ✓ If e is odd, additionally r' = ra and e' = (e-1)/2; then r'a'^{e'} = ra(a²)^{(e-1)/2} = raᵉ. ✓
Termination: e strictly decreases (halving) and is a non-negative integer, so the loop ends at e = 0, where the invariant reads r · 1 = a₀^{e₀}. ∎ Both a variant and an invariant, exactly the Chapter 3 template.
- Theory
How would you compute aⁿ when n is given as a decimal string of 10⁶ digits?
Solution
Two approaches.
(a) Reduce the exponent. If gcd(a,m)=1, compute e = n mod φ(m) by Horner over the digit string (O(|n|)), then one ordinary exponentiation. If not coprime, use the generalised Euler theorem with the +φ(m) correction.
(b) Horner in the exponent. Process digits left to right maintaining r = a^{prefix}: for each digit d, r arrow r^{10} · a^d. Cost O(|n| log m) — no totient needed, and correct even when gcd(a,m)>1.
Method (b) is 4 × 10⁷ multiplications for a million digits — a few seconds, and worth knowing because it sidesteps the coprimality condition entirely.
- Theory
Show that computing Fₙ via matrix power is O(log n) multiplications, and write out the matrix identity.
Solution
⎡1 1⎤ⁿ ⎡F_{n+1} F_n ⎤ ⎢ ⎥ = ⎢ ⎥ ⎣1 0⎦ ⎣F_n F_{n−1}⎦Proof by induction: true at n = 1. Multiplying by the matrix once applies (x,y) ↦ (x+y, x), which is the Fibonacci recurrence, so the entries advance by one index. ∎
Binary exponentiation on 2×2 matrices costs 8 scalar multiplications per matrix product, so O(log n) matrix products = O(log n) scalar operations with a constant near 12. At n = 10¹⁸ that is about 60 matrix products — instant, where the iterative loop would take 10¹⁸ steps.
The fast-doubling identities F_{2k} = F_k(2F_{k+1}-F_k) and F_{2k+1} = F_k²+F_{k+1}² do the same thing with 3 multiplications per step instead of 8 — the specialised version of the same idea.
- Theory
Explain how to compute the n-th power of a permutation in O(k) rather than O(k log n).
Solution
Decompose the permutation into disjoint cycles. Within a cycle of length L, applying the permutation n times moves each element forward by n mod L positions. So:
for each cycle c of length L: shift = n % L for i in range(L): result[c[i]] = c[(i + shift) % L]O(k) total, since the cycles partition the k elements and each is processed once. Binary exponentiation would be O(k log n) — asymptotically worse and with a much larger constant.
Corollary: the order of a permutation is the lcm of its cycle lengths, which can be astronomically large (Landau's function: the maximum order of a permutation of 100 elements is 232,792,560). That is why reducing n per cycle, not globally, is the right move.
- Theory
Why is the naive
if (e & 1)branch a security problem, and how does the Montgomery ladder fix it?Solution
The branch makes the number of operations depend on popcount(e), and which operations run depends on each bit. An attacker measuring total time learns the Hamming weight; measuring a power trace or cache timing per iteration can recover individual bits, and recovering the private exponent breaks RSA entirely. This is not theoretical — practical timing attacks on OpenSSL's RSA have been demonstrated over a network.
The Montgomery ladder performs exactly one multiplication and one squaring per bit regardless of its value; only which register receives the result differs, and that can be made branch-free with conditional swaps (
cmov) so the memory access pattern is identical too.Additional defences used in practice: exponent blinding (compute a^{d + rφ(n)} for random r) and base blinding (compute (ar^e)^d / r), which randomise the observable behaviour even if some leakage remains.
- Theory
Give a monoid whose exponentiation solves "shortest path using exactly n edges", and state the complexity.
Solution
Use the (min, +) tropical semiring: matrices over ℝ ∪ {∞} with product (A ⊗ B)_{ij} = min_k (A_{ik} + B_{kj}) and identity the matrix with 0 on the diagonal and ∞ elsewhere.
The operation is associative (both sides of (A⊗ B)⊗ C equal min_{k,l}(A_{ik}+B_{kl}+C_{lj})), so binary exponentiation applies. (W^{⊗ n})_{ij} is the minimum weight of a walk from i to j using exactly n edges.
Complexity: O(V³ log n). At V = 100 and n = 10¹⁸ that is 6×10⁷ operations — feasible, and there is no other reasonable way to answer the question.
The same construction with (max,+) gives longest walks, and with (+,×) over booleans gives reachability in exactly n steps.
- Program
Implement
power(a, n)for integer, matrix and permutation bases with one generic function.Solution
def monoid_pow(x, n, mul, identity): """n >= 0; mul must be associative with the given identity.""" r = identity while n: if n & 1: r = mul(r, x) x = mul(x, x) n >>= 1 return r # integers mod m monoid_pow(a, n, lambda p,q: p*q % m, 1) # k×k matrices mod m monoid_pow(A, n, lambda P,Q: matmul(P,Q,m), eye(k)) # permutations (as tuples) monoid_pow(s, n, lambda p,q: tuple(p[i] for i in q), tuple(range(k))) # (min,+) matrices monoid_pow(W, n, tropical_mul, tropical_identity(k))Writing it once as a monoid makes the abstraction explicit and prevents four separate off-by-one bugs. The one caveat:
mulmust be genuinely associative —lambda p,q: p-qwould produce silent nonsense, and there is no way for the function to check.The order of arguments in the permutation composition matters and is the usual bug:
tuple(p[i] for i in q)means "apply q then p", and swapping it computes the inverse power for non-commuting elements. - Program
Compute Fₙ mod 10⁹+7 for n ≤ 10¹⁸ by matrix power and by fast doubling. Benchmark.
Solution
def fib_doubling(n, m): """returns (F(n), F(n+1)) mod m""" if n == 0: return (0, 1) a, b = fib_doubling(n >> 1, m) c = a * ((2*b - a) % m) % m # F(2k) = F(k)(2F(k+1) - F(k)) d = (a*a + b*b) % m # F(2k+1) = F(k)² + F(k+1)² return (d, (c+d) % m) if n & 1 else (c, d)Fast doubling uses 3 modular multiplications per bit; the 2×2 matrix version uses 8 (or 12 with the general routine). Measured: about 2.5× faster, and it needs no matrix code at all.
Watch the
(2*b - a) % m: in a language with truncating%this can go negative and must be normalised. In Python it is safe. Both versions are O(log n), so at n = 10¹⁸ either answers in microseconds. - Program
Implement modular exponentiation with 4-bit windowing and measure the improvement over the plain version for 2048-bit exponents.
Solution
def pow_window(a, e, m, k=4): # precompute a^1, a^3, a^5, ..., a^{2^k - 1} (odd powers only) tbl = {1: a % m} a2 = a*a % m for i in range(3, 1<<k, 2): tbl[i] = tbl[i-2] * a2 % m r, i = 1, e.bit_length() - 1 while i >= 0: if not (e >> i) & 1: r = r*r % m; i -= 1 else: lo = max(0, i-k+1) while not (e >> lo) & 1: lo += 1 # window must end in a 1 bit w = (e >> lo) & ((1 << (i-lo+1)) - 1) for _ in range(i-lo+1): r = r*r % m r = r * tbl[w] % m i = lo - 1 return rFor a 2048-bit exponent: plain binary uses 2047 squarings + ~1024 multiplications = 3071 operations. Sliding-window with k = 4 uses 2047 squarings + ~410 multiplications + 8 precomputation = about 2465 — a 20% saving. At k = 5 the saving is 23% but precomputation costs 16 values; the optimum k grows slowly with exponent size, and k = 5 is standard for RSA-2048.
Note the "window must end in a 1 bit" loop: only odd powers are tabulated, halving the table and improving the average window utilisation.
- Program
Given a string s and n ≤ 10¹⁸, compute the hash of s repeated n times, in O(|s| + log n).
Solution
def repeat_hash(s, n, B, M): h = 0 for c in s: h = (h*B + ord(c)) % M # H(s) L = pow(B, len(s), M) # B^{|s|} # H(s^n) = H(s) * (L^{n-1} + L^{n-2} + ... + 1) = H(s) * geometric sum if L == 1: g = n % M else: g = (pow(L, n, M) - 1) * pow(L-1, M-2, M) % M return h * g % MThe hash of a concatenation is H(xy) = H(x)B^{|y|} + H(y), so repeating n times gives H(s)∑_{i=0}^{n-1} L^i — a geometric series, closed-form in O(log n) via modular exponentiation and one inverse.
The L = 1 case must be handled separately, since the closed form divides by L-1 = 0. It occurs when B^{|s|} ≡ 1, which is not rare — it happens whenever |s| is a multiple of ord_M(B). Missing this branch is a subtle bug that only fires on specific input lengths.
- Program
Solve: a linear recurrence a_i = c₁a_{i-1} + … + cₖa_{i-k} with k ≤ 100, find a_n for n ≤ 10¹⁸. Give both the matrix and the Kitamasa method.
Solution
# Matrix: O(k³ log n) — k=100 gives 10⁶·60 = 6×10⁷, fine # Kitamasa / polynomial: O(k² log n) — 10⁴·60 = 6×10⁵, much better def kitamasa(c, init, n, mod): k = len(c) # f(x) = x^k - c1 x^{k-1} - ... - ck (characteristic polynomial) f = [(-c[k-1-i]) % mod for i in range(k)] + [1] def polymul(a, b): # multiply then reduce mod f r = [0]*(len(a)+len(b)-1) for i, x in enumerate(a): if x: for j, y in enumerate(b): r[i+j] = (r[i+j] + x*y) % mod for i in range(len(r)-1, k-1, -1): # reduce degree if r[i]: for j in range(k): r[i-k+j] = (r[i-k+j] - r[i]*f[j]) % mod r[i] = 0 return r[:k] res = monoid_pow([0,1] + [0]*(k-2), n, polymul, [1] + [0]*(k-1)) return sum(res[i]*init[i] for i in range(k)) % modThe insight: a_n = ∑_i r_i a_i where x^n ≡ ∑ r_i x^i (mod f(x)) and f is the characteristic polynomial. So the problem becomes computing x^n in the ring ℤ_m[x]/f(x) — a monoid, so binary exponentiation applies, at O(k²) per multiplication instead of O(k³).
With NTT (Chapter 51) the polynomial product drops to O(k log k), giving O(k log k log n) overall — the Bostan–Mori algorithm, which handles k = 10⁵.
- Program
Write
powerhandling negative exponents, zero, and INT_MIN correctly for both floating-point and modular cases.Solution
double fpow(double a, long long n){ if (n == 0) return 1.0; // including 0^0 = 1 by convention if (n < 0){ // -n overflows when n == LLONG_MIN; widen first unsigned long long m = (unsigned long long)(-(n + 1)) + 1; return 1.0 / fpow_u(a, m); } return fpow_u(a, (unsigned long long)n); } long long mpow(long long a, long long n, long long m){ if (n < 0){ a = modinv(a, m); n = -n; } // requires gcd(a,m)=1 ... }The INT_MIN issue is genuine: in two's complement,
-LLONG_MINis undefined behaviour because the positive value is not representable. The idiom(unsigned long long)(-(n+1)) + 1computes the magnitude without ever overflowing a signed type.Other edge cases worth testing: a = 0 with n < 0 (division by zero — should throw); m = 1 (every result is 0, and
r = 1initialisation must be reduced); and negative a with modular arithmetic (normalise first). Apowerfunction that handles all of these is noticeably rarer than one that handles none. - Program
Implement a segment tree over a non-commutative monoid (matrix product) supporting range query and point update, and use it to answer "state after applying transformations l..r".
Solution
class MonoidSegTree: def __init__(self, xs, mul, e): self.n = 1 while self.n < len(xs): self.n *= 2 self.mul, self.e = mul, e self.t = [e]*(2*self.n) for i, x in enumerate(xs): self.t[self.n+i] = x for i in range(self.n-1, 0, -1): self.t[i] = mul(self.t[2*i], self.t[2*i+1]) # LEFT then RIGHT def set(self, i, x): i += self.n; self.t[i] = x; i //= 2 while i: self.t[i] = self.mul(self.t[2*i], self.t[2*i+1]); i //= 2 def query(self, l, r): # [l, r) resl, resr = self.e, self.e l += self.n; r += self.n while l < r: if l & 1: resl = self.mul(resl, self.t[l]); l += 1 if r & 1: r -= 1; resr = self.mul(self.t[r], resr) l //= 2; r //= 2 return self.mul(resl, resr)The two accumulators
reslandresrare what makes this correct for a non-commutative monoid: left-side fragments must be combined on the left in increasing order, right-side fragments on the right in decreasing order. A single accumulator works for sum or min and silently produces wrong answers for matrices — one of the most common segment-tree bugs.Combined with matrix exponentiation this answers "apply transformations l..r, each repeated k_i times" by storing M_i^{k_i} at each leaf.
The Chinese Remainder Theorem
A number modulo coprime moduli is determined by, and reconstructible from, its residues — so a computation modulo a large number splits into independent computations modulo its prime-power factors. It is both a theorem and a parallelisation strategy.
CRT: if n₁,…,nₖ are pairwise coprime with product N, then the system
has a unique solution modulo N. Equivalently, the ring map ℤ/N → ℤ/n₁ × … × ℤ/nₖ sending x to its tuple of residues is a ring isomorphism.
Uniqueness: two solutions differ by a multiple of every nᵢ, hence (by coprimality) of their product N. Existence: the map is injective between sets of equal size N, hence surjective. That counting argument is the slick proof; the constructive one gives the algorithm.
The construction
This is a "partition of unity": the numbers eᵢ = NᵢMᵢ satisfy eᵢ ≡ 1 (mod nᵢ) and eᵢ ≡ 0 modulo every other, so they form an idempotent basis and x is just a linear combination.
// merge x ≡ a1 (mod m1) with x ≡ a2 (mod m2); handles NON-coprime moduli
pair<long long,long long> crt(long long a1, long long m1,
long long a2, long long m2){
long long p, q, g = extgcd(m1, m2, p, q);
if ((a2 - a1) % g) return {-1, -1}; // inconsistent
long long lcm = m1 / g * m2;
long long t = ((a2 - a1) / g) % (m2 / g) * p % (m2 / g);
long long x = (a1 + (__int128)t * m1) % lcm;
return {(x % lcm + lcm) % lcm, lcm};
}Merging two at a time keeps intermediates below the running lcm, which matters: the closed form multiplies aᵢNᵢMᵢ where Nᵢ may already be near N, overflowing before the final reduction.
When the moduli are not coprime, a solution exists iff aᵢ ≡ a₃ (mod gcd(nᵢ,n₃)) for every pair, and it is then unique modulo lcm(nᵢ). The code above implements exactly this — never assume coprimality unless the problem guarantees it.
What CRT is actually used for
Splitting a computation. To compute something modulo n = pᵃqᵇ, compute it modulo pᵃ and qᵇ separately and recombine. RSA decryption is 4× faster this way (Chapter 20). Binomials modulo a composite are computed per prime power then merged.
Avoiding big integers. A determinant or matrix inverse over ℤ with huge entries can be computed modulo several machine-word primes and reconstructed by CRT plus rational reconstruction — the standard technique in computer algebra, and the reason exact linear algebra is fast.
Convolution modulo an arbitrary number. NTT needs an NTT-friendly prime; to convolve modulo 10⁹+7, run three NTTs modulo three friendly primes whose product exceeds the largest possible coefficient, then CRT.
Counting. "How many x in a range satisfy several congruences" becomes a single congruence modulo the lcm, then a division.
CRT is the reason a hash of a value modulo two different primes is much stronger than one: colliding modulo p₁p₂ requires colliding modulo both, and by CRT those are independent, so the collision probability multiplies. That is the practical content of "use a double hash".
- Theory
Solve x ≡ 2 (mod 3), x ≡ 3 (mod 5), x ≡ 2 (mod 7) by hand.
Solution
N = 105. N₁ = 35, N₂ = 21, N₃ = 15.
M₁ = 35^{-1} mod 3 = 2^{-1} mod 3 = 2; M₂ = 21^{-1} mod 5 = 1^{-1} = 1; M₃ = 15^{-1} mod 7 = 1^{-1} = 1.
x = 2·35·2 + 3·21·1 + 2·15·1 = 140 + 63 + 30 = 233 ≡ 233 - 2·105 = 23 (mod 105).
Check: 23 = 3·7+2 ✓, 23 = 4·5+3 ✓, 23 = 3·7+2 ✓. (This is the classical Sun Tzu problem from the 3rd century.)
- Theory
Prove that ℤ/mn ≅ ℤ/m × ℤ/n as rings when gcd(m,n)=1, and deduce φ(mn) = φ(m)φ(n).
Solution
The map ψ(x) = (x mod m, x mod n) is a ring homomorphism (reduction respects + and ×). Its kernel is {x : m|x and n|x} = mnℤ by coprimality, so it is injective on ℤ/mn; both sides have mn elements, so it is bijective. ∎
A ring isomorphism maps units to units bijectively. The units of a product ring are the pairs of units, so |(ℤ/mn)^×| = |(ℤ/m)^×| · |(ℤ/n)^×|, i.e. φ(mn) = φ(m)φ(n). ∎
This is the structural reason the totient is multiplicative — not a computational coincidence, and the same argument shows why every function defined by counting residues with a local condition is multiplicative.
- Theory
When do x ≡ a (mod m) and x ≡ b (mod n) have a solution if gcd(m,n) = g > 1?
Solution
A solution exists iff a ≡ b ±od g, and then it is unique modulo lcm(m,n).
Necessity: g | m and g | n, so x reduces to a mod g and to b mod g simultaneously.
Sufficiency: write x = a + mt; the second congruence becomes mt ≡ b-a ±od n, solvable iff g | (b-a) (Chapter 15), with n/g distinct t modulo n, giving one x modulo mn/g = lcm(m,n). ∎
Example: x≡1(4) and x≡3(6) — g=2, 1 ≢ 3? In fact 1 ≡ 3 ≡ 1 ±od 2, so it is solvable: x ≡ 9 (mod 12).
- Theory
Use CRT to prove that x² ≡ 1 (mod pq) has exactly 4 solutions for distinct odd primes p, q, and explain the cryptographic consequence.
Solution
By CRT the equation splits into x² ≡ 1 ±od p and x²≡1±od q. Modulo a prime, x²-1 = (x-1)(x+1) ≡ 0 forces x ≡ ±1, giving 2 solutions each (distinct since p > 2). So 2 × 2 = 4 combinations, hence 4 solutions mod pq. ∎
Consequence: two of them are ±1, but the other two are "non-trivial" square roots of 1. If you find such an x, then pq | (x-1)(x+1) while pq ∤ (x±1), so gcd(x-1, pq) is p or q — you have factored n.
This is the engine of Miller–Rabin (a witness is essentially a non-trivial square root of 1) and of Shor's algorithm (order-finding produces one with good probability). It is also why Rabin encryption has four decryptions and needs redundancy to disambiguate.
- Theory
How many x ∈ [1, 10⁶] satisfy x ≡ 3 (mod 7) and x ≡ 5 (mod 11) and x ≡ 1 (mod 13)?
Solution
The moduli are pairwise coprime with product 7·11·13 = 1001, so the system has a unique solution x₀ mod 1001, and the solutions in [1, 10⁶] are x₀, x₀+1001, x₀+2002, ….
The count is ⌊ (10⁶ - x₀)/1001 ⌋ + 1 if x₀ ≤ 10⁶. Since 10⁶/1001 = 999.0, the answer is 999 or 1000 depending on x₀ — solving gives x₀ = 599, and 599 + 999·1001 = 1{,}000{,}598 > 10⁶, so the count is 999.
The general lesson: after CRT, a counting question over an interval is one division. Getting the boundary right (whether x₀ itself and the last term are inside) is the only real work.
- Theory
Explain how to compute C(n,k) mod m for composite m using CRT, and what has to happen at each prime power.
Solution
Factor m = ∏ p_i^{e_i}, compute C(n,k) mod p_i^{e_i} for each, and CRT the results (the prime powers are pairwise coprime).
At each prime power the difficulty is that k! may be divisible by p, so it is not invertible. The fix is Andrew Granville's generalisation of Lucas: define the "p-stripped factorial" (n!)_p = the product of integers up to n not divisible by p, computed with the recursion n! = p^{⌊ n/p⌋}⌊ n/p⌋! · (n!)_p. Track the total exponent of p by Legendre's formula, invert the stripped part (now coprime to p), and multiply back p^{exponent}.
For the special case e = 1 this reduces to Lucas' theorem: C(n,k) ≡ ∏ C(n_i, k_i) ±od p over the base-p digits — much simpler, and enough for most problems.
- Theory
Show that CRT gives a strengthening of the birthday bound for double hashing.
Solution
Hash with two independent primes p₁, p₂. Two distinct strings collide overall iff they collide modulo both. By CRT, a residue pair modulo (p₁, p₂) corresponds bijectively to a single residue modulo p₁p₂, so the effective hash space has size p₁p₂.
With p₁, p₂ ≈ 10⁹, the effective space is 10¹⁸, and the birthday threshold moves from √(10⁹) ≈ 3×10⁴ strings to √(10¹⁸) = 10⁹. That is the difference between a collision being likely in a single test case and being effectively impossible.
The independence assumption matters: if the two hashes share the same base and the primes are related, the CRT argument does not apply. Choose the bases independently at random.
- Program
Implement general CRT for k congruences with possibly non-coprime moduli, returning (x, lcm) or reporting inconsistency.
Solution
def crt(rems, mods): x, m = 0, 1 for a, n in zip(rems, mods): g, p, q = extgcd(m, n) if (a - x) % g: return None # inconsistent lcm = m // g * n t = ((a - x) // g * p) % (n // g) x = (x + m * t) % lcm m = lcm return x % m, mMerging pairwise keeps every intermediate below the running lcm, so with Python integers there is no overflow at all and with 64-bit types it is safe as long as the final lcm fits. The consistency check
(a - x) % gis what makes it correct for non-coprime moduli — omitting it produces plausible-looking wrong answers.Test cases worth keeping: all-coprime (classical CRT), a consistent non-coprime pair, an inconsistent pair, duplicate moduli, and modulus 1 (which constrains nothing).
- Program
Use CRT to compute a large convolution modulo 10⁹+7 with three NTT-friendly primes.
Solution
P = [167772161, 469762049, 998244353] # all c·2^k + 1, primitive root 3 def conv_mod(a, b, mod): res = [ntt_convolve(a, b, p) for p in P] n = len(res[0]); out = [0]*n for i in range(n): x, m = crt([r[i] for r in res], P) out[i] = x % mod return outWhy three primes: a coefficient of the true convolution is at most n · (mod-1)² ≈ 10⁶ × 10¹⁸ = 10²⁴. The product of the three primes is about 7.9 × 10²⁵ > 10²⁴, so CRT recovers the exact integer coefficient before the final reduction. Two primes (≈ 8×10¹⁶) would not suffice and would give silently wrong answers for large inputs.
The alternative is splitting: write each coefficient as a = a_{hi}· 2^{15} + a_{lo} and do four floating-point FFTs of half-magnitude values. Both are standard; the CRT version is exact and easier to reason about.
- Program
Given k ≤ 10⁵ congruences with moduli up to 10⁹, find the smallest positive solution or report none, without big integers where possible.
Solution
def smallest_positive(rems, mods): r = crt(rems, mods) if r is None: return None x, m = r return x if x > 0 else x + m # x==0 means m is the smallest positiveThe catch is that the lcm can explode: with 10⁵ pairwise-coprime moduli near 10⁹, the lcm has about 9 × 10⁵ digits. Two responses:
- Detect early: if the running lcm exceeds a bound (say 10¹⁸) and the problem guarantees the answer fits, you can stop merging once the lcm exceeds the answer bound — every remaining congruence must then be checked, not merged.
- Use big integers if the answer genuinely can be astronomically large; Python handles it, C++ needs a bignum.
Deciding which case you are in requires reading the constraints, and this is a place where "it worked on the samples" and "it is correct" diverge sharply.
- Program
Implement Garner's algorithm — mixed-radix CRT reconstruction — and explain its advantage over the standard formula.
Solution
def garner(rems, mods): """returns coefficients x_i with x = x_0 + x_1*m_0 + x_2*m_0*m_1 + ... (mixed radix)""" k = len(mods) x = [0]*k for i in range(k): cur, prod = rems[i], 1 for j in range(i): cur = (cur - x[j] * prod) % mods[i] prod = prod * mods[j] % mods[i] x[i] = cur * pow(prod, -1, mods[i]) % mods[i] # prod = m_0...m_{i-1} return xAdvantage: every intermediate stays below max m_i, so the whole computation runs in machine words even when the reconstructed number has thousands of digits. The standard formula ∑ a_i N_i M_i requires arithmetic modulo the full product N.
The output is a mixed-radix representation, which is often more useful than the integer: it lets you compare two CRT-represented numbers (compare from the top digit down) and reduce modulo a new modulus by Horner, both without ever materialising the big integer. This is how competitive-programming solutions handle "answer modulo an arbitrary given m" after a multi-prime NTT.
- Program
Solve: find the smallest n such that n leaves remainder i-1 when divided by i, for i = 2..20.
Solution
from math import lcm # n ≡ -1 (mod i) for every i in 2..20 ⟺ n + 1 ≡ 0 (mod lcm(2..20)) L = 1 for i in range(2, 21): L = lcm(L, i) print(L - 1) # 232792559Recognising the uniform shape n ≡ -1 ±od i collapses 19 congruences into one: n+1 is a common multiple of 2..20, so the smallest positive n is lcm(2..20) - 1 = 232{,}792{,}559.
Running general CRT would also work and is a good cross-check, but it is far more code and the moduli are not pairwise coprime — so it exercises exactly the non-coprime branch. That the two agree is a strong test of the CRT implementation.
- Program
Implement exact integer matrix determinant via multi-modular computation and CRT.
Solution
def det_exact(A): n = len(A) # Hadamard bound: |det| ≤ ∏ ||row_i||₂ import math bound = 1 for row in A: bound *= math.isqrt(sum(x*x for x in row)) + 1 primes, prod = [], 1 p = (1 << 31) - 1 while prod <= 2*bound: # 2× to cover negatives p = next_prime_below(p); primes.append(p); prod *= p dets = [det_mod(A, q) for q in primes] # Gaussian elimination mod q x, m = crt(dets, primes) return x - m if x > m // 2 else x # map to the symmetric rangeEach
det_modis O(n³) in machine words with no coefficient growth — the crucial advantage, since fraction-free Gaussian elimination over ℤ produces intermediates with O(n log n) digits and is far slower.The Hadamard bound is what makes the method rigorous: it says how many primes are enough, so the CRT reconstruction is provably exact rather than probably exact. The final line maps the residue into (-m/2, m/2] because the determinant may be negative.
- Program
Write a test that verifies your CRT against brute force over all small moduli combinations.
Solution
import itertools for m1 in range(1, 13): for m2 in range(1, 13): for a1 in range(m1): for a2 in range(m2): sols = [x for x in range(m1*m2) if x % m1 == a1 and x % m2 == a2] got = crt([a1, a2], [m1, m2]) if not sols: assert got is None, (a1,m1,a2,m2,got) else: x, m = got assert m == lcm(m1, m2) assert x % m1 == a1 and x % m2 == a2 assert x == min(s for s in sols if s < m)This 12×12 sweep covers every structural case: coprime, sharing a factor, one modulus dividing the other, modulus 1, and every consistent/inconsistent remainder pair. It takes a fraction of a second and catches the three classic CRT bugs — missing the consistency check, returning the product instead of the lcm, and returning a negative representative.
Exhaustive testing over a small domain is a proof for that domain, and for a function whose logic does not depend on magnitude it is nearly as good as a proof in general — the same argument as Chapter 2 problem 14.
Primality testing and integer factorisation
Deciding whether a number is prime is easy; finding its factors is believed to be hard, and the whole of public-key cryptography rests on that gap. Miller-Rabin settles primality for 64-bit integers with certainty in microseconds; Pollard's rho factors them in milliseconds.
Trial division to √n is O(√n) — fine to 10¹⁴, hopeless for a 2048-bit RSA modulus. Everything below is about getting past that.
The Fermat test and why it fails
If n is prime then a^{n-1} ≡ 1 (mod n) for all a coprime to n. Testing a few random a is a fast compositeness test: any a failing proves n composite.
It is not a primality test, because of Carmichael numbers — composites passing for every coprime base. The smallest is 561 = 3 · 11 · 17, and there are infinitely many. No number of random bases helps.
Miller–Rabin
The repair uses a second fact about primes: modulo a prime, the only square roots of 1 are ±1. Write n - 1 = 2ₛ · d with d odd. If n is prime then for any a either
because the sequence aᵈ, a^{2d}, …, a^{n-1} ends at 1 and each term is the square of the previous — so it must reach 1 through -1, or start at 1. A base for which neither holds is a witness to compositeness, and unlike the Fermat test, witnesses are abundant: at least 3/4 of all bases are witnesses for any composite n. So k random rounds fail with probability under 4^{-k}.
bool is_prime(uint64_t n){
if (n < 2) return false;
for (uint64_t p : {2,3,5,7,11,13,17,19,23,29,31,37})
if (n % p == 0) return n == p;
uint64_t d = n - 1; int s = 0;
while (!(d & 1)) d >>= 1, s++;
for (uint64_t a : {2,3,5,7,11,13,17,19,23,29,31,37}){ // proven sufficient
uint64_t x = power(a, d, n);
if (x == 1 || x == n-1) continue;
bool composite = true;
for (int r = 1; r < s; r++){
x = mulmod(x, x, n);
if (x == n-1) { composite = false; break; }
}
if (composite) return false;
}
return true;
}Those twelve bases are proved sufficient for every n < 3.3 × 10²⁴, so for 64-bit inputs Miller–Rabin is deterministic, not probabilistic. (Seven bases suffice below 3.4 × 10¹⁴; three below 3.2 × 10⁹.)
A Miller–Rabin witness is essentially a non-trivial square root of 1, and finding one factors n via gcd(x-1, n). So the test does not merely detect compositeness — it is one step away from a factorisation, which is why the same idea appears in Shor's algorithm.
Pollard's rho
To factor n, iterate x arrow x² + c mod n — a pseudorandom walk. If p | n, the walk taken modulo p cycles after about √p steps by the birthday bound. When two walk positions collide mod p but not mod n, gcd(|x_i - x_j|, n) is a non-trivial factor.
The walk mod p is invisible to us — but Floyd or Brent cycle detection on the visible sequence finds the collision anyway, because a collision mod p makes the difference divisible by p. Expected cost O(n^{1/4}), so factoring a 64-bit semiprime takes about 2^{16} steps.
uint64_t pollard(uint64_t n){
if (!(n & 1)) return 2;
for (uint64_t c = 1; ; c++){
auto f = [&](uint64_t x){ return (mulmod(x,x,n) + c) % n; };
uint64_t x = 2, y = 2, prod = 1, g = 1;
int steps = 0;
do {
x = f(x); y = f(f(y));
if (x == y) break; // cycle without a factor
prod = mulmod(prod, x > y ? x-y : y-x, n); // batch the gcds
if (++steps % 64 == 0){ g = gcd(prod, n); if (g != 1) break; }
} while (true);
if (g == 0 || g == n) continue; // retry with a new c
if (g != 1) return g;
}
}Batching the gcds — accumulating a product and testing every 64 steps — is what makes rho fast in practice, since gcd is far more expensive than a modular multiplication. The cost is that a batch may capture all factors at once (g = n), which the retry handles.
The landscape
| Method | Cost | Practical range |
|---|---|---|
| Trial division | O(√n) | n ≤ 10¹⁴ |
| Pollard rho | O(n^{1/4}) | n ≤ 10²⁴ |
| Pollard p-1 | fast if p-1 smooth | special-form primes |
| Elliptic curve (ECM) | e^{O(√(log p loglog p))} | factors to ~60 digits |
| Quadratic sieve | e^{O(√(log n loglog n))} | n to ~100 digits |
| Number field sieve | e^{O((log n)^{1/3}···)} | record: 829-bit RSA-250 |
| AKS primality | O(log⁶ n) polynomial | theoretical only |
"Is this number prime?" for a 64-bit input has the answer "deterministic Miller–Rabin with the 12 known bases, O(log³ n), microseconds." The follow-up "why can't you factor it as easily?" wants the honest answer: nobody knows a polynomial algorithm, it is not proven hard, and RSA's security is a conjecture about the difficulty of exactly this asymmetry.
- Theory
Verify that 561 is a Carmichael number using Korselt's criterion.
Solution
Korselt: n is Carmichael iff n is composite, squarefree, and (p-1) | (n-1) for every prime p | n.
561 = 3 · 11 · 17 — composite and squarefree ✓. n - 1 = 560. Check: 2 | 560 ✓, 10 | 560 ✓ (560 = 56·10), 16 | 560 ✓ (560 = 35·16). All conditions hold, so 561 is Carmichael. ∎
Consequence: a^{560} ≡ 1 (mod 561) for every a coprime to 561, so the Fermat test declares it prime for all φ(561) = 320 valid bases. Only a base sharing a factor with 561 (probability 241/560 ≈ 43% among all bases, since many share 3 or 11) exposes it — and by then you have simply found a factor by gcd.
- Theory
Prove that modulo a prime p, the only square roots of 1 are ±1, and show this fails modulo a composite.
Solution
If x² ≡ 1 ±od p then p | (x-1)(x+1). Since p is prime, Euclid's lemma gives p | x-1 or p | x+1, i.e. x ≡ ±1. ∎
Modulo n = 15: 4² = 16 ≡ 1 and 11² = 121 ≡ 1, so there are four square roots {1, 4, 11, 14}. The step that fails is Euclid's lemma — 15 | 3 · 5 without dividing either factor.
And the failure is exploitable: gcd(4-1, 15) = 3. Every non-trivial square root of 1 hands you a factor, which is exactly what Miller–Rabin detects.
- Theory
Prove that at least 3/4 of the bases in [1, n-1] are Miller–Rabin witnesses for any odd composite n.
Solution
Sketch of Rabin's argument. Let n-1 = 2^s d. Define the set of strong liars L = {a : aᵈ ≡ 1 or a^{2^r d} ≡ -1 for some r < s}.
One shows L is contained in a proper subgroup of (ℤ/n)^×. Choose the largest j such that some a has a^{2^j d} ≡ -1, and consider B = {a : a^{2^j d} ≡ ±1 ±od n}, a subgroup containing L. Using CRT and the fact that n has at least two distinct prime power factors, one exhibits an element outside B — so [G : B] ≥ 2. A finer analysis, treating the cases "n has ≥3 prime factors" and "n = pᵃqᵇ" separately, sharpens the index to at least 4. Hence |L| ≤ φ(n)/4 < n/4. ∎
The 3/4 bound is worst case; empirically the fraction of witnesses is far higher, which is why 20 rounds gives a failure probability under 4^{-20} ≈ 10^{-12} — smaller than the chance of a hardware error.
- Theory
Explain why Pollard's rho takes O(n^{1/4}) expected steps.
Solution
Let p be the smallest prime factor, so p ≤ √n. The iteration x ↦ x²+c behaves like a random function on ℤ/p, so by the birthday bound the sequence x_i mod p repeats after about √(π p/2) ≈ 1.25√p steps.
At that collision, p | (x_i - x_j) while n ∤ (x_i - x_j) with high probability, so gcd(x_i-x_j, n) is a proper factor.
Hence expected steps O(√p) = O(n^{1/4}). ∎ At n = 10¹⁸ that is about 3 × 10⁴ iterations — a millisecond. The "rho" name comes from the shape of the walk: a tail leading into a cycle, like the Greek letter.
Note the cost depends on the smallest factor, not on n. A semiprime with two 40-digit factors is hard for rho; a 100-digit number with a 10-digit factor is easy.
- Theory
Describe Pollard's p-1 method and the countermeasure it forces on RSA key generation.
Solution
If p | n and p-1 is B-smooth (all prime factors ≤ B), then (p-1) | B!, so by Fermat a^{B!} ≡ 1 ±od p, and gcd(a^{B!} - 1, n) reveals p. Compute a^{B!} incrementally as a arrow a^k for k = 2,3,…,B, taking gcds periodically.
Countermeasure: use safe primes, where p = 2q+1 with q also prime. Then p-1 = 2q has a huge prime factor, so no feasible B works. FIPS 186 requires that p ± 1 each have a prime factor of at least 100 bits.
The analogous p+1 method (Williams) and ECM generalise this: ECM replaces the group (ℤ/p)^× of fixed order p-1 with an elliptic curve group whose order varies with the curve, so you can retry until you find a smooth one — which is why ECM has no such countermeasure and is the best method for finding medium-sized factors.
- Theory
Given a non-trivial square root of 1 modulo n, show how to factor n.
Solution
Suppose x² ≡ 1 ±od n with x ≢ ±1. Then n | (x-1)(x+1) but n ∤ (x-1) and n ∤ (x+1).
Let g = gcd(x-1, n). Then g > 1 (some prime factor of n divides x-1, otherwise all of n would have to divide x+1) and g < n (since n ∤ x-1). So g is a proper factor. ∎
g = gcd(x-1, n); assert 1 < g < n; print(g, n//g)This is the final step of Shor's algorithm: the quantum part finds the order r of a random a modulo n; if r is even and a^{r/2} ≢ -1, then x = a^{r/2} is a non-trivial square root of 1 and the classical gcd finishes the job. The probability of a usable r is at least 1/2 for random a.
- Theory
Why is AKS, a polynomial-time deterministic primality test, not used in practice?
Solution
AKS (2002) settled a major open problem: PRIMES is in P, unconditionally and deterministically. Its complexity is \tilde{O}(log^{6} n) after improvements (originally \tilde{O}(log^{12} n)).
But Miller–Rabin is O(k log³ n) with a tiny constant, and for 64-bit inputs it is also deterministic thanks to the verified base sets. For cryptographic sizes, 40 rounds of Miller–Rabin give a failure probability below 10^{-24} — far below the probability of a cosmic-ray bit flip during the computation. Practitioners are content with that.
Where determinism is genuinely required (certified primes for standards), the tool is ECPP (elliptic curve primality proving), which is heuristically \tilde{O}(log⁴ n) and produces a short, independently checkable certificate — a much more useful output than a yes/no. AKS remains a landmark theoretical result with no practical niche.
- Program
Implement deterministic Miller–Rabin for 64-bit integers and test it against a sieve for all n < 10⁷.
Solution
def is_prime(n): if n < 2: return False for p in (2,3,5,7,11,13,17,19,23,29,31,37): if n % p == 0: return n == p d, s = n-1, 0 while d % 2 == 0: d //= 2; s += 1 for a in (2,3,5,7,11,13,17,19,23,29,31,37): x = pow(a, d, n) if x == 1 or x == n-1: continue for _ in range(s-1): x = x*x % n if x == n-1: break else: return False return True comp = sieve(10**7) assert all(is_prime(n) == (not comp[n]) for n in range(2, 10**7))The
for…elseis the idiomatic Python for "the inner loop never broke, so this base is a witness". In C++ use an explicit flag, and remembermulmodwith__int128— plainx*x % noverflows for n > 2³².Exhaustive verification to 10⁷ takes a minute and is worth running once: it catches base-set typos, the
s-1off-by-one, and the small-prime special cases in one go. - Program
Implement Pollard's rho with Brent's cycle detection and factor 100 random 64-bit semiprimes. Report timing.
Solution
import random, math def pollard(n): if n % 2 == 0: return 2 while True: x = random.randrange(2, n); y = x c = random.randrange(1, n); d = 1 while d == 1: x = (x*x + c) % n y = (y*y + c) % n; y = (y*y + c) % n d = math.gcd(abs(x - y), n) if d != n: return d def factorise(n): if n == 1: return [] if is_prime(n): return [n] d = pollard(n) return sorted(factorise(d) + factorise(n // d))Timing for 64-bit semiprimes (two ~32-bit primes): about 1 ms each in C++, 30 ms in Python. The n^{1/4} = 2^{16} estimate matches.
Brent's improvement replaces Floyd's tortoise-and-hare with powers-of-two backtracking, cutting the number of
fevaluations by about 25%; combined with batched gcds it is roughly 4× faster overall. Both refinements matter when factoring is in an inner loop. - Program
Factorise 10⁵ integers up to 10¹⁸ as fast as possible, combining sieve, trial division and rho.
Solution
SMALL = sieve_list(10**6) # ~78k primes, precomputed once def factor(n): f = [] for p in SMALL: if p*p > n: break while n % p == 0: n //= p; f.append(p) if n == 1: break if n > 1: if is_prime(n): f.append(n) else: f += factorise_rho(n) # n has ≤ 2 factors, both > 10^6 return fStrategy: strip all factors below 10⁶ by trial division against a precomputed prime list, which handles the overwhelming majority of inputs cheaply. What survives is 1, a prime, or a semiprime with both factors above 10⁶ — and rho dispatches that in O(n^{1/4}).
The trial-division loop is the bottleneck for prime inputs (78,498 divisions). An improvement: run Miller–Rabin first, which costs 12 exponentiations and immediately settles the common "n is prime" case. That reordering alone roughly halves the average time on random inputs.
- Program
Implement Pollard's p-1 and demonstrate it factoring a number chosen to be vulnerable, then show it failing on a safe-prime modulus.
Solution
def pollard_p_minus_1(n, B=10**5): a = 2 for q in primes_upto(B): e = int(math.log(B, q)) a = pow(a, q**e, n) # accumulate q^e into the exponent g = math.gcd(a-1, n) if 1 < g < n: return g return None # vulnerable: p-1 = 2·3·5·7·11·13·17·19·23 + ... (smooth) p = 2*3*5*7*11*13*17*19*23*29*31 + 1 # if prime # safe: p = 2q+1 with q prime — p-1 has a huge factor, so p-1 method failsOn a modulus with a smooth p-1, the method finds p within a second at B = 10⁵. On a safe-prime modulus it exhausts B and returns
None, because p-1 = 2q requires B ≥ q, which is astronomically large.The experiment makes concrete why key-generation standards mandate safe or strong primes: an attacker who can factor your modulus in one second because you picked a convenient-looking prime has broken the whole system, and the defence costs only a slightly slower key generation.
- Program
Write a program that finds the smallest composite passing Miller–Rabin for bases {2}, {2,3} and {2,3,5}.
Solution
def mr_base(n, a): # single-base Miller-Rabin if n % a == 0: return n == a d, s = n-1, 0 while d % 2 == 0: d //= 2; s += 1 x = pow(a, d, n) if x in (1, n-1): return True for _ in range(s-1): x = x*x % n if x == n-1: return True return False comp = sieve(10**8) for bases in ([2], [2,3], [2,3,5]): for n in range(9, 10**8, 2): if not comp[n]: continue if all(mr_base(n, a) for a in bases): print(bases, "first liar:", n); breakAnswers: base {2} first fails at 2047 = 23·89; {2,3} at 1,373,653; {2,3,5} at 25,326,001; {2,3,5,7} at 3,215,031,751.
These thresholds are exactly what licenses the deterministic variants: below 2047, one base suffices; below 3.2 × 10⁹, four bases do; below 3.3×10²⁴, the twelve of the chapter. Computing them yourself is the most convincing way to trust a base set you found in a table.
- Program
Implement a Fermat-test-based factoriser for n = pq where p and q are close, and state when it beats rho.
Solution
def fermat_factor(n): a = math.isqrt(n) if a*a < n: a += 1 while True: b2 = a*a - n b = math.isqrt(b2) if b*b == b2: return (a-b, a+b) a += 1Idea: n = a² - b² = (a-b)(a+b), so search a upward from ⌈√n⌉ until a²-n is a perfect square. If p, q are close, then a = (p+q)/2 is barely above √n and the search ends almost immediately.
Cost is O((p+q)/2 - √n) iterations, which is O((q-p)²/√n). It beats rho whenever |p-q| < n^{3/8} roughly — and it is catastrophic for the victim: an RSA modulus whose primes were generated close together (a real bug, seen in the wild when both primes came from the same short-range search) falls in milliseconds.
Use
math.isqrt, not floating point: at 2048 bits adoublesquare root is meaningless, and the perfect-square test must be exact. - Program
Build a certificate-based primality prover (Pratt certificate) and verify a 64-bit prime.
Solution
def pratt(n): """returns (a, [(q, cert_q), ...]) proving n prime, or None""" if n == 2: return (1, []) f = factorise(n-1) # needs factoring n-1 for a in range(2, 100): if pow(a, n-1, n) != 1: continue if all(pow(a, (n-1)//q, n) != 1 for q in set(f)): return (a, [(q, pratt(q)) for q in set(f)]) return None def verify(n, cert): a, subs = cert if n == 2: return True if pow(a, n-1, n) != 1: return False qs = [q for q, _ in subs] # check the qs really are the prime factors of n-1 m = n-1 for q in qs: while m % q == 0: m //= q if m != 1: return False return all(pow(a, (n-1)//q, n) != 1 for q in qs) and \ all(verify(q, c) for q, c in subs)This is the Lucas primality test: if some a has order exactly n-1 modulo n, then (ℤ/n)^× has n-1 elements, forcing n prime. Verifying the certificate needs only O(log² n) modular exponentiations and no factoring — the hard work is done once by the prover.
The recursion terminates because each q < n, and the certificate has O(log n) nodes. Its existence proves PRIMES is in NP; that it is also in co-NP (a factor is a certificate of compositeness) was strong evidence, long before AKS, that primality is not NP-complete.
Primitive roots, quadratic residues and discrete logarithms
Modulo a prime the units form a cyclic group, so there is a "logarithm" turning multiplication into addition. Computing that logarithm is believed hard, and the gap between the structure existing and being computable is what Diffie-Hellman sells.
A primitive root modulo n is an element of order φ(n) — a generator of the unit group. When one exists, every unit is a power of it, and the group is cyclic.
Existence: primitive roots exist exactly for n = 1, 2, 4, pᵏ, 2pᵏ with p an odd prime. Notably (ℤ/2ᵏ)^× for k ≥ 3 is not cyclic — it is ℤ₂ × ℤ_{2^{k-2}}, generated by -1 and 3.
Count: when they exist there are φ(φ(n)) of them. For a prime p that is φ(p-1) — typically around 0.3p, so random search finds one in a few tries.
long long primitive_root(long long p){
auto fac = distinct_prime_factors(p - 1); // the only factoring needed
for (long long g = 2; g < p; g++){
bool ok = true;
for (long long q : fac)
if (power(g, (p-1)/q, p) == 1) { ok = false; break; }
if (ok) return g;
}
return -1;
}The test is: g is a primitive root iff g^{(p-1)/q} ≠ 1 for every prime q | p-1. Sufficiency holds because the order divides p-1, and any proper divisor is contained in some (p-1)/q.
Quadratic residues
a is a quadratic residue mod p if x² ≡ a has a solution. Exactly (p-1)/2 non-zero residues are, because x ↦ x² is 2-to-1 on the units.
Euler's criterion makes the test one modular exponentiation. Reciprocity makes it O(log p) without any exponentiation at all — a gcd-like recursion, which is how the Jacobi symbol is computed.
To actually find the square root, use Tonelli–Shanks: O(log² p) in general, and when p ≡ 3 ±od 4 it collapses to the one-liner x = a^{(p+1)/4} — because then x² = a^{(p+1)/2} = a · a^{(p-1)/2} = a by Euler.
Discrete logarithm
Given g, h and p, find x with gᵪ ≡ h ±od p. No polynomial algorithm is known, and the best generic method is baby-step giant-step at O(√p) time and space.
long long dlog(long long g, long long h, long long p){
long long m = ceil(sqrt(p));
unordered_map<long long,long long> tbl;
long long cur = h;
for (long long j = 0; j < m; j++){ // baby steps: h·g^j
tbl[cur] = j; cur = cur * g % p;
}
long long gm = power(g, m, p), y = 1;
for (long long i = 1; i <= m; i++){ // giant steps: g^{im}
y = y * gm % p;
if (tbl.count(y)) return i*m - tbl[y];
}
return -1;
}Writing x = im - j with 0 ≤ j < m and 1 ≤ i ≤ m covers every x in [0, p), and the equation becomes g^{im} = h g^{j} — a meet in the middle. Pollard's rho for logarithms achieves the same O(√p) time in O(1) space.
Discrete log is only hard in a well-chosen group. If p-1 factors into small primes, Pohlig–Hellman solves the problem in each prime-power subgroup and CRTs the results, costing O(∑ e_i(log n + √{q_i})) — fast. That is why cryptographic primes are chosen with p-1 having a large prime factor.
Diffie–Hellman: Alice sends gᵃ, Bob sends gᵇ, both compute g^{ab}. An eavesdropper sees g, gᵃ, gᵇ and must compute g^{ab} — believed as hard as discrete log. The interesting answer notes it is unauthenticated and therefore trivially man-in-the-middleable, which is precisely why TLS signs the DH parameters.
- Theory
Find all primitive roots modulo 11 and verify the count formula.
Solution
φ(11) = 10 = 2 · 5. Test g by checking g⁵ ≠ 1 and g² ≠ 1.
2⁵ = 32 ≡ 10 ≠ 1 and 2² = 4 ≠ 1 ✓ — 2 is a primitive root. Its powers generate everything, and 2ᵏ is a primitive root iff gcd(k,10)=1, i.e. k ∈ {1,3,7,9}, giving 2, 8, 7, 6.
Count: φ(φ(11)) = φ(10) = 4 ✓, matching {2, 6, 7, 8}.
- Theory
Prove that (ℤ/p)^× is cyclic for prime p.
Solution
Let m be the maximal order of any element. Claim: every element's order divides m. If some a had order d with d ∤ m, then some prime power qᵉ divides d but not m; combining a suitable power of a with the maximal-order element produces an element of order lcm > m, contradiction.
So every one of the p-1 units satisfies xᵐ = 1. But a polynomial of degree m over the field ℤ/p has at most m roots, so p-1 ≤ m. Since m | p-1 by Lagrange, m = p-1 and the element of that order is a generator. ∎
The proof uses that ℤ/p is a field — a degree-m polynomial having at most m roots fails modulo a composite (x²-1 has four roots mod 15), which is exactly why (ℤ/n)^× need not be cyclic.
- Theory
Use Euler's criterion to decide whether 5 is a QR mod 13 and mod 23.
Solution
Mod 13: 5⁶ mod 13. 5²=25≡12≡-1, so 5⁴ ≡ 1 and 5⁶ = 5⁴·5² ≡ -1. So (5|13) = -1 — not a QR. Check: squares mod 13 are {1,3,4,9,10,12}, and 5 is absent ✓.
Mod 23: 5^{11} mod 23. 5²=25≡2, 5⁴≡4, 5⁸≡16, so 5^{11}=5⁸·5²·5≡16·2·5=160≡160-6·23=22≡-1. Not a QR either.
By reciprocity, since 5 ≡ 1 ±od 4, (5|p) = (p|5). For p=13: 13≡3±od5 and the QRs mod 5 are {1,4}, so (13|5)=-1 ✓ — the same answer with no exponentiation.
- Theory
Show that when p ≡ 3 ±od 4, the square root of a QR a is a^{(p+1)/4}.
Solution
Let x = a^{(p+1)/4}, which is well defined because 4 | p+1 when p ≡ 3 ±od 4. Then
x² = a^{(p+1)/2} = a · a^{(p-1)/2} = a · (a|p) = a · 1 = a
using Euler's criterion and the assumption that a is a QR. ∎ The other root is -x.
This is why cryptographic curve primes are often chosen ≡ 3 ±od 4 — point decompression needs a square root, and this makes it one exponentiation instead of Tonelli–Shanks. Curve25519's field prime 2^{255}-19 is ≡ 5 ±od 8, which has its own (slightly longer) closed form.
- Theory
Explain Pohlig–Hellman and compute the discrete log 3ᵪ ≡ 13 (mod 17) with it.
Solution
p-1 = 16 = 2⁴ — entirely smooth, so Pohlig–Hellman is very fast here: solve the log bit by bit in the 2-power subgroup.
3 is a primitive root mod 17. Determine x mod 2: 13⁸ mod 17. 13²=169≡16≡-1, so 13⁴≡1, 13⁸≡1, so x is even. Continue lifting: x = 2x', and 9^{x'} ≡ 13 where 9 = 3². Working through gives x = 4: check 3⁴ = 81 = 4·17+13 ≡ 13 ✓.
General method: for p-1 = ∏ q_i^{e_i}, find x mod q_i^{e_i} by working in the order-q_i^{e_i} subgroup (BSGS within it, cost √{q_i} per digit), then CRT. Total O(∑ e_i √{q_i}) instead of O(√p) — devastating when p-1 is smooth.
- Theory
Why must a Diffie–Hellman prime satisfy that p-1 has a large prime factor?
Solution
If p-1 = ∏ q_i^{e_i} with all q_i small, Pohlig–Hellman breaks the discrete log in O(∑ e_i(log p + √{q_i})) — polynomial in log p, hence trivially fast. The shared secret is recovered and the exchange is broken.
Standard practice: use a safe prime p = 2q+1 with q prime, and work in the order-q subgroup generated by g = h². Then Pohlig–Hellman gains nothing (the only factors are 2 and q), and the subgroup has no small-order elements to leak information.
Related attack: small-subgroup confinement. If a peer sends an element of small order, the shared secret lies in a tiny set and is guessable. Defence: validate that the received value has order q by checking y^q ≡ 1 and y ≠ 1. RFC 7919's named groups exist precisely so implementations stop generating their own parameters badly.
- Theory
Prove that the product of two non-residues is a residue, and interpret the Legendre symbol as a group homomorphism.
Solution
By Euler's criterion, (ab|p) ≡ (ab)^{(p-1)/2} = a^{(p-1)/2}b^{(p-1)/2} ≡ (a|p)(b|p). So the Legendre symbol is completely multiplicative, and (-1)(-1) = +1: two non-residues multiply to a residue. ∎
Structurally, (·|p) : (ℤ/p)^× → {±1} is a surjective group homomorphism with kernel the residues — so the residues form an index-2 subgroup, confirming there are exactly (p-1)/2 of them.
Practical consequence: you cannot test "is a a residue" by checking whether it is a product of residues — and the fact that (ab|p)=+1 does not mean both are residues is the basis of the Goldwasser–Micali cryptosystem, where the quadratic residuosity problem modulo a composite is the hard problem.
- Program
Find the smallest primitive root modulo every prime below 10⁶ and plot the distribution.
Solution
def smallest_primitive_root(p): if p == 2: return 1 fac = set(prime_factors(p-1)) for g in range(2, p): if all(pow(g, (p-1)//q, p) != 1 for q in fac): return g from collections import Counter c = Counter(smallest_primitive_root(p) for p in primes_below(10**6)) print(c.most_common(10))The distribution is heavily concentrated: 2 is the smallest primitive root for about 37% of primes, 3 for 22%, 5 for 12%, and values above 20 are rare. The smallest primitive root is conjectured to be O(log^{6} p), and under GRH it is O(log⁶ p) — but unconditionally the best bound is a weak O(p^{1/4+ε}).
The practical upshot: a linear search from 2 finds a primitive root essentially immediately, so no cleverness is needed. The expensive part is factoring p-1, which the algorithm requires.
- Program
Implement Tonelli–Shanks for modular square roots and verify on 10⁶ random (a, p) pairs.
Solution
def tonelli(a, p): a %= p if a == 0: return 0 if pow(a, (p-1)//2, p) != 1: return None # not a residue if p % 4 == 3: return pow(a, (p+1)//4, p) # fast path q, s = p-1, 0 while q % 2 == 0: q //= 2; s += 1 z = 2 while pow(z, (p-1)//2, p) != p-1: z += 1 # any non-residue m, c, t, r = s, pow(z, q, p), pow(a, q, p), pow(a, (q+1)//2, p) while t != 1: i, t2 = 0, t while t2 != 1: t2 = t2*t2 % p; i += 1 # order of t is 2^i b = pow(c, 1 << (m-i-1), p) m, c = i, b*b % p t = t*c % p r = r*b % p return rThe invariant is r² = a t with t of 2-power order < 2ᵐ; each iteration strictly decreases m, so it terminates in at most s steps. Total O(log² p), dominated by the inner order-finding loop.
Verify with
assert (r*r - a) % p == 0and also check theNonepath against Euler's criterion — a common bug is returning a bogus root for non-residues, which then poisons everything downstream silently. - Program
Implement BSGS for discrete log and solve gᵪ ≡ h ±od p for p ≈ 10^{12}.
Solution
def bsgs(g, h, p): m = math.isqrt(p - 1) + 1 tbl = {} cur = h % p for j in range(m): tbl.setdefault(cur, j) # setdefault keeps the SMALLEST j cur = cur * g % p gm = pow(g, m, p); y = 1 for i in range(1, m+1): y = y * gm % p if y in tbl: return i*m - tbl[y] return NoneAt p = 10¹², m = 10⁶ — a million-entry hash table, roughly 50 MB in Python, well under a second. At p = 10¹⁸ it needs 10⁹ entries and becomes infeasible on memory, which is where Pollard's rho for logarithms takes over: same O(√p) time, O(1) memory, using a partitioned random walk and Floyd cycle detection.
Use
setdefaultrather than assignment so the table records the smallest j for a repeated value — otherwise the returned exponent may exceed the group order and fail a strict equality check. - Program
Implement Pohlig–Hellman and demonstrate it breaking a discrete log where p-1 is smooth.
Solution
def pohlig_hellman(g, h, p, order_factors): rems, mods = [], [] n = p - 1 for q, e in order_factors.items(): qe = q**e gi = pow(g, n//qe, p) hi = pow(h, n//qe, p) x = bsgs(gi, hi, p) # subgroup of size q^e; BSGS costs √(q^e) rems.append(x); mods.append(qe) return crt(rems, mods)[0]With p-1 = 2^a 3^b 5^c ··· smooth, each subgroup log costs O(√{q^e}) — tiny. A 60-bit prime with fully smooth p-1 falls in milliseconds, versus 2^{30} operations for plain BSGS.
Constructing the demonstration is instructive: pick p = 2 · 3 · 5 ··· 47 · k + 1 and test for primality until it hits. That such primes are easy to construct is exactly why key-generation standards forbid them, and the code makes the danger concrete rather than abstract.
- Program
Implement the Jacobi symbol in O(log n) using reciprocity, and use it as a fast pre-filter before Euler's criterion.
Solution
def jacobi(a, n): assert n > 0 and n % 2 == 1 a %= n; result = 1 while a: while a % 2 == 0: a //= 2 if n % 8 in (3, 5): result = -result # (2|n) a, n = n, a # reciprocity flip if a % 4 == 3 and n % 4 == 3: result = -result a %= n return result if n == 1 else 0The structure is the Euclidean algorithm with sign bookkeeping — O(log n) with no exponentiation, so roughly 30× faster than Euler's criterion.
Important caveat: for composite n the Jacobi symbol being +1 does not imply a is a residue (both factors could be non-residues). So it is a valid filter (Jacobi = -1 proves non-residue) but not a test. For prime n it agrees with Legendre exactly, which is what makes it usable as a fast path in Tonelli–Shanks and in the Solovay–Strassen primality test.
- Program
Implement a toy Diffie–Hellman exchange with a safe prime and demonstrate a small-subgroup attack against an implementation that skips validation.
Solution
q = safe_prime_q() # p = 2q+1, both prime p = 2*q + 1 g = pow(h, 2, p) # generator of the order-q subgroup a = random.randrange(2, q); A = pow(g, a, p) b = random.randrange(2, q); B = pow(g, b, p) assert pow(B, a, p) == pow(A, b, p) # shared secret # ATTACK: Mallory sends y = p-1, which has order 2 y = p - 1 shared = pow(y, a, p) # equals 1 if a even, p-1 if a odd # → one bit of Alice's private key leaks, and the "shared secret" # is one of only two values, so the session key is guessable. # DEFENCE: def validate(y, p, q): return 1 < y < p-1 and pow(y, q, p) == 1The attack recovers a mod 2 from a single exchange. In groups where p-1 has more small factors, repeating with elements of order 3, 5, 7… recovers a modulo each, and CRT reconstructs the private key entirely — the full small-subgroup attack.
The two-line
validatestops it: rejecting 1 and p-1, and checking the element genuinely lies in the order-q subgroup. Real protocols that omitted this (several TLS and IKE implementations) had practical key-recovery vulnerabilities. - Program
Count quadratic residues modulo p for many primes and verify the count and the distribution of consecutive residues.
Solution
for p in primes_between(100, 200): qr = {x*x % p for x in range(1, p)} assert len(qr) == (p-1)//2 # exactly half runs = sum(1 for a in range(1, p-1) if a in qr and a+1 in qr) print(p, len(qr), runs, (p-5)/4)The count is always (p-1)/2 ✓. The number of consecutive residue pairs is (p-4-(-1|p))/4, which is close to p/4 — exactly what you would expect if residues were placed at random, and the count of each of the four consecutive-pattern types (RR, RN, NR, NN) is p/4 + O(1).
This "pseudo-randomness" of the Legendre symbol is quantified by the Weil bound, which says character sums over intervals are O(√p log p) — and it is why the Legendre symbol is used as a source of pseudo-random bits and why the least quadratic non-residue is conjectured to be O(log^{1+ε} p), making the search in Tonelli–Shanks effectively constant time.
The Mobius function and inversion
Mobius inversion is inclusion-exclusion over the divisor lattice. It converts "count the things divisible by d" - which is easy - into "count the things with gcd exactly d" - which is what you actually wanted.
The Möbius function:
Its defining property is the identity
Proof: only squarefree divisors contribute. If n has k distinct primes, the squarefree divisors are the 2ᵏ subsets, and the sum is ∑_{j=0}^{k} C(k,j)(-1)₌ = (1-1)ᵏ = 0 for k ≥ 1, and 1 for k = 0. ∎ That is the binomial theorem, and it is why Möbius inversion is inclusion–exclusion.
Möbius inversion
In convolution language: g = f * 1 \iff f = g * μ, because μ is the Dirichlet inverse of the constant function 1. The second ("divisor-multiple") form is the one used in practice, because "count multiples of d" is usually the easy quantity.
The pattern for almost every coprimality-counting problem: let F(d) = the count of objects where d divides everything relevant (easy — usually a division or a prefix count), and let f(d) = the count where the gcd is exactly d (what you want). Then F = ∑_{d | e} f(e), so f(1) = ∑_d μ(d) F(d). Write down F, sum against μ, done.
# How many pairs (i,j) with 1 ≤ i,j ≤ n have gcd(i,j) = 1?
# F(d) = #pairs where d divides both = ⌊n/d⌋²
# answer = Σ_{d=1..n} μ(d) ⌊n/d⌋²
def coprime_pairs(n):
mu = mobius_sieve(n)
return sum(mu[d] * (n//d)**2 for d in range(1, n+1))
# density: → 6/π² ≈ 0.6079, since Σ μ(d)/d² = 1/ζ(2) = 6/π²That last line is worth remembering: two random integers are coprime with probability 6/π² ≈ 0.608. The general statement ∑_d μ(d)/d^k = 1/ζ(k) gives the probability that k random integers are jointly coprime.
Möbius over other lattices
The divisor version is a special case. For any locally finite poset there is a Möbius function with an inversion formula; the subset lattice gives ordinary inclusion–exclusion, with μ(S,T) = (-1)^{|T \setminus S|}. Recognising them as the same theorem explains why the alternating signs appear in both.
Computing it
vector<int> mobius(int n){
vector<int> mu(n+1, 1); vector<bool> comp(n+1, false);
vector<int> primes;
for (int i = 2; i <= n; i++){
if (!comp[i]) { primes.push_back(i); mu[i] = -1; }
for (int p : primes){
if ((long long)p*i > n) break;
comp[p*i] = true;
if (i % p == 0) { mu[p*i] = 0; break; } // p² divides p*i
mu[p*i] = -mu[i];
}
}
return mu;
}
// Mertens: M(n) = Σ_{k≤n} μ(k). Grows like O(√n) empirically;
// the Mertens conjecture |M(n)| < √n is FALSE (disproved 1985)."Count pairs with gcd 1" and "count subsets whose gcd is 1" are the two standard Möbius questions. The reframing to say aloud: counting things divisible by d is trivial; the Möbius sum converts trivial counts into exact-gcd counts, and it is inclusion–exclusion over the primes.
- Theory
Prove ∑_{d|n} μ(d) = [n=1].
Solution
For n = 1 the only divisor is 1 and μ(1) = 1. For n > 1 with distinct primes p₁,…,pₖ, divisors with a squared factor contribute 0, so only the 2ᵏ squarefree divisors matter. A squarefree divisor is a subset S of the primes, contributing (-1)^{|S|}. Grouping by size,
∑_{j=0}^{k} C(k,j)(-1)₌ = (1 + (-1))ᵏ = 0. ∎
This is exactly the inclusion–exclusion cancellation identity of Chapter 4 problem 2, and it is the reason both techniques produce the same alternating signs.
- Theory
Prove the Möbius inversion formula.
Solution
Assume g(n) = ∑_{d|n} f(d). Then
∑_{d|n} μ(d)g(n/d) = ∑_{d|n} μ(d)∑_{e | (n/d)} f(e) = ∑_{e|n} f(e) ∑_{d | (n/e)} μ(d)
(swapping the order: the pairs (d,e) with de | n are the same set either way). The inner sum is [n/e = 1], which is 1 only when e = n. So the whole thing is f(n). ∎
Structurally: g = f * 1 and μ * 1 = ε, so g * μ = f * 1 * μ = f * ε = f. The theorem is just associativity plus the fact that μ inverts 1.
- Theory
Derive φ(n) = ∑_{d|n} μ(d)n/d from ∑_{d|n}φ(d) = n.
Solution
The hypothesis says id = φ * 1. Inverting, φ = id * μ, i.e.
φ(n) = ∑_{d|n} μ(d) · (n)/(d) = n∑_{d|n}(μ(d))/(d) = n∏_{p|n}(1 - \frac1p),
the last step because the sum is multiplicative and at a prime power only d = 1, p contribute. ∎
So the familiar product formula for the totient is a Möbius inversion in disguise — and this derivation generalises: any multiplicative function defined by a divisor sum has a product formula obtained the same way.
- Theory
Count the number of pairs (i,j) with 1 ≤ i,j ≤ n and gcd(i,j) = k.
Solution
gcd(i,j)=k iff i = ki', j = kj' with gcd(i',j')=1 and i',j' ≤ ⌊ n/k⌋. So the answer is the coprime-pair count for m = ⌊ n/k ⌋:
∑_{d=1}^{m} μ(d)⌊ m/d⌋².
Sanity check by summing over all k: ∑_k #{gcd = k} = n², which is the statement that every pair has some gcd. ✓
Asymptotically the coprime count is (6)/(π²)m² + O(mlog m), so the density of pairs with gcd exactly k is (6)/(π²k²) — and those densities sum to (6)/(π²)·(π²)/(6) = 1 ✓, a satisfying consistency check.
- Theory
Prove that the probability two random integers are coprime is 6/π².
Solution
The probability both are divisible by d is 1/d². By Möbius inversion the probability their gcd is exactly 1 is ∑_{d\ge1} μ(d)/d².
Now ∑_d μ(d)/d^s = 1/ζ(s), because ζ(s)∑_dμ(d)d^{-s} = ∑_n n^{-s}∑_{d|n}μ(d) = ∑_n n^{-s}[n=1] = 1. With s = 2 and ζ(2) = π²/6, the answer is 6/π² ≈ 0.6079. ∎
An independent heuristic gives the same thing: ∏_p (1 - 1/p²) = 1/ζ(2) by the Euler product, i.e. "for each prime, they are not both divisible by it, and the events are independent". That the two derivations agree is the Euler product formula.
- Theory
Count squarefree numbers up to n.
Solution
By inclusion–exclusion over squares of primes: subtract multiples of 4, of 9, of 25, add back multiples of 36, and so on. In Möbius form:
Q(n) = ∑_{d=1}^{√n} μ(d)⌊ n/d²⌋.
Only d ≤ √n contributes, so this is O(√n) after a μ sieve to √n. Asymptotically Q(n) ~ n/ζ(2) = 6n/π² ≈ 0.6079n — the same constant, and for the same reason: "squarefree" and "coprime to a random number" are both "avoid every p²".
At n = 10¹⁸ this is a million terms — instant, where enumerating and testing would be impossible.
- Theory
State the general Möbius function of a poset and specialise it to the subset lattice.
Solution
For a locally finite poset P, define μ(x,x) = 1 and, recursively, μ(x,y) = -∑_{x ≤ z < y} μ(x,z) for x < y. Then
g(y) = ∑_{x ≤ y} f(x) \iff f(y) = ∑_{x≤ y} μ(x,y)g(x).
- Divisor lattice (x ≤ y iff x|y): μ(x,y) = μ(y/x) — the classical case.
- Subset lattice (S ⊆ T): μ(S,T) = (-1)^{|T|-|S|} — ordinary inclusion–exclusion.
- Chain (i ≤ j in ℤ): μ(i,j) = 1, -1, 0 for j-i = 0,1,\ge2 — finite differences.
So inclusion–exclusion, Möbius inversion and the difference operator are one theorem in three lattices. That unification is the point of the general definition.
- Program
Sieve μ(i) for i ≤ 10⁷ and verify ∑_{d|n}μ(d) = [n=1] for all n ≤ 10⁵.
Solution
def mobius_sieve(n): mu = [1]*(n+1); primes = []; comp = bytearray(n+1) for i in range(2, n+1): if not comp[i]: primes.append(i); mu[i] = -1 for p in primes: if p*i > n: break comp[p*i] = 1 if i % p == 0: mu[p*i] = 0; break mu[p*i] = -mu[i] return mu mu = mobius_sieve(10**5) for n in range(1, 10**5+1): s = sum(mu[d] for d in range(1, n+1) if n % d == 0) assert s == (1 if n == 1 else 0), nThe
if i % p == 0: mu[p*i] = 0; breakline is where the squarefree condition enters: if p already divides i, then p² | pi, so μ(pi) = 0, and the break maintains the linear-sieve invariant that each composite is written once by its smallest prime.Verifying the defining identity directly is a strong test: it exercises every branch of the sieve and would catch a sign error immediately, which a spot check of a few values might not.
- Program
Count pairs (i,j), 1 ≤ i < j ≤ n, with gcd(i,j)=1, for n ≤ 10⁷, two ways.
Solution
# (a) Möbius: O(n) def coprime_pairs_mu(n): mu = mobius_sieve(n) return sum(mu[d] * (n//d) * (n//d - 1) // 2 for d in range(1, n+1)) # (b) Totient sum: O(n), often simpler # pairs with i < j and gcd=1 = Σ_{j=2..n} φ(j) def coprime_pairs_phi(n): phi = phi_sieve(n) return sum(phi[j] for j in range(2, n+1))Both are O(n) and must agree. The second is the cleaner derivation: for each j, the valid i are precisely those in [1, j-1] coprime to j, of which there are φ(j).
Asymptotically both give (3)/(π²)n² + O(n log n) — half the ordered-pair count. Having two independent computations that agree to the last digit is the best correctness check available for this kind of code.
- Program
Compute ∑_{i=1}^{n}∑_{j=1}^{m}gcd(i,j) for n,m ≤ 10⁷.
Solution
def gcd_sum_2d(n, m): # Σ gcd(i,j) = Σ_d d · #{(i,j): gcd = d} # = Σ_d d · Σ_e μ(e) ⌊n/(de)⌋⌊m/(de)⌋ # substitute k = de: = Σ_k ⌊n/k⌋⌊m/k⌋ · (Σ_{d|k} d·μ(k/d)) # and the inner sum is exactly φ(k). phi = phi_sieve(min(n, m)) return sum(phi[k] * (n//k) * (m//k) for k in range(1, min(n,m)+1))The derivation collapses a double Möbius sum into a single sum against φ, because id * μ = φ — the identity from problem 3, used in the forward direction. O(n) after the sieve.
For n, m ≤ 10¹², replace the linear sum with divisor blocks over ⌊ n/k⌋ and a memoised prefix-φ function (Chapter 19 problem 14), giving O(n^{2/3}). This is the standard shape of a hard number-theory contest problem: a Möbius derivation to collapse the sum, then divisor blocks to evaluate it.
- Program
Count squarefree integers in [1, n] for n ≤ 10¹⁸.
Solution
import math def squarefree_count(n): r = math.isqrt(n) mu = mobius_sieve(r) return sum(mu[d] * (n // (d*d)) for d in range(1, r+1))O(√n) = 10⁹ at n = 10¹⁸ — a few seconds, and the μ sieve to 10⁹ needs 1 GB as a byte array. Reduce memory by processing in segments, or accept n ≤ 10¹⁴ with a 10⁷ sieve.
Cross-check against the asymptotic 6n/π²: at n = 10¹⁸ the formula predicts 607{,}927{,}101{,}854{,}026{,}628 and the exact count differs by about O(√n) = 10⁹ — a relative error of 10^{-9}, which is exactly the size of the known error term.
- Program
Given an array of n ≤ 10⁵ values up to 10⁶, count subsets whose gcd is exactly 1.
Solution
MOD = 10**9 + 7 MAX = 10**6 + 1 cnt = [0]*MAX for v in a: cnt[v] += 1 mult = [0]*MAX for d in range(1, MAX): for m in range(d, MAX, d): mult[d] += cnt[m] # how many are divisible by d mu = mobius_sieve(MAX-1) ans = 0 for d in range(1, MAX): if mu[d]: ans = (ans + mu[d] * (pow(2, mult[d], MOD) - 1)) % MOD print(ans % MOD)F(d) = 2^{mult[d]} - 1 counts the non-empty subsets whose gcd is a multiple of d — easy, because it only depends on how many array elements d divides. Möbius then extracts "gcd exactly 1".
Cost: O(A log A) for the multiple counts plus O(A log MOD) for the powers — about 3×10⁷ operations. Precomputing all powers of 2 up to n turns the second term into O(A).
- Program
Compute the Mertens function M(n) = ∑_{k≤ n}μ(k) for n ≤ 10¹² and investigate its growth.
Solution
from functools import lru_cache SMALL = 10**7 mu = mobius_sieve(SMALL) pre = [0]*(SMALL+1) for i in range(1, SMALL+1): pre[i] = pre[i-1] + mu[i] @lru_cache(None) def M(n): if n <= SMALL: return pre[n] r, i = 1, 2 # from Σ_{k≤n} M(n//k) = 1 while i <= n: q = n//i; j = n//q r -= (j - i + 1) * M(q) i = j + 1 return rThe recurrence comes from ∑_{d|m}μ(d) = [m=1] summed over m ≤ n: ∑_{k=1}^{n} M(⌊ n/k⌋) = 1. Complexity O(n^{2/3}) with the cutoff tuned.
Growth: M(n) oscillates around 0 with magnitude roughly n^{1/2}. Mertens conjectured |M(n)| < √n; Odlyzko and te Riele disproved it in 1985, though no explicit counterexample is known (the smallest is believed to exceed 10^{16}). Had it been true it would have implied the Riemann hypothesis — RH is equivalent to M(n) = O(n^{1/2+ε}), which remains open.
- Program
Implement a general "count with exact gcd" helper: given F(d) as a callable, return f(1) = ∑_d μ(d)F(d), and apply it to three different problems.
Solution
def exact_gcd_one(F, limit, mu=None): """F(d) counts configurations where d divides everything relevant.""" if mu is None: mu = mobius_sieve(limit) return sum(mu[d] * F(d) for d in range(1, limit+1) if mu[d]) # 1. coprime pairs in [1,n]² exact_gcd_one(lambda d: (n//d)**2, n) # 2. non-empty subsets of an array with gcd 1 exact_gcd_one(lambda d: pow(2, mult[d], MOD) - 1, MAX-1) # 3. lattice points visible from the origin in an n×n grid exact_gcd_one(lambda d: (n//d)**2, n) # same as (1): visible ⟺ gcd=1Factoring out the pattern makes it obvious that these are the same problem with different F — which is the real content of this chapter. Once you can write down "how many configurations have d dividing everything", the Möbius sum is mechanical.
The third example is a nice one to know: a lattice point (x,y) is visible from the origin (no other lattice point on the segment) exactly when gcd(x,y)=1, so the visible fraction of the grid is 6/π² — the same constant yet again.
The counting principles and bijective proof
Almost all of combinatorics is four rules applied carefully. The hard part is never the arithmetic; it is being certain you counted each object exactly once, and the discipline that guarantees it is exhibiting a bijection.
Four principles, each trivial and each the source of a characteristic error.
Sum fails silently when the cases overlap — the repair is inclusion–exclusion (Chapter 30). Product fails when the second choice's count depends on the first; it is fine if the count is the same even though the options differ (choosing a second card from 51 works, whichever card was first). Division fails when the map is not uniformly k-to-1, which is the commonest subtle error in counting arrangements with symmetry.
Bijective proof
The strongest form of a counting argument is a bijection: instead of computing both sides, exhibit a reversible correspondence. It is more convincing, it usually generalises, and it often is an algorithm.
SUBSETS ↔ BINARY STRINGS
S ⊆ {1..n} ↔ the n-bit string with bit i set iff i ∈ S
⟹ |P(A)| = 2ⁿ, and enumeration by bitmask is the bijection made code.
COMPOSITIONS ↔ STARS AND BARS
x₁+...+x_k = n with xᵢ ≥ 0 ↔ arrangements of n stars and k−1 bars
⟹ C(n+k−1, k−1) solutions.
LATTICE PATHS ↔ SEQUENCES OF MOVES
monotone paths (0,0)→(m,n) ↔ strings with m R's and n U's
⟹ C(m+n, m) paths.Stars and bars deserves emphasis because it converts a huge class of problems into one binomial. To place n indistinguishable items into k distinguishable boxes, write n stars and k-1 bars in a row; the bars cut the stars into k groups. Every arrangement gives a solution and vice versa, so the count is C(n+k-1, k-1). With the constraint x_i ≥ 1, first give each box one item and count the rest: C(n-1, k-1).
Before counting, answer two questions explicitly: are the objects distinguishable? and does order matter? The four combinations give four different formulas, and nearly every wrong answer in combinatorics comes from answering one of them wrong rather than from an arithmetic slip.
The twelvefold way
Placing n balls into k boxes, under the four distinguishability choices and three constraints, gives twelve counts — and essentially every elementary counting problem is one cell of this table.
| Balls / Boxes | any | injective (≤1 each) | surjective (≥1 each) |
|---|---|---|---|
| distinct / distinct | kⁿ | k^{\underline{n}} | k!S(n,k) |
| identical / distinct | C(n+k-1,n) | C(k,n) | C(n-1,k-1) |
| distinct / identical | ∑_{j≤ k} S(n,j) | [n≤ k] | S(n,k) |
| identical / identical | p_k(n) | [n≤ k] | p(n,k) |
Here k^{\underline{n}} = k(k-1)···(k-n+1) is the falling factorial, S(n,k) is a Stirling number of the second kind (Chapter 33), and p(n,k) counts partitions of n into exactly k parts.
"How many ways to…" is answered by locating the cell: distinguishable? ordered? repetition allowed? Saying "the balls are distinct and the boxes are distinct with no constraint, so kⁿ" shows a model rather than a recalled formula, and it survives the follow-up where one constraint changes.
- Theory
How many 5-card poker hands are there? How many contain exactly one pair?
Solution
Total: C(52,5) = 2{,}598{,}960.
Exactly one pair: choose the pair's rank (13), choose 2 of its 4 suits (C(4,2)=6), choose 3 distinct other ranks (C(12,3)=220), and a suit for each (4³=64):
13 × 6 × 220 × 64 = 1{,}098{,}240, a probability of about 42.3%.
The trap is choosing the three kickers as C(12,1)C(11,1)C(10,1), which counts each set of kickers 3! = 6 times — the division principle applied wrongly by forgetting to divide. Choosing the set with C(12,3) avoids it entirely.
- Theory
Prove stars and bars: the number of non-negative integer solutions to x₁+···+x_k = n is C(n+k-1, k-1).
Solution
Map each solution to a string of n stars and k-1 bars: write x₁ stars, a bar, x₂ stars, a bar, …, x_k stars.
This is injective: different solutions give different star counts between bars. It is surjective: any arrangement of n stars and k-1 bars decodes to a solution by reading off the gaps (possibly empty). So it is a bijection, and the number of arrangements is the number of ways to choose which k-1 of the n+k-1 positions hold bars: C(n+k-1,k-1). ∎
For x_i ≥ 1, substitute y_i = x_i - 1 ≥ 0 summing to n-k, giving C(n-1,k-1). For x_i ≥ c_i, subtract the lower bounds first — the same substitution handles every lower-bound variant.
- Theory
How many ways to seat n people around a circular table, where rotations are considered the same? What if reflections are also identified?
Solution
Linear arrangements: n!. Each circular arrangement corresponds to exactly n linear ones (the n rotations), so by the division principle the count is n!/n = (n-1)!.
With reflections identified, each circular arrangement is counted twice more, giving (n-1)!/2 for n ≥ 3.
The n ≥ 3 caveat matters: for n = 1, 2 a reflection fixes the arrangement, so the map is not uniformly 2-to-1 and the division principle does not apply. This is exactly the situation Burnside's lemma (Chapter 34) handles correctly in general — division is the special case where no arrangement is fixed by a non-identity symmetry.
- Theory
Count the anagrams of MISSISSIPPI, and state the general multinomial formula.
Solution
11 letters: M×1, I×4, S×4, P×2. Treating them as distinct gives 11! arrangements, but each genuine anagram arises 1!4!4!2! times (permuting identical letters among themselves).
(11!)/(1!4!4!2!) = \frac{39{,}916{,}800}{1152} = 34{,}650.
General multinomial coefficient: with n = n₁+···+n_k objects of k types,
C(n,n₁,…,n_k) = (n!)/(n₁!··· n_k!),
which also counts the ways to split n distinct objects into labelled groups of the given sizes — the same number, by a bijection between arrangements and assignments of positions to types.
- Theory
Give a bijective proof that C(n,k) = C(n,n-k) and that ∑_k C(n,k) = 2ⁿ.
Solution
Symmetry: the map S ↦ {1..n}\setminus S sends k-subsets to (n-k)-subsets, and it is its own inverse — hence a bijection. ∎ No algebra needed, and the identity becomes obvious rather than verified.
Row sum: both sides count subsets of an n-set. The left counts them by size; the right counts them by the bit-string bijection. ∎
This is the value of bijective proof: (n!)/(k!(n-k)!) = (n!)/((n-k)!k!) is a true but uninformative verification, whereas "choosing what to include is the same as choosing what to exclude" explains why and immediately suggests generalisations.
- Theory
How many monotone lattice paths from (0,0) to (m,n) avoid the point (a,b)?
Solution
Total paths: C(m+n, m). Paths through (a,b): C(a+b,a) · C(m-a+n-b, m-a) — reach it, then continue, by the product principle (the two halves are independent given the intermediate point).
Answer: C(m+n,m) - C(a+b,a)C(m+n-a-b, m-a).
Avoiding several points requires inclusion–exclusion, and if the forbidden points form a staircase the alternating sum becomes the Lindström–Gessel–Viennot determinant. Avoiding an entire diagonal is the reflection principle, which produces the Catalan numbers of Chapter 32.
- Theory
Count the binary strings of length n containing no two consecutive 1s, and count those with no three consecutive 1s.
Solution
No two consecutive 1s: condition on the last character. Ending in 0 leaves any valid string of length n-1; ending in 1 forces a 0 before it, leaving any valid string of length n-2. So a_n = a_{n-1}+a_{n-2} with a₀=1, a₁=2 — the Fibonacci numbers, a_n = F_{n+2}.
No three consecutive: condition on the length of the trailing run of 1s (0, 1 or 2), giving a_n = a_{n-1}+a_{n-2}+a_{n-3} — tribonacci, with a₀=1,a₁=2,a₂=4.
The general pattern: forbidding a run of k gives a k-term linear recurrence. There is also a direct binomial count — choosing j non-adjacent 1s among n positions is C(n-j+1, j), so F_{n+2} = ∑_j C(n-j+1,j), the "shallow diagonal" identity of Pascal's triangle.
- Theory
In how many ways can 8 rooks be placed on a chessboard so that none attacks another? Generalise to n × n with k rooks.
Solution
Non-attacking means one rook per row and one per column. Assign to each row a distinct column: that is a permutation of 8 elements, so 8! = 40{,}320.
With k rooks on n× n: choose k rows (C(n,k)), choose k columns (C(n,k)), and match them (k!):
C(n,k)² k!.
Check k=n: 1 · 1 · n! = n! ✓. The rook-polynomial machinery generalises this to boards with forbidden squares, which is exactly the setting where derangements (Chapter 31) arise — a derangement is a non-attacking rook placement avoiding the main diagonal.
- Program
Write generators for permutations, combinations and subsets in lexicographic order, without recursion, and verify the counts.
Solution
# next permutation in lex order (the std::next_permutation algorithm) def next_perm(a): i = len(a) - 2 while i >= 0 and a[i] >= a[i+1]: i -= 1 # find the rightmost ascent if i < 0: return False j = len(a) - 1 while a[j] <= a[i]: j -= 1 # rightmost element > a[i] a[i], a[j] = a[j], a[i] a[i+1:] = reversed(a[i+1:]) # the suffix was descending return True # next k-combination of range(n) def next_comb(c, n, k): i = k - 1 while i >= 0 and c[i] == n - k + i: i -= 1 if i < 0: return False c[i] += 1 for j in range(i+1, k): c[j] = c[j-1] + 1 return TrueBoth are O(1) amortised per step (the suffix reversal in
next_permaverages a constant number of swaps, by the same geometric argument as Chapter 12).Verify by counting: n! permutations, C(n,k) combinations, 2ⁿ subsets — and additionally assert the output is strictly increasing lexicographically, which catches the classic bug of using
>instead of>=in the ascent search when duplicates are present. - Program
Count solutions to x₁+···+x_k = n with per-variable upper bounds x_i ≤ u_i, for k ≤ 20 and n ≤ 10⁵.
Solution
from math import comb def bounded_solutions(n, us): k = len(us); total = 0 for mask in range(1 << k): # inclusion–exclusion over violations s, bits = n, 0 for i in range(k): if mask >> i & 1: s -= us[i] + 1; bits += 1 if s < 0: continue total += (-1)**bits * comb(s + k - 1, k - 1) return totalWhy: forcing x_i ≥ u_i+1 is a substitution reducing n by u_i+1, so the count of solutions violating a given set of bounds is another stars-and-bars value. Inclusion–exclusion over which bounds are violated gives the alternating sum.
Cost O(2ᵏ k) — fine to k = 20. For larger k with small bounds, use a DP over the sum instead: O(k n). Choosing between the two is a matter of which parameter is small, and both are worth having.
- Program
Implement ranking and unranking for k-subsets: given a subset return its index in lexicographic order, and vice versa.
Solution
from math import comb def rank_comb(c, n): # c sorted ascending, elements in [0,n) k = len(c); r = 0; prev = -1 for i, v in enumerate(c): for x in range(prev+1, v): r += comb(n - x - 1, k - i - 1) # subsets starting with x here prev = v return r def unrank_comb(r, n, k): c = []; x = 0 for i in range(k): while True: cnt = comb(n - x - 1, k - i - 1) if r < cnt: break r -= cnt; x += 1 c.append(x); x += 1 return cO(nk) as written, reducible to O(k log n) with binary search over the counts.
Ranking is what lets you sample a uniformly random k-subset (unrank a random index in [0, C(n,k))), store a subset compactly, or distribute an enumeration across machines by index range. The same technique works for permutations via the factorial number system, which is Chapter 27.
- Program
Count the monotone lattice paths from (0,0) to (n,n) that avoid a given set of blocked cells, for n ≤ 10⁵ and up to 20 blocked cells.
Solution
MOD = 10**9+7 def paths(a, b, c, d): # (a,b) → (c,d) if c < a or d < b: return 0 return C(c-a + d-b, c-a) def count_avoiding(n, blocked): pts = sorted(blocked) + [(n, n)] m = len(pts) f = [0]*m # f[i] = paths to pts[i] hitting no earlier block for i, (x, y) in enumerate(pts): f[i] = paths(0, 0, x, y) for j in range(i): px, py = pts[j] if px <= x and py <= y: f[i] -= f[j] * paths(px, py, x, y) % MOD f[i] %= MOD return f[-1]O(m²) after O(n) factorial precomputation. The recurrence classifies each path by the first blocked cell it hits, which makes the cases disjoint — the sum principle applied correctly, and the reason sorting the blocked cells matters.
This is the standard technique for "grid paths with few obstacles" and it scales to n = 10⁵ because the grid is never materialised; only the obstacle interactions are.
- Program
Verify the twelvefold way computationally for small n and k by brute-force enumeration.
Solution
from itertools import product from collections import Counter def brute(n, k, balls_distinct, boxes_distinct, constraint): seen = set() for assign in product(range(k), repeat=n): # ball i → box assign[i] cnt = Counter(assign) if constraint == 'inj' and any(v > 1 for v in cnt.values()): continue if constraint == 'sur' and len(cnt) < k: continue if balls_distinct and boxes_distinct: key = assign elif balls_distinct: key = frozenset(map(frozenset, ([i for i in range(n) if assign[i]==b] for b in range(k)))) elif boxes_distinct: key = tuple(cnt[b] for b in range(k)) else: key = tuple(sorted(cnt[b] for b in range(k))) seen.add(key) return len(seen)Run for n, k ≤ 5 and compare against the twelve formulas. This is the fastest way to settle which cell a problem belongs to when the wording is ambiguous — and it repeatedly surprises: the "identical balls, identical boxes" cells are partition counts with no closed form, which the brute force makes tangible.
Note the key construction is the whole content: it encodes exactly what "indistinguishable" means by quotienting the assignment by the relevant symmetry group. Getting that key right is getting the combinatorics right.
- Program
Given a multiset of letters, generate all distinct anagrams in lexicographic order without generating duplicates.
Solution
def anagrams(s): a = sorted(s) out = [''.join(a)] while next_perm(a): # next_perm handles duplicates correctly out.append(''.join(a)) return out # recursive alternative with explicit dedup at each level: def rec(counts, cur, n, out): if len(cur) == n: out.append(''.join(cur)); return for ch in sorted(counts): if counts[ch]: counts[ch] -= 1; cur.append(ch) rec(counts, cur, n, out) cur.pop(); counts[ch] += 1next_permalready produces each distinct arrangement exactly once for a multiset — the>=in the ascent search is precisely what skips duplicates. That is why it is preferable to generating all n! and deduplicating, which for MISSISSIPPI would generate 39.9 million strings to keep 34,650.Both run in O(1) amortised per output, so the total cost is proportional to the answer size — optimal.
- Program
Write a uniform random sampler for each of: a k-subset of n, a permutation, a composition of n into k parts, and verify uniformity statistically.
Solution
import random def rand_perm(n): # Fisher–Yates a = list(range(n)) for i in range(n-1, 0, -1): j = random.randint(0, i) # INCLUSIVE of i — the classic bug a[i], a[j] = a[j], a[i] return a def rand_subset(n, k): # Floyd's algorithm, O(k) s = set() for j in range(n-k, n): t = random.randrange(j+1) s.add(t if t not in s else j) return sorted(s) def rand_composition(n, k): # via stars and bars bars = sorted(random.sample(range(n+k-1), k-1)) prev, out = -1, [] for b in bars + [n+k-1]: out.append(b - prev - 1); prev = b return outFisher–Yates must pick j ∈ [0,i] inclusive; using [0,i) gives the "Sattolo" algorithm, which produces only cyclic permutations — uniform over a set of size (n-1)!, not n!. The bug is invisible without a statistical test.
Testing uniformity: for n = 4, sample 10⁶ permutations and chi-square the 24 counts against 10⁶/24. The Sattolo variant fails immediately (6 of the 24 get everything). This is the right way to test any random combinatorial generator — the small case where exhaustive counting is possible.
Permutations, combinations and multisets
Four formulas cover ordered/unordered selection with and without repetition, and a permutation has structure beyond its count - cycles, inversions and a factorial number system that make it rankable, composable and analysable.
Those four are the whole of elementary selection. The bottom-right is stars and bars: choosing k items from n types with repetition is the same as distributing k identical items into n boxes.
Cycle structure
Every permutation decomposes uniquely into disjoint cycles. This is the single most useful structural fact about permutations, and it turns many questions into arithmetic on cycle lengths:
That last fact is startling and useful: in a random permutation, the cycle through any fixed element is uniformly distributed over lengths 1..n. It is the basis of the famous "100 prisoners and 100 boxes" problem, where following cycles gives a success probability of about 31% instead of 2^{-100}.
Inversions and the factorial number system
An inversion is a pair i < j with σ(i) > σ(j). The number of inversions ranges from 0 to C(n,2), and the generating function is
because the Lehmer code c_i = #{j > i : σ(j) < σ(i)} is a bijection between permutations and sequences with 0 ≤ c_i ≤ n-i. That is the factorial number system, and it gives ranking and unranking directly.
def perm_rank(p):
n = len(p); bit = Fenwick(n)
for i in range(n): bit.add(i, 1)
r, f = 0, factorial(n-1)
for i, v in enumerate(p):
r += bit.query(v) * f # how many unused values are smaller
bit.add(v, -1)
if n-i-1: f //= (n-i-1)
return r
def perm_unrank(r, n):
avail = list(range(n)); out = []
for i in range(n):
f = factorial(n-i-1)
j, r = divmod(r, f)
out.append(avail.pop(j))
return outNumber of inversions = minimum number of adjacent transpositions to sort (Chapter 7) = the length of the permutation as a word in the simple reflections. So "sortedness" has an exact numerical measure, computable in O(nlog n), and it is what makes insertion sort Θ(n + inv).
Derangement-adjacent counts
Permutations with constraints are usually inclusion–exclusion (Chapter 30) or a determinant. Two to know:
- Fixed points: the number of permutations of n with exactly k fixed points is C(n,k)D_{n-k} where D_m is the derangement count.
- Involutions (σ² = id): I(n) = I(n-1) + (n-1)I(n-2) — element n is either fixed or swapped with one of the others.
"Find the next permutation" and "count inversions" are both standard, and the connection between them is worth stating: next-permutation walks the lexicographic order, inversions measure distance from sorted, and both come from the same Lehmer-code structure. Mentioning that std::next_permutation is O(1) amortised, not O(n), is a good detail.
- Theory
How many ways to arrange the letters of BOOKKEEPER? How many with all three E's together?
Solution
10 letters: B, O×2, K×2, E×3, P, R. Total (10!)/(2!2!3!) = \frac{3{,}628{,}800}{24} = 151{,}200.
With the E's together, glue them into a single block: now 8 objects (B, O, O, K, K, [EEE], P, R) with O×2 and K×2 repeated:
(8!)/(2!2!) = \frac{40{,}320}{4} = 10{,}080.
The block does not get an internal factor because the E's are identical. If they were distinguishable you would multiply by 3! — the standard slip.
- Theory
Prove that a permutation's parity is well defined, i.e. every decomposition into transpositions has the same parity.
Solution
Consider the polynomial Δ = ∏_{i<j}(x_i - x_j). A permutation σ acts by permuting the variables, sending Δ to ±Δ (each factor maps to another factor, up to sign). Define sgn(σ) as that sign.
A single transposition (ab) negates Δ: the factor (x_a-x_b) flips sign, and the other affected factors pair up with matching sign changes. So sgn is a homomorphism sending every transposition to -1, and a product of k transpositions has sign (-1)ᵏ. Since sgn(σ) depends only on σ, all decompositions have the same parity. ∎
Equivalently: each transposition changes the inversion count by an odd amount, so sgn(σ) = (-1)^{inv(σ)}. This is the invariant behind the 15-puzzle argument of Chapter 7.
- Theory
What is the expected number of fixed points in a random permutation of n? The variance?
Solution
Let X_i = [σ(i)=i], so P(X_i=1) = 1/n. By linearity, E[X] = n · \frac1n = 1 — exactly 1, for every n.
For the variance: E[X²] = ∑_i E[X_i] + ∑_{i≠ j} P(X_i=X_j=1) = 1 + n(n-1)(1)/(n(n-1)) = 2, so Var = 2 - 1 = 1.
Mean 1, variance 1 — and in fact the number of fixed points converges in distribution to Poisson(1) as n→∞. So the probability of no fixed points tends to e^{-1} ≈ 0.368, which is the derangement result of Chapter 31 arriving from a different direction.
- Theory
Show that the expected number of cycles in a random permutation of n is H_n.
Solution
Build the permutation by the "Chinese restaurant" process: place elements 1..n one at a time; element i either starts a new cycle or is inserted into an existing one. When adding element i there are i choices (join after any of the i-1 already placed, or start a new cycle), each equally likely, so a new cycle starts with probability 1/i.
By linearity, E[#cycles] = ∑_{i=1}^{n} \frac1i = H_n ≈ ln n. ∎
The process is the Feller coupling, and it also gives the variance H_n - H_n^{(2)} ≈ ln n because the indicators are independent. This is the same 1/i pattern as the record-counting problem of Chapter 11 — and indeed the number of cycles and the number of records are identically distributed, via the "fundamental bijection" that writes each cycle starting from its largest element.
- Theory
Explain the 100-prisoners problem and why the cycle-following strategy succeeds with probability about 31%.
Solution
100 prisoners, 100 boxes containing a random permutation of their numbers; each may open 50 boxes and must find their own number; all must succeed.
Strategy: prisoner i opens box i, then the box numbered by what he found, and so on — following the cycle containing i. He succeeds iff that cycle has length ≤ 50.
So all succeed iff the permutation has no cycle longer than 50. The probability that a random permutation of 2m has a cycle longer than m is ∑_{k=m+1}^{2m} \frac1k = H_{2m} - H_m → ln 2. Hence success probability 1 - ln 2 ≈ 0.3069, and for n=100 exactly 31.18%.
The reason a long cycle has probability 1/k: the number of permutations of n with a cycle of length k > n/2 is C(n,k)(k-1)!(n-k)! = n!/k, and such cycles are unique so the events are disjoint. Independent random guessing gives 2^{-100} — the strategy's power comes entirely from making the prisoners' outcomes maximally correlated.
- Theory
Count the permutations of 1..n with exactly k inversions, and give the generating function.
Solution
By the Lehmer code bijection, a permutation corresponds to (c₁,…,c_n) with 0 ≤ c_i ≤ n-i and inv = ∑ c_i. So
∑_{σ} q^{inv(σ)} = ∏_{i=1}^{n}(1 + q + ··· + q^{i-1}) = ∏_{i=1}^{n}(1-q^i)/(1-q) = [n]_q!,
the q-factorial. The count with exactly k inversions is the coefficient of qᵏ, the Mahonian number T(n,k), satisfying T(n,k) = ∑_{j=0}^{min(k,n-1)} T(n-1,k-j).
Computationally: a DP over O(n · C(n,2)) states with prefix sums gives O(n³), or O(n²) per row with a sliding window. Setting q=1 recovers n! ✓.
- Theory
How many permutations of n are involutions? Derive the recurrence and the asymptotic.
Solution
An involution has only fixed points and 2-cycles. Condition on element n: either it is fixed (leaving an involution of n-1) or it is paired with one of the n-1 others (leaving an involution of n-2):
I(n) = I(n-1) + (n-1)I(n-2), with I(0)=I(1)=1.
Values: 1, 1, 2, 4, 10, 26, 76, 232, … The explicit form is I(n) = ∑_k (n!)/(k!2ᵏ(n-2k)!) — choose k disjoint pairs.
Asymptotically I(n) ~ (1)/(√2)((n)/(e))^{n/2}e^{√n}, i.e. roughly √{n!}. By the RSK correspondence, I(n) also equals the number of standard Young tableaux with n cells — a striking identity that a purely recursive derivation never suggests.
- Program
Decompose a permutation into cycles, compute its order and parity, and apply it n times efficiently.
Solution
def cycles(p): n = len(p); seen = [False]*n; out = [] for i in range(n): if seen[i]: continue c = [] j = i while not seen[j]: seen[j] = True; c.append(j); j = p[j] out.append(c) return out def order(p): from math import lcm o = 1 for c in cycles(p): o = lcm(o, len(c)) return o def parity(p): return (len(p) - len(cycles(p))) % 2 # 0 = even, 1 = odd def power(p, n): res = [0]*len(p) for c in cycles(p): L = len(c); s = n % L for i in range(L): res[c[i]] = c[(i+s) % L] return resAll are O(n). The order can be astronomically large — Landau's function g(n), the maximum order of a permutation of n elements, is e^{(1+o(1))√{nln n}}, reaching 232,792,560 at n=100. So compute it with big integers or modulo something, and never by repeated application.
- Program
Count inversions in O(nlog n) and additionally report, for each element, how many smaller elements lie to its right.
Solution
def count_smaller_right(a): order = {v: i for i, v in enumerate(sorted(set(a)))} m = len(order); bit = [0]*(m+1) res = [0]*len(a) for idx in range(len(a)-1, -1, -1): k = order[a[idx]] # count already-seen values < a[idx] i = k while i > 0: res[idx] += bit[i]; i -= i & -i i = k + 1 while i <= m: bit[i] += 1; i += i & -i return res, sum(res)Sweeping right to left with a Fenwick tree gives both the per-element counts (the Lehmer code) and their total (the inversion count) in one pass, O(nlog n).
The Lehmer code is exactly what
perm_rankneeds, so this function does double duty. It is also the answer to LeetCode's "count of smaller numbers after self", and the observation that the answer is the Lehmer code is what connects that exercise to the factorial number system. - Program
Implement rank and unrank for permutations of n ≤ 20 and verify they are inverse over all permutations of n = 8.
Solution
from math import factorial def unrank(r, n): avail = list(range(n)); out = [] for i in range(n): f = factorial(n - i - 1) j, r = divmod(r, f) out.append(avail.pop(j)) return out def rank(p): n = len(p); avail = list(range(n)); r = 0 for i, v in enumerate(p): j = avail.index(v) # O(n); use a BIT for O(log n) r += j * factorial(n - i - 1) avail.pop(j) return r from itertools import permutations for i, p in enumerate(permutations(range(8))): assert rank(list(p)) == i and unrank(i, 8) == list(p)At n = 20, 20! ≈ 2.4×10^{18} still fits in a 64-bit integer — so a permutation of 20 elements can be stored as a single
long long, which is occasionally exactly what a memoisation table needs.The exhaustive 8! = 40{,}320 check against
itertools.permutationsconfirms both the bijection and that the ordering agrees with lexicographic — a two-line test that would catch any indexing error. - Program
Simulate the 100-prisoners problem and compare the cycle strategy to random guessing.
Solution
import random def trial(n=100, limit=50): p = list(range(n)); random.shuffle(p) for start in range(n): cur, steps = start, 0 while p[cur] != start: cur = p[cur]; steps += 1 if steps >= limit: return False return True wins = sum(trial() for _ in range(100000)) print(wins/100000) # ≈ 0.3118 # equivalently, and far faster: check the longest cycle def trial_fast(n=100, limit=50): p = list(range(n)); random.shuffle(p) return max(len(c) for c in cycles(p)) <= limitThe simulation converges to 0.3118, matching 1 - (H_{100}-H_{50}) = 1 - ∑_{k=51}^{100}1/k. Random guessing gives 2^{-100} ≈ 10^{-30} — the difference is 30 orders of magnitude from a strategy that adds no information, only correlation.
A worthwhile extension: vary the limit and plot the success probability. It stays near 1 until the limit drops below n/2, then falls off as 1 - ln(n/limit) — and it is provably optimal, which is the surprising part.
- Program
Count permutations of n with exactly k inversions for n,k ≤ 10⁵ modulo 10⁹+7.
Solution
MOD = 10**9+7 def mahonian_row(n, K): dp = [0]*(K+1); dp[0] = 1 for i in range(2, n+1): pre = [0]*(K+2) for j in range(K+1): pre[j+1] = (pre[j] + dp[j]) % MOD for j in range(K+1): lo = max(0, j - i + 1) dp[j] = (pre[j+1] - pre[lo]) % MOD # sliding window of width i return dpThe recurrence T(n,k) = ∑_{j=0}^{i-1} T(n-1,k-j) is a sliding-window sum, so prefix sums make each row O(K) and the whole table O(nK) — 10^{10} at the stated limits, too slow.
For large n and k there is a closed form via the q-factorial and the pentagonal number theorem: T(n,k) = ∑_j (-1)₌ C(n + k - g_j - 1, n-1) summed over generalised pentagonal numbers g_j ≤ k, of which there are only O(√k). That gives O(√k) per query after factorial precomputation — the practical answer at these limits.
- Program
Generate a uniformly random permutation with exactly k fixed points.
Solution
import random def rand_with_k_fixed(n, k): pts = random.sample(range(n), k) # which points are fixed rest = [i for i in range(n) if i not in set(pts)] d = random_derangement(len(rest)) # derange the remainder p = list(range(n)) for idx, v in enumerate(rest): p[v] = rest[d[idx]] return p def random_derangement(m): # rejection sampling while True: a = list(range(m)); random.shuffle(a) if all(a[i] != i for i in range(m)): return aUniformity: every permutation with exactly k fixed points arises from exactly one (fixed-point set, derangement) pair, and both components are sampled uniformly — so the result is uniform over the C(n,k)D_{n-k} such permutations.
Rejection sampling for the derangement accepts with probability D_m/m! → e^{-1} ≈ 0.368, so it needs about 2.7 attempts on average regardless of m — constant expected overhead, which is why the naive approach is perfectly good here. (A linear-time direct algorithm exists but is rarely worth the complexity.)
- Program
Given a permutation as a product of cycles, compute the minimum number of transpositions to sort it, and produce them.
Solution
def sort_by_transpositions(p): n = len(p); p = p[:]; swaps = [] for i in range(n): while p[i] != i: j = p[i] p[i], p[j] = p[j], p[i] # send p[i] home swaps.append((i, j)) return swaps # length = n - (number of cycles)Minimality: each transposition changes the cycle count by exactly ±1 (it merges two cycles or splits one). The identity has n cycles; a permutation with c cycles therefore needs at least n-c transpositions, and the algorithm achieves exactly that — every swap places an element correctly and splits off a fixed point. ∎
O(n) total, since each iteration of the inner loop fixes one element permanently. Note this is the minimum over arbitrary transpositions; restricting to adjacent transpositions gives the inversion count instead, which can be Θ(n²) — the two "sorting distances" are genuinely different measures.
Binomial coefficients and Pascal identities
A dozen identities cover nearly every binomial manipulation you will need, and each has a combinatorial proof that is shorter and more memorable than the algebra. Learn the proofs, not the formulas.
The proofs worth knowing
Pascal. Classify k-subsets of {1..n} by whether they contain n. Those that do correspond to (k-1)-subsets of the rest; those that do not are k-subsets of the rest. Disjoint and exhaustive. ∎
Vandermonde. To choose k items from m+n objects split into two groups, choose i from the first and k-i from the second, and sum over i. ∎ Setting m=n=k and using symmetry gives ∑_k C(n,k)² = C(2n,n).
Hockey stick. Count (k+1)-subsets of {1..n+1} by their largest element: if the largest is i+1, the remaining k come from {1..i}. ∎
Absorption. Count pairs (a k-subset, a distinguished element of it) two ways: choose the subset then the element (C(n,k)· k), or choose the element then the rest (n · C(n-1,k-1)). ∎
Almost every binomial identity is double counting: find a set of objects that both sides count, in two different orders. If you cannot find one, try the generating function route — multiply out (1+x)^m(1+x)^n = (1+x)^{m+n} and compare coefficients, which proves Vandermonde in one line.
Divisibility and size
The last bound is the workhorse in algorithm analysis: it turns "how many subsets of size at most k" into (en/k)^k, which is polynomial when k is constant and is exactly the bound used in parameterised complexity.
Computing binomials
# 1. Pascal's triangle DP: exact, O(n²) time and space, n ≤ few thousand
C = [[0]*(n+1) for _ in range(n+1)]
for i in range(n+1):
C[i][0] = 1
for j in range(1, i+1): C[i][j] = C[i-1][j-1] + C[i-1][j]
# 2. Multiplicative, exact integers: O(k), no overflow if the answer fits
def binom(n, k):
k = min(k, n-k); r = 1
for i in range(k): r = r * (n-i) // (i+1) # divisible at every step
return r
# 3. Factorials mod prime p: O(1) per query after O(n) precompute (Ch. 17)
# 4. lgamma for floating point: any n, ~13 significant digits (Ch. 11)In method 2 the running product r is always C(n, i+1) after step i, so the division is exact — but only in that order. Computing the whole numerator first overflows; dividing at each step does not.
"How many paths in a grid" is C(m+n,m); "with obstacles" is inclusion–exclusion or DP; "staying below the diagonal" is Catalan. Recognising which of the three you are in, and saying why, is the whole question — the formula is the easy part.
- Theory
Prove ∑_k C(n,k)² = C(2n,n) combinatorially.
Solution
Count n-subsets of a 2n-set split into two halves A and B of size n each. If the subset takes k elements from A, it takes n-k from B, giving C(n,k)C(n,n-k) = C(n,k)² by symmetry. Summing over k counts every n-subset exactly once, so the total is C(2n,n). ∎
This is Vandermonde with m=n and k=n. The algebraic proof compares coefficients of xⁿ in (1+x)ⁿ(1+x)ⁿ = (1+x)^{2n} — equally short, and it generalises to q-analogues where the combinatorial argument does not.
- Theory
Prove the hockey-stick identity ∑_{i=k}^{n} C(i,k) = C(n+1,k+1) two ways.
Solution
Combinatorially: count (k+1)-subsets of {1,…,n+1} by their largest element. If the largest is i+1, the other k elements form a k-subset of {1,…,i}: C(i,k) ways. Summing over i = k..n covers every subset once. ∎
By telescoping: Pascal gives C(i+1,k+1) - C(i,k+1) = C(i,k). Summing i=k..n, the left telescopes to C(n+1,k+1) - C(k,k+1) = C(n+1,k+1). ∎
Practical use: it turns a running sum of a column of Pascal's triangle into a single lookup, which is how prefix sums over binomials are answered in O(1).
- Theory
Prove that C(n,k) is always an integer, without invoking its combinatorial meaning.
Solution
Induct using Pascal: C(n,k) = C(n-1,k-1) + C(n-1,k). Base cases C(n,0)=C(n,n)=1 are integers, and a sum of integers is an integer. ∎
Alternatively, by Legendre's formula the exponent of any prime p in n!/(k!(n-k)!) is
∑_{i\ge1}(⌊ n/p^i⌋ - ⌊ k/p^i⌋ - ⌊ (n-k)/p^i⌋),
and each term is 0 or 1 because ⌊ x+y ⌋ ≥ ⌊ x⌋ + ⌊ y ⌋. So every exponent is non-negative. ∎ This second proof is Kummer's theorem in embryo: the number of 1-terms is exactly the number of carries when adding k and n-k in base p.
- Theory
Compute ∑_k kC(n,k) and ∑_k k²C(n,k).
Solution
First: by absorption kC(n,k) = nC(n-1,k-1), so the sum is n∑_j C(n-1,j) = n2^{n-1}.
Second: write k² = k(k-1) + k. Applying absorption twice, k(k-1)C(n,k) = n(n-1)C(n-2,k-2), so ∑ k(k-1)C(n,k) = n(n-1)2^{n-2}. Adding the first result gives
∑_k k² C(n,k) = n(n-1)2^{n-2} + n2^{n-1} = n(n+1)2^{n-2}.
Probabilistic reading: dividing by 2ⁿ, these are E[X] = n/2 and E[X²] = n(n+1)/4 for X ~ Bin(n,1/2), giving Var = n/4 ✓ — which matches npq = n/4.
- Theory
State and prove Kummer's theorem, then use it to find the largest power of 3 dividing C(1000,500).
Solution
Kummer: the exponent of p in C(m+n,m) equals the number of carries when adding m and n in base p.
Proof: by Legendre, the exponent is ∑_i (⌊ (m+n)/p^i⌋ - ⌊ m/p^i ⌋ - ⌊ n/p^i⌋). Each term is 1 exactly when a carry occurs out of position i-1 in the base-p addition, and 0 otherwise. ∎
For C(1000,500) and p=3: 500_{10} = 200122_3. Adding 200122_3 + 200122_3: digits 2+2=4=11_3 (carry), 2+2+1=5=12_3 (carry), 1+1+1=3=10_3 (carry), 0+0+1=1 (no carry), 0+0=0, 2+2=4 (carry). That is 4 carries, so 3⁴ ‖ C(1000,500).
Corollary worth knowing: C(n,k) is odd iff k & (n-k) = 0 in binary, i.e. iff k is a submask of n — no carries in base 2. That is the reason Pascal's triangle mod 2 is the Sierpiński triangle.
- Theory
How many binary strings of length n have exactly k ones and no two ones adjacent?
Solution
Place the n-k zeros in a row, creating n-k+1 gaps (including the ends). Choose k of these gaps to hold a single one each — no two ones can then be adjacent, and every valid string arises exactly once.
Answer: C(n-k+1, k).
Summing over k gives ∑_k C(n-k+1,k) = F_{n+2}, the shallow-diagonal identity of Pascal's triangle and the closed form behind Chapter 26 problem 7. The "gaps" technique — place the unconstrained items first, then slot the constrained ones into the gaps they create — solves a whole family of adjacency problems.
- Theory
Prove C(n,k) ≤ (en/k)^k and use it to bound the number of subsets of size at most k.
Solution
C(n,k) = (n(n-1)···(n-k+1))/(k!) ≤ (n^k)/(k!), and k! ≥ (k/e)^k (from e^k = ∑_j k^j/j! ≥ k^k/k!). Hence C(n,k) ≤ (n^k)/((k/e)^k) = (en/k)^k. ∎
Summing, ∑_{i≤ k} C(n,i) ≤ (en/k)^k as well for k ≤ n/2, since the terms grow by a factor of at least e... more carefully, the standard bound is ∑_{i≤ k}C(n,i) ≤ ((en)/(k))^k.
This is the bound that makes O(2^k · poly) parameterised algorithms meaningful: enumerating all subsets of size ≤ k from n elements is n^{O(k)}, which is polynomial for fixed k — the difference between FPT and XP.
- Theory
Show that the rows of Pascal's triangle mod 2 form the Sierpiński triangle.
Solution
By Kummer with p=2, C(n,k) is odd iff adding k and n-k in binary produces no carries, i.e. iff the binary representations of k and n-k have disjoint 1-bits, i.e. iff k is a submask of n: k & n = k.
So row n has exactly 2^{s(n)} odd entries, where s(n) is the popcount of n. Plotting the odd entries produces the self-similar Sierpiński pattern, because "submask of n" is a self-similar condition under doubling: row 2n is row n with gaps inserted. ∎
This gives an O(1) parity test for a binomial and, via Lucas' theorem (Chapter 29), the general statement modulo any prime: C(n,k) ≡ ∏_i C(n_i,k_i) ±od p over base-p digits, which is zero exactly when some k_i > n_i.
- Program
Compute exact C(n,k) for n ≤ 10⁵ as a big integer, and count its digits without materialising it.
Solution
def binom_exact(n, k): k = min(k, n-k) if k < 0: return 0 num = den = 1 for i in range(k): num *= n - i den *= i + 1 g = math.gcd(num, den) # keep the numbers small num //= g; den //= g return num // den def binom_digits(n, k): # via lgamma, no big integers import math lg = (math.lgamma(n+1) - math.lgamma(k+1) - math.lgamma(n-k+1)) / math.log(10) return int(lg) + 1Reducing by the gcd at every step keeps the intermediates near the size of the answer rather than near n^k. Even so, C(10⁵, 500{,}000) has about 301,000 digits, so the multiplication cost dominates; the divide-and-conquer product tree plus a single final division is asymptotically better.
binom_digitsgives the size in microseconds for any n, which is often all you need — for instance to check whether the answer fits in a fixed-width type before computing it. - Program
Build Pascal's triangle mod 2 and render the Sierpiński pattern; verify the odd-entry count is 2^{popcount(n)}.
Solution
for n in range(64): row = ''.join('#' if (k & n) == k else '.' for k in range(n+1)) print(row.center(70)) assert sum(1 for k in range(n+1) if (k & n) == k) == 1 << bin(n).count('1')The test
(k & n) == kis the submask condition from problem 8, and it replaces any actual binomial computation — an O(1) parity oracle.Summing the odd counts over rows 0..2^m-1 gives 3^m (each bit independently contributes a factor of 3: absent, or present in n only, or present in both), which is the Hausdorff-dimension statement log 3/log 2 ≈ 1.585 for the Sierpiński triangle — the same exponent as Karatsuba's, for the same self-similar reason.
- Program
Answer 10⁶ queries for C(n,k) mod 10⁹+7 with n ≤ 2×10⁶, and separately handle n up to 10¹⁸ with k ≤ 10⁵.
Solution
# Case A: n bounded — factorials + inverse factorials, O(1) per query (Ch. 17) # Case B: huge n, small k — multiply k terms directly def binom_huge(n, k, MOD, invfact): if k < 0 or k > n: return 0 num = 1 for i in range(k): num = num * ((n - i) % MOD) % MOD return num * invfact[k] % MODCase B is O(k) per query — fine for k ≤ 10⁵ and a handful of queries, but not for 10⁶ of them. The reduction
(n - i) % MODis essential: n itself exceeds the modulus, and forgetting it gives silent overflow in C++.If both n and the query count are huge, the answer is Lucas' theorem (Chapter 29), which reduces to base-p digits and costs O(log_p n) per query.
- Program
Verify Vandermonde's convolution numerically for all m,n,k ≤ 30, and implement a convolution-based binomial product.
Solution
from math import comb for m in range(31): for n in range(31): for k in range(m+n+1): lhs = sum(comb(m,i)*comb(n,k-i) for i in range(k+1)) assert lhs == comb(m+n,k), (m,n,k)The identity says the sequence (C(n,k))_k convolved with (C(m,k))_k is (C(m+n,k))_k — which is exactly the polynomial identity (1+x)^m(1+x)^n = (1+x)^{m+n}.
Practical consequence: binomial rows can be combined with FFT/NTT in O(nlog n) rather than O(n²). More usefully, it means any DP whose transition is "distribute k items between two independent halves" is a convolution and can be accelerated the same way (Chapter 51).
- Program
Count lattice paths from (0,0) to (m,n) using only right and up moves, that also never go above the line y = x.
Solution
def paths_below_diagonal(m, n): # requires n <= m from math import comb if n > m: return 0 # reflection principle: bad paths correspond bijectively to # paths to the reflected endpoint (n-1, m+1) return comb(m+n, n) - comb(m+n, n-1)Reflection principle: a bad path touches the line y = x+1; reflect the portion before its first touch across that line. This is a bijection between bad paths to (m,n) and all paths to (n-1, m+1), so the bad count is C(m+n, n-1). ∎
With m = n this gives C(2n,n) - C(2n,n-1) = (1)/(n+1)C(2n,n) — the Catalan numbers, which Chapter 32 develops. The reflection argument is the cleanest derivation of them and generalises to the ballot problem and to random-walk hitting probabilities.
- Program
Implement C(n,k) mod m for arbitrary composite m, combining Lucas per prime power with CRT.
Solution
def binom_mod_composite(n, k, m): res, mods = [], [] for p, e in factorise(m).items(): res.append(binom_mod_prime_power(n, k, p, e)) mods.append(p**e) return crt(res, mods)[0] def binom_mod_prime_power(n, k, p, e): pe = p**e # Granville/Andrew: track the exponent of p separately via Kummer, # and compute the p-stripped factorials, which ARE invertible mod p^e. ex = carries_base_p(k, n-k, p) # exponent of p in C(n,k) if ex >= e: return 0 num = stripped_fact(n, p, e) den = stripped_fact(k, p, e) * stripped_fact(n-k, p, e) % pe return pow(p, ex, pe) * num % pe * pow(den, -1, pe) % peThe two ingredients: Kummer gives the exponent of p (so you know when the answer is ≡ 0), and the p-stripped factorial (n!)_p = ∏_{i ≤ n, p ∤ i} i is coprime to p and therefore invertible mod p^e.
This handles the notorious "compute C(n,k) mod 10⁶" problems, where the naive factorial-inverse approach fails because 10⁶ = 2⁶5⁶ is composite and factorials are not invertible. Total cost O(p^e + log_p n) per prime power.
- Program
Write a search that finds all binomial coefficients equal to a given value N, for N ≤ 10^{15}.
Solution
def find_binomials(N): out = [(N, 1), (N, N-1)] if N > 1 else [] # trivial C(N,1)=N k = 2 while True: # smallest n with C(n,k) >= N is roughly k + (N k!)^{1/k} n = k while binom_exact(n, k) < N: n += 1 if binom_exact(n, k) == N: out += [(n, k), (n, n-k)] if binom_exact(2*k, k) > N: break # k too large from here on k += 1 return sorted(set(out))The loop terminates because C(2k,k) ≈ 4^k/√{π k} grows exponentially, so k ≤ log₄ N ≈ 25 at N = 10^{15}. For each k the inner search is a binary search over n.
Interesting outputs: 3003 = C(3003,1) = C(78,2) = C(15,5) = C(14,6) — the only known number appearing eight times in Pascal's triangle. Singmaster's conjecture says the number of appearances is bounded by an absolute constant; the largest known is 8, and nobody has proved a bound better than O(log N).
Binomials modulo a prime: Lucas and prime powers
Factorial-inverse binomials break down the moment n exceeds the modulus, because the factorial contains the modulus as a factor. Lucas' theorem repairs it by working digit by digit in base p, and Granville's generalisation extends it to prime powers.
The standard O(1) method — precompute n! and (n!)^{-1} mod p — requires n < p. Beyond that, n! contains p as a factor and is ≡ 0, so the inverse does not exist and the formula is meaningless.
Lucas' theorem
Proof sketch: in 𝔽_p[x], the Frobenius map gives (1+x)^{p} ≡ 1 + x^{p}, hence (1+x)^{p^i} ≡ 1 + x^{p^i}. So
Expanding the right side, the coefficient of xᵏ picks exactly one term from each factor, and base-p representation is unique — so the coefficient is ∏ C(n_i,k_i). ∎
long long lucas(long long n, long long k, long long p,
const vector<long long>& fact, const vector<long long>& ifact){
long long r = 1;
while (n || k){
long long ni = n % p, ki = k % p;
if (ki > ni) return 0; // a digit fails
r = r * fact[ni] % p * ifact[ki] % p * ifact[ni-ki] % p;
n /= p; k /= p;
}
return r;
}
// precompute fact/ifact only up to p — so p must be small enough to sieveCost: O(log_p n) per query after O(p) precomputation. With p = 10⁹+7 the precomputation is the bottleneck, so Lucas is used when p is small (a few million) and n is enormous.
Choose the method by comparing n to p. n < p: factorials and inverse factorials, O(1) per query. n ≥ p, p small and prime: Lucas. modulus is a prime power: Granville/Andrew with p-stripped factorials. modulus composite: do each prime power and CRT. There is no single method that covers all four.
Prime powers
Modulo p^e the obstruction is the same but the repair is more involved. Define the p-stripped factorial
Then C(n,k) = p^{E} · ((n!)^*)/((k!)^*((n-k)!)^*) where E is the exponent of p (by Kummer, the number of carries) and the starred quantities are the fully recursed stripped products, which are units mod p^e. If E ≥ e the answer is 0.
The stripped factorial modulo p^e is periodic with period p^e, so it is computed as (a full period)⌊n/p^e⌋ times a partial product — O(p^e + log_p n) after precomputing one period.
Wilson's theorem and the period
The full-period product has a clean value: (p^e!)_p ≡ -1 ±od {p^e} for p odd or p^e = 4, and +1 for p = 2, e ≥ 3. For e = 1 this is Wilson's theorem: (p-1)! ≡ -1 ±od p, because every unit pairs with its distinct inverse except ±1.
"Compute C(n,k) mod m" is a question about which of four regimes you are in, and the strongest answer names the regime before writing code: "m is prime and larger than n, so factorials and inverse factorials, O(n) precompute and O(1) per query." If m is 10⁶ rather than 10⁹+7, noticing that it is composite is the entire question.
- Theory
Compute C(1000, 500) mod 7 using Lucas.
Solution
Base 7: 1000 = 2·343 + 6·49 + 2·7 + 6 = (2,6,2,6)_7. And 500 = 1·343 + 3·49 + 1·7 + 3 = (1,3,1,3)_7.
Every digit satisfies k_i ≤ n_i, so the answer is nonzero:
C(2,1)C(6,3)C(2,1)C(6,3) = 2 · 20 · 2 · 20 = 1600 ≡ 1600 - 228·7 = 4 ±od 7.
Note how cheap this is compared with computing a 300-digit number and reducing it — four small binomials and a product.
- Theory
Prove (1+x)^p ≡ 1 + x^p ±od p and deduce Lucas.
Solution
By the binomial theorem, (1+x)^p = ∑_k C(p,k)x^k. For 0 < k < p, C(p,k) = (p!)/(k!(p-k)!) has a factor p in the numerator and none in the denominator (all factors there are < p), so p | C(p,k). Only k = 0 and k = p survive. ∎
Iterating, (1+x)^{p^i} ≡ 1 + x^{p^i}. Writing n = ∑ n_i p^i,
(1+x)^n ≡ ∏_i (1+x^{p^i})^{n_i} ±od p.
Comparing the coefficient of x^k and using uniqueness of base-p representation gives C(n,k) ≡ ∏_i C(n_i,k_i). ∎
- Theory
For how many k in [0,n] is C(n,k) odd? Prove it.
Solution
By Lucas with p=2, C(n,k) is odd iff every binary digit satisfies k_i ≤ n_i, i.e. iff k is a submask of n. The number of submasks of n is 2^{s(n)} where s(n) is the popcount.
So the count is 2^{popcount(n)}. ∎ For n = 2^m - 1 (all ones) every entry is odd — 2^m = n+1 of them, the full row; for n = 2^m only two are.
This is the Sierpiński structure of Chapter 28 problem 8, now derived rather than observed, and it generalises: the number of entries in row n not divisible by p is ∏_i (n_i + 1) over the base-p digits.
- Theory
Prove Wilson's theorem and its converse.
Solution
Forward: in (ℤ/p)^×, pair each element with its inverse. An element is its own inverse iff x²≡1, i.e. x ≡ ±1 (Chapter 23). So all elements except 1 and p-1 pair off into products of 1, leaving (p-1)! ≡ 1 · (p-1) ≡ -1 ±od p. ∎
Converse: if n is composite and n > 4, let d be a divisor with 1 < d < n. Then d | (n-1)!, so (n-1)! ≢ -1 ±od n (it is ≡ 0 ±od d while -1 is not). The case n=4 gives 3! = 6 ≡ 2 ≢ -1. ∎
So (n-1)! ≡ -1 ±od n characterises primes exactly — a beautiful primality test that is useless in practice, since computing (n-1)! mod n takes Θ(n) multiplications.
- Theory
Compute C(20,10) mod 9 by hand, showing why plain Lucas does not apply.
Solution
9 is not prime, so Lucas is inapplicable. Use the prime-power method with p=3, e=2.
Exponent of 3 (Kummer): 10 = 101_3. Adding 101_3 + 101_3: 1+1=2 (no carry), 0+0=0, 1+1=2 — no carries, so 3^0 ‖ C(20,10), i.e. 3 does not divide it.
Then C(20,10) = 184{,}756, and 184756 = 20528·9 + 4, so the answer is 4. Consistent with the exponent being 0 (the answer is a unit mod 9). ✓
The general algorithm would compute this without knowing the value: strip the factorials of their factors of 3, invert mod 9, multiply by 3^0.
- Theory
Given m = 10⁶, explain the full strategy for C(n,k) mod m with n up to 10¹⁸.
Solution
10⁶ = 2⁶ · 5⁶. Compute C(n,k) modulo 2⁶ = 64 and modulo 5⁶ = 15{,}625 separately, then CRT.
For each prime power p^e:
- Compute the exponent E of p in C(n,k) by Legendre/Kummer, in O(log_p n). If E ≥ e, the result mod p^e is 0.
- Otherwise compute the three p-stripped factorial products modulo p^e, using the period-p^e structure and recursion on ⌊ n/p ⌋. Precompute one period: 64 and 15,625 entries — trivial.
- Combine: p^E · A · B^{-1} · C^{-1} mod p^e.
Total O(p^e + log_p n) per prime power, so microseconds. The key realisation is that the small size of p^e — not of n — is what makes it tractable.
- Theory
Count the number of nonzero entries in row n of Pascal's triangle modulo a prime p.
Solution
By Lucas, C(n,k) ≢ 0 ±od p iff k_i ≤ n_i for every base-p digit. For each digit position there are n_i + 1 admissible values of k_i, chosen independently, so the count is
∏_i (n_i + 1) over the base-p digits of n. ∎
Check with p=2: ∏ (n_i+1) = 2^{popcount(n)} ✓, matching problem 3. For p=3 and n = 100 = (10201)_3: 2·1·3·1·2 = 12 nonzero entries out of 101 — the row is 88% zeros mod 3.
Summing over n < p^m gives ((p(p+1))/(2))^m nonzero entries in the whole triangle, so the density of nonzero entries tends to 0 — Pascal's triangle mod p is mostly zeros, and self-similarly so.
- Program
Implement Lucas' theorem and verify against exact big-integer binomials for n ≤ 500 and several small primes.
Solution
def lucas(n, k, p, fact, ifact): r = 1 while n or k: ni, ki = n % p, k % p if ki > ni: return 0 r = r * fact[ni] % p * ifact[ki] % p * ifact[ni-ki] % p n //= p; k //= p return r from math import comb for p in (2,3,5,7,11,13,101): f = [1]*p for i in range(1,p): f[i] = f[i-1]*i % p inv = [pow(x, p-2, p) for x in f] for n in range(500): for k in range(n+1): assert lucas(n,k,p,f,inv) == comb(n,k) % p, (n,k,p)The exhaustive cross-check against
math.combfor n ≤ 500 is decisive — it exercises the zero case, the carry-free case, and multi-digit products. Once it passes for several primes there is essentially nothing left that can be wrong.Note
ifactis computed with Fermat, which requires p prime; asserting that up front prevents the silent-wrong-answer failure mode of Chapter 17. - Program
Compute C(n,k) mod p^e using the Granville/Andrew method for p^e ≤ 10⁷ and n ≤ 10¹⁸.
Solution
def binom_pp(n, k, p, e): pe = p**e # one period of the p-stripped factorial, mod p^e f = [1]*(pe+1) for i in range(1, pe+1): f[i] = f[i-1] if i % p == 0 else f[i-1]*i % pe def stripped(m): # (m!)* fully recursed, and the p-exponent r, e_cnt = 1, 0 while m: r = r * pow(f[pe], m // pe, pe) % pe * f[m % pe] % pe m //= p e_cnt += m return r, e_cnt a, ea = stripped(n) b, eb = stripped(k) c, ec = stripped(n-k) E = ea - eb - ec # exponent of p in C(n,k) if E >= e: return 0 return pow(p, E, pe) * a % pe * pow(b*c % pe, -1, pe) % peO(p^e) for the period table plus O(log_p n) per query. The exponent E comes out of the same recursion as Legendre's formula, so no separate Kummer computation is needed.
Test it against
comb(n,k) % p**efor n ≤ 300 and small p^e — including cases where the answer is 0, which is where the E ≥ e branch is exercised. - Program
Answer 10⁵ queries of C(n,k) mod m with arbitrary composite m ≤ 10⁶ and n ≤ 10¹⁸.
Solution
def make_solver(m): parts = [(p, e, p**e) for p, e in factorise(m).items()] tables = {(p,e): build_period(p,e) for p,e,_ in parts} def solve(n, k): rems = [binom_pp(n, k, p, e, tables[(p,e)]) for p,e,_ in parts] mods = [pe for _,_,pe in parts] return crt(rems, mods)[0] return solvePrecompute the period tables once (total size ∑ p^e ≤ m ≤ 10⁶), then each query is O(ω(m)log_p n) — a few hundred operations. 10⁵ queries is well under a second.
The design point: separating setup from query is what makes this practical. Factoring m and building tables per query would be 10⁵ factorisations; doing it once amortises to nothing.
- Program
Count the number of k ∈ [0,n] with C(n,k) ≡ 0 ±od p, for n ≤ 10^{18}.
Solution
def zero_entries(n, p): total = n + 1 nonzero = 1 m = n while m: nonzero *= (m % p) + 1 m //= p return total - nonzeroO(log_p n) using problem 7's formula. At n = 10^{18} and p = 2, the nonzero count is 2^{popcount(n)} — typically around 2^{30}, so more than 99.9999% of the row is even.
A nice sanity check: for n = p^m - 1 every digit is p-1, so nonzero = p^m = n+1 and there are zero zeros — the full row is nonzero, which matches the fact that C(p^m-1, k) ≡ (-1)^k ±od p.
- Program
Verify Kummer's theorem computationally: for random n, k and p, compare the carry count to the actual p-adic valuation.
Solution
def carries(a, b, p): c = cnt = 0 while a or b: s = a % p + b % p + c c = 1 if s >= p else 0 cnt += c a //= p; b //= p return cnt def valuation(x, p): v = 0 while x % p == 0: x //= p; v += 1 return v from math import comb import random for _ in range(20000): n = random.randrange(1, 400); k = random.randrange(0, n+1) p = random.choice([2,3,5,7,11]) assert carries(k, n-k, p) == valuation(comb(n,k), p)The identity v_p(C(n,k)) = carries(k, n-k) in base p is exactly Kummer, and 20,000 random checks make it concrete. The
carriesfunction must run until both operands and the carry are exhausted — a final carry out of the top position counts, and forgetting it is the one bug here.Practical use: the valuation tells you immediately whether C(n,k) ≡ 0 ±od {p^e} without computing anything, which is the early exit in the prime-power algorithm.
- Program
Given n ≤ 10^{18}, count the number of odd entries in the first n rows of Pascal's triangle.
Solution
def odd_entries_upto(n): # rows 0..n-1 # Σ_{m < n} 2^{popcount(m)} — digit DP over the binary representation total = 0 ones = 0 for i in range(n.bit_length()-1, -1, -1): if n >> i & 1: total += (3**i) * (1 << ones) # this bit 0, rest free ones += 1 return totalWhy 3^i: fixing a prefix and letting the remaining i bits vary freely, ∑_{free bits} 2^{popcount} = (1+2)^i = 3^i — each free bit contributes a factor of 1 + 2.
So the total over the first 2^m rows is 3^m, giving the Sierpiński dimension log₂3 ≈ 1.585: the odd entries occupy n^{1.585} of the n²/2 positions, a vanishing fraction. This digit-DP pattern — walk the bits, count what the free suffix contributes — is the general technique for any "count over all m < n" problem with a digit-based condition.
- Program
Implement a fallback-chain
binom_mod(n, k, m)that automatically selects the right algorithm.Solution
def binom_mod(n, k, m, cache={}): if k < 0 or k > n: return 0 if m == 1: return 0 key = m if key not in cache: f = factorise(m) if len(f) == 1: (p, e), = f.items() if e == 1 and n < p: cache[key] = ('direct', p) elif e == 1: cache[key] = ('lucas', p) else: cache[key] = ('pp', p, e) else: cache[key] = ('crt', f) kind = cache[key][0] if kind == 'direct': return direct_binom(n, k, m) if kind == 'lucas': return lucas(n, k, cache[key][1]) if kind == 'pp': return binom_pp(n, k, *cache[key][1:]) return crt_binom(n, k, cache[key][1])The dispatch encodes the Principle box as executable logic, and caching on m means the factorisation and table building happen once per distinct modulus.
Worth adding: an assertion in the
'direct'branch that n < p still holds, since a later call with a larger n through the same cached path would silently produce garbage. Caching the decision rather than re-deriving it is a small optimisation with a real correctness hazard, and the assertion is what makes it safe.
Inclusion-exclusion
Counting things that avoid every one of k bad properties is done by counting things that violate each subset of them, with alternating signs. It converts an intractable "avoid everything" problem into 2^k easy "at least these" problems.
Proof: fix an object x belonging to exactly m ≥ 1 of the sets. On the right it is counted once for each subset S of those m sets, with sign (-1)^{|S|}, so its total contribution is ∑_{j=0}^{m}C(m,j)(-1)^j = (1-1)^m = 0. An object in none of the sets contributes exactly 1 (from S = ∅). So the right side counts exactly the objects with no property. ∎
That binomial cancellation is the same one behind Möbius inversion (Chapter 25) — inclusion–exclusion is Möbius inversion over the subset lattice.
Inclusion–exclusion is worth using when N(S) — the count with at least the properties in S — is easy, which is usually the case when the properties are "divisible by d_i", "misses colour i", or "position i is fixed". If N(S) depends only on |S|, the 2^n sum collapses to n terms with a binomial factor — which is what makes derangements and surjections closed-form.
The standard applications
Bonferroni inequalities
Truncating the alternating sum after an even number of terms gives an upper bound; after an odd number, a lower bound:
This matters when the full sum is intractable: the union bound P(∪ A_i) ≤ ∑ P(A_i) is the first Bonferroni inequality and is used constantly in probabilistic analysis (Chapter 38).
Computing it
# plain: 2^n subsets, each evaluated in O(n) → O(2^n · n)
ans = 0
for mask in range(1 << n):
sign = -1 if bin(mask).count('1') % 2 else 1
ans += sign * N(mask)
# sum over subsets (SOS / zeta transform): O(2^n · n) for ALL masks at once
f = base[:] # f[mask] = value at exactly this mask
for i in range(n):
for mask in range(1 << n):
if mask >> i & 1:
f[mask] += f[mask ^ (1 << i)] # now f[mask] = Σ_{S ⊆ mask}
# Möbius (inverse) transform: same loop with -= recovers `base`The zeta/Möbius transform pair is the subset-lattice analogue of prefix sums and differences. It computes all 2^n subset sums in O(2^n n) rather than O(3^n), and it is the engine behind subset convolution and many bitmask DPs.
"How many numbers up to n are divisible by 2, 3, or 5?" is a three-set inclusion–exclusion done in your head: ⌊ n/2⌋ + ⌊ n/3⌋ + ⌊ n/5⌋ - ⌊ n/6⌋ - ⌊ n/10⌋ - ⌊ n/15⌋ + ⌊ n/30⌋. The follow-up "now with 20 divisors" wants 2^{20} subsets — a million terms, still instant — and the observation that lcm overflow is the real hazard.
- Theory
How many integers in [1, 1000] are divisible by none of 2, 3, 5?
Solution
1000 - (500 + 333 + 200) + (166 + 100 + 66) - 33 = 1000 - 1033 + 332 - 33 = 266.
Check against the density argument: 1000 · \frac12·\frac23·\frac45 = 1000·(4)/(15) = 266.67 ✓ — the multiplicative form of the same computation, and exactly φ(30)/30 · 1000 since φ(30) = 8 and 8/30 = 4/15.
The floors are why the two differ slightly: inclusion–exclusion with floors is exact, the density product is asymptotic.
- Theory
Derive the derangement formula D_n = n!∑_{k=0}^{n}((-1)^k)/(k!).
Solution
Property A_i: "σ(i) = i". Then D_n counts permutations with no property. For a set S of size k, permutations fixing at least those k points number (n-k)!, and there are C(n,k) such sets. Since N(S) depends only on |S|, the 2^n sum collapses:
D_n = ∑_{k=0}^{n}(-1)^k C(n,k)(n-k)! = ∑_{k=0}^{n}(-1)^k(n!)/(k!). ∎
Since ∑_k(-1)^k/k! → e^{-1}, we get D_n = [(n!)/(e)] (nearest integer) for n ≥ 1, and the probability a random permutation is a derangement tends to 1/e ≈ 0.368 — remarkably fast, correct to 6 decimals by n = 10.
- Theory
Count surjections from an n-set onto a k-set.
Solution
Property A_i: "element i of the codomain is missed". Functions missing at least a given set of j codomain elements number (k-j)^n. So
Sur(n,k) = ∑_{j=0}^{k}(-1)^j C(k,j)(k-j)^n.
Equivalently k!S(n,k) where S is a Stirling number of the second kind: choose the partition into k unlabelled blocks, then label them. ∎
Sanity: Sur(n,n) = n! ✓, and Sur(n,k) = 0 for k > n — the alternating sum genuinely cancels to zero, which is a good test of an implementation.
- Theory
Count permutations of 1..n with no element in its original position and none in position i+1 (the "ménage-lite" problem).
Solution
Forbid σ(i) ∈ {i, i+1} — 2n-1 forbidden cells forming a "staircase" board. By the rook-polynomial method, the count is
∑_{k}(-1)^k r_k (n-k)!
where r_k is the number of ways to place k non-attacking rooks on the forbidden board. For this staircase, the forbidden cells form a path, and r_k = C(2n-1-k, k) — the "choose k non-adjacent cells from a path of 2n-1" count of Chapter 28 problem 6.
So the answer is ∑_k (-1)^k C(2n-1-k,k)(n-k)!. This is the general shape of "permutations avoiding a forbidden board": inclusion–exclusion over rook placements, with the rook polynomial encoding all the intersection sizes at once.
- Theory
State and prove the first two Bonferroni inequalities.
Solution
∑|A_i| - ∑_{i<j}|A_i∩ A_j| ≤ |∪ A_i| ≤ ∑ |A_i|.
Proof: an element in exactly m of the sets contributes ∑_{j=1}^{t}(-1)^{j+1}C(m,j) to the sum truncated after t terms. The partial sums of ∑_j (-1)^{j+1}C(m,j) alternate around the limit 1, being ≥ 1 after an odd number of terms and ≤ 1 after an even number. Summing over elements gives the inequalities. ∎
The t=1 case is the union bound, used everywhere in probability: P(∪ A_i) ≤ ∑ P(A_i). It is crude but requires no independence, which is exactly why it survives in settings where nothing else does.
- Theory
Show that inclusion–exclusion is Möbius inversion over the subset lattice.
Solution
Let f(S) be the number of objects having exactly the property set S, and g(S) = ∑_{T \supseteq S} f(T) the number having at least S. The general Möbius inversion (Chapter 25 problem 7) over the subset lattice, whose Möbius function is μ(S,T) = (-1)^{|T|-|S|}, gives
f(S) = ∑_{T\supseteq S}(-1)^{|T|-|S|}g(T).
Taking S = ∅ yields f(∅) = ∑_T (-1)^{|T|}g(T) — exactly the complementary form of inclusion–exclusion. ∎
The practical payoff of seeing them as one theorem: the zeta/Möbius transform code computes all f(S) simultaneously in O(2^n n), not just f(∅) — so problems asking "for each subset, how many objects have exactly these properties" are no harder than the single-answer version.
- Theory
Derive the chromatic polynomial via inclusion–exclusion over edges.
Solution
Count k-colourings where every edge is properly coloured. Property A_e: "edge e is monochromatic". Colourings satisfying at least the edges in a set S ⊆ E are exactly the colourings constant on each connected component of the graph (V,S), of which there are k^{c(S)} where c(S) is the component count.
P(G,k) = ∑_{S⊆ E}(-1)^{|S|}k^{c(S)}. ∎
This is Whitney's theorem, and it shows immediately that P(G,k) is a polynomial in k of degree |V| with leading coefficient 1 and alternating signs. It is exponential in |E| to evaluate directly; the deletion–contraction recurrence P(G,k) = P(G-e,k) - P(G/e,k) is usually better, and both are outperformed by the O(2^n n) subset-DP for chromatic number.
- Program
Count integers in [1,n] divisible by none of k given numbers, with k ≤ 20 and n ≤ 10^{18}.
Solution
from math import lcm def count_none(n, ds): k = len(ds); total = 0 for mask in range(1 << k): L = 1; bits = 0 for i in range(k): if mask >> i & 1: L = lcm(L, ds[i]); bits += 1 if L > n: break # early exit: contributes 0 if L > n: continue total += (-1)**bits * (n // L) return totalO(2^k k) — about 2×10⁷ at k=20, a second or two. The early exit when L > n both saves time and, crucially, prevents overflow: the lcm of 20 numbers can exceed any fixed-width type even when every term contributes nothing.
In C++ compute the lcm as
L / gcd(L, d) * dand checkL > n / dbefore multiplying, so the overflow never happens rather than being detected after the fact. - Program
Implement the subset-sum (zeta) and Möbius transforms, and verify they are inverse.
Solution
def zeta(f, n): # f[mask] → Σ_{S ⊆ mask} f[S] f = f[:] for i in range(n): for mask in range(1 << n): if mask >> i & 1: f[mask] += f[mask ^ (1 << i)] return f def mobius(g, n): # the exact inverse g = g[:] for i in range(n): for mask in range(1 << n): if mask >> i & 1: g[mask] -= g[mask ^ (1 << i)] return g import random n = 12; f = [random.randrange(-100, 100) for _ in range(1 << n)] assert mobius(zeta(f, n), n) == fO(2^n n) each. The two loops are identical except for the sign — which is exactly the statement that the zeta and Möbius transforms are inverse, dimension by dimension, like a multidimensional prefix sum and its difference.
The loop order matters: iterating i outermost processes one "dimension" at a time, and each mask reads a value already updated for dimensions < i but not for i. Swapping the loops gives a different (wrong) transform.
- Program
Count the permutations of n elements avoiding a given set of forbidden (position, value) pairs, using the permanent and Ryser's formula.
Solution
# perm(A) = Σ_{S ⊆ [n]} (-1)^{n-|S|} ∏_i (Σ_{j ∈ S} A[i][j]) def ryser(A): n = len(A); total = 0 for mask in range(1, 1 << n): prod = 1 for i in range(n): s = 0 for j in range(n): if mask >> j & 1: s += A[i][j] prod *= s if prod == 0: break bits = bin(mask).count('1') total += prod if (n - bits) % 2 == 0 else -prod return totalWith A[i][j] = 1 iff position i may hold value j, the permanent counts exactly the valid permutations. Ryser's formula is O(2^n n²), or O(2^n n) using Gray-code order so each mask differs from the previous by one bit and the row sums update incrementally.
Computing the permanent is #P-complete (Valiant), so no polynomial algorithm is expected — in sharp contrast to the determinant, whose only difference is the missing signs, and which is O(n³). That contrast is one of the most striking facts in complexity theory.
- Program
Count the number of ways to place n non-attacking rooks on a board with forbidden squares, for n ≤ 20.
Solution
# bitmask DP over which columns are used — O(2^n · n) def rook_placements(allowed): # allowed[i] = bitmask of legal columns n = len(allowed) dp = [0]*(1 << n); dp[0] = 1 for mask in range(1 << n): if not dp[mask]: continue row = bin(mask).count('1') if row == n: continue free = allowed[row] & ~mask while free: b = free & -free dp[mask | b] += dp[mask] free ^= b return dp[(1 << n) - 1]O(2^n n) — about 2×10⁷ at n = 20. Note
row = popcount(mask): the number of columns used equals the number of rows filled, so the row index is determined by the mask and needs no extra dimension. That observation halves the state space and is the standard trick for assignment-style bitmask DPs.This computes the same permanent as the previous problem but in O(2^n n) rather than O(2^n n²), and it is easier to get right. Use Ryser when memory for 2^n longs is the constraint.
- Program
Count derangements D_n mod 10⁹+7 for n ≤ 10⁷, three ways, and compare.
Solution
MOD = 10**9+7 # (a) recurrence D_n = (n-1)(D_{n-1} + D_{n-2}) — O(n), no factorials def der_rec(n): a, b = 1, 0 # D_0, D_1 for i in range(2, n+1): a, b = b, (i-1)*(a+b) % MOD return b if n else 1 # (b) recurrence D_n = n·D_{n-1} + (-1)^n — O(n), one multiplication def der_rec2(n): d = 1 for i in range(1, n+1): d = (i*d + (1 if i % 2 == 0 else -1)) % MOD return d # (c) inclusion–exclusion sum — O(n) with precomputed inverse factorials def der_ie(n, fact, ifact): return fact[n] * sum((-1)**k * ifact[k] for k in range(n+1)) % MODAll three are O(n) and must agree. (b) is fastest — one multiplication and one addition per step. (a) has the nicest combinatorial proof: element n swaps with some j (n-1 choices), and then either j goes to n's place (leaving D_{n-2}) or not (leaving D_{n-1}).
(c) is the inclusion–exclusion form and is the one to use when you need D_n for a single huge n with factorials already precomputed. Cross-checking all three against each other for n ≤ 10⁵ is a strong test.
- Program
Given n sets described by predicates, compute for every subset how many elements satisfy exactly that subset of predicates.
Solution
def exact_counts(elements, preds): n = len(preds) exact = [0]*(1 << n) for x in elements: m = 0 for i, p in enumerate(preds): if p(x): m |= 1 << i exact[m] += 1 return exact # already "exactly", by construction # and if you are GIVEN the "at least" counts instead: def exact_from_atleast(atleast, n): return superset_mobius(atleast, n) # the ⊇ version of the Möbius transformWhen you can evaluate the predicates per element, computing the exact counts directly is O(|E|n) and inclusion–exclusion is unnecessary. Inclusion–exclusion earns its place only when the "at least" counts are available in closed form while individual elements are too numerous to enumerate — as with "integers up to 10^{18} divisible by d_i".
Knowing which situation you are in prevents the common mistake of writing a 2^n alternating sum when a single linear pass would do.
- Program
Solve: count the strings of length n over an alphabet of size k that use every letter at least once, for n ≤ 10⁶ and k ≤ 10⁶.
Solution
MOD = 10**9+7 def surjections(n, k, fact, ifact): if k > n: return 0 total = 0 for j in range(k+1): term = fact[k] * ifact[j] % MOD * ifact[k-j] % MOD * pow(k-j, n, MOD) % MOD total = (total + (term if j % 2 == 0 else -term)) % MOD return totalO(k log n) — the exponentiations dominate. At k = 10⁶ that is 3×10⁷ multiplications, a couple of seconds; precomputing j^n for all j via a linear sieve on the completely multiplicative function j ↦ j^n reduces it to O(k + π(k)log n).
Sanity checks that catch sign errors: Sur(n,1) = 1, Sur(n,n) = n!, and Sur(n,k) = 0 for k > n — the last one especially, since it requires the alternating sum to cancel exactly and will expose any modular-sign mistake immediately.
- Program
Implement subset convolution: given f and g on subsets, compute h(S) = ∑_{T ⊆ S} f(T)g(S\setminus T) in O(2^n n²).
Solution
def subset_convolution(f, g, n): N = 1 << n # rank by popcount so that "disjoint union" becomes "sizes add" fh = [[0]*N for _ in range(n+1)] gh = [[0]*N for _ in range(n+1)] for m in range(N): fh[bin(m).count('1')][m] = f[m] gh[bin(m).count('1')][m] = g[m] for i in range(n+1): fh[i] = zeta(fh[i], n); gh[i] = zeta(gh[i], n) hh = [[0]*N for _ in range(n+1)] for i in range(n+1): for j in range(n+1-i): for m in range(N): hh[i+j][m] += fh[i][m]*gh[j][m] return [mobius(hh[bin(m).count('1')], n)[m] for m in range(N)]The trick: the ordinary zeta transform computes the subset convolution over all pairs T ⊆ S, including overlapping ones. Adding the popcount as a second index forces |T| + |S\setminus T| = |S|, which happens exactly when T and S\setminus T are disjoint — so the rank bookkeeping selects the disjoint pairs.
O(2^n n²) versus O(3^n) for the naive submask enumeration. At n = 20 that is 4×10⁸ against 3.5×10⁹ — the difference between feasible and not, and it is what makes set-cover and graph-colouring DPs run at n = 20.
Derangements, occupancy and the ballot problem
A family of classic counting problems that share one shape: an inclusion-exclusion or a reflection argument turns an "avoid this" condition into a closed form. They recur constantly, so it is worth knowing them by name.
Derangements
A derangement is a permutation with no fixed point. From Chapter 30:
Permutations with exactly k fixed points: C(n,k)D_{n-k}. Summing over k recovers n!, and dividing by n! gives a distribution converging to Poisson(1) — mean 1, variance 1, as computed in Chapter 27.
Occupancy: balls in boxes
| Question (n balls, k boxes) | Count |
|---|---|
| distinct balls, any distribution | k^n |
| distinct balls, no box empty | k!S(n,k) = ∑_j(-1)^jC(k,j)(k-j)^n |
| identical balls, any | C(n+k-1,k-1) |
| identical balls, no box empty | C(n-1,k-1) |
| identical, each box ≤ c | inclusion–exclusion on overfull boxes |
| expected empty boxes (n distinct, random) | k(1-1/k)^n ≈ k e^{-n/k} |
| expected max load (n = k, random) | Θ(log n/loglog n) |
The last two are the balls-in-bins results that govern hash tables, load balancers and sharding. With n = k, about n/e ≈ 37% of boxes are empty and the fullest holds Θ(log n/loglog n) — which is why a single hash function gives noticeably uneven load and why "power of two choices" (place in the lesser-loaded of two random bins, dropping the max to Θ(loglog n)) is such a large practical win.
The ballot problem and the reflection principle
In an election where A gets p votes and B gets q < p, the probability A is strictly ahead throughout the count is (p-q)/(p+q).
Proof by reflection: a path that ties at some point can be reflected before its first tie, giving a bijection between the bad paths starting with A and the bad paths starting with B. Since every path starting with B is bad, the bad paths number 2·(q)/(p+q) of the total, leaving 1 - (2q)/(p+q) = (p-q)/(p+q). ∎
The reflection principle counts lattice paths avoiding a line: reflect the prefix up to the first touch, obtaining a bijection with all paths to a reflected endpoint. It gives the ballot problem, the Catalan numbers, the distribution of the maximum of a random walk, and the hitting-time formulas — all as one argument with different endpoints.
"Hat check problem": n people, hats returned at random, probability nobody gets their own is D_n/n! → 1/e. The surprising part worth stating is that it is essentially independent of n — 0.375 at n=4 and 0.3679 at n=∞. Intuition says more people should make it harder; it does not.
- Theory
Prove D_n = n D_{n-1} + (-1)^n from the closed form.
Solution
D_n = n!∑_{k=0}^{n}((-1)^k)/(k!) = n[(n-1)!∑_{k=0}^{n-1}((-1)^k)/(k!)] + n!·((-1)^n)/(n!) = nD_{n-1} + (-1)^n. ∎
A combinatorial proof also exists but is subtler: it pairs derangements of n with derangements of n-1 via a near-bijection that fails by exactly one object, and the sign tracks which direction it fails in. The algebraic derivation is the honest one here.
- Theory
Prove the combinatorial recurrence D_n = (n-1)(D_{n-1}+D_{n-2}).
Solution
In a derangement σ of {1..n}, element 1 maps to some j ≠ 1 — n-1 choices. Two cases:
- σ(j) = 1 (a 2-cycle): the remaining n-2 elements form a derangement — D_{n-2} ways.
- σ(j) ≠ 1: consider the map on {2,…,n} obtained by sending j to σ(1)'s role — formally, derangements of {2..n} where j is forbidden from mapping to 1, which is D_{n-1} by relabelling.
So D_n = (n-1)(D_{n-2} + D_{n-1}). ∎
This recurrence needs no factorials or inverses, so it is the one to use for computing D_n mod m for a composite m where inverse factorials are unavailable.
- Theory
What is the probability that a random permutation of 52 cards has at least one fixed point? Compute to 6 decimals.
Solution
1 - D_{52}/52! = 1 - ∑_{k=0}^{52}((-1)^k)/(k!).
The alternating series converges to e^{-1} with error at most the first omitted term, 1/53! ≈ 10^{-70}. So the answer is 1 - e^{-1} = 0.632121 to far more precision than asked.
Practical reading: shuffling a deck and dealing it against a reference order, there is a 63% chance some card lands in its original position. Because the convergence is so fast, the same answer holds for 5 cards (0.6333) and for a million.
- Theory
n balls into n boxes uniformly at random. Find the expected number of empty boxes and the expected number of boxes with exactly one ball.
Solution
Empty: box i is empty with probability (1-1/n)^n → e^{-1}. By linearity, E[empty] = n(1-1/n)^n ≈ n/e ≈ 0.368n.
Exactly one: probability n · \frac1n(1-\frac1n)^{n-1} = (1-1/n)^{n-1} → e^{-1}, so also ≈ n/e.
So about 37% of boxes are empty, 37% have exactly one ball, and 26% have two or more — the Poisson(1) distribution again, since the load of a single box is Bin(n,1/n) → Poisson(1).
This is the load distribution of a hash table at load factor 1, and it is why chaining hash tables have a meaningful fraction of empty buckets and a meaningful fraction of collisions even at "perfect" load.
- Theory
Prove the ballot theorem: A leads throughout with probability (p-q)/(p+q).
Solution
Represent a count as a path of +1 (A) and -1 (B) steps, ending at p-q > 0. Total paths: C(p+q,p).
A path is bad if it ever returns to 0. Every path starting with a B vote is bad. For paths starting with an A vote that later tie, reflect the segment up to the first tie (swap all signs): this maps them bijectively onto paths starting with B. So bad paths starting with A are equinumerous with all paths starting with B, and
P(bad) = 2P(first vote is B) = (2q)/(p+q), hence P(good) = (p-q)/(p+q). ∎
Note the elegance: the answer depends only on the final margin relative to the total, not on the path shape. Setting p = q+1 gives (1)/(2q+1), and the corresponding count is the Catalan number.
- Theory
How many ways can n couples be seated so that no person sits next to their partner in a row of 2n seats?
Solution
Inclusion–exclusion on the set of couples seated adjacently. If a specific set of k couples is glued (each glued pair can be ordered 2 ways), we have 2n-k objects to arrange:
∑_{k=0}^{n}(-1)^k C(n,k)2^k(2n-k)!.
Check n=1: 2! - 1·2·1! = 2-2 = 0 ✓ (two seats, one couple — they must be adjacent). Check n=2: 4! - 2·2·3! + 1·4·2! = 24-24+8 = 8.
The circular version with alternating genders is the classical ménage problem, whose answer involves the ménage numbers U_n = ∑_k(-1)^k(2n)/(2n-k)C(2n-k,k)(n-k)! — the extra difficulty is that the forbidden positions wrap around, so the rook board is a cycle rather than a path.
- Theory
Show that the expected maximum load when n balls go into n bins is Θ(log n/loglog n).
Solution
Upper bound: P(bin i has ≥ k) ≤ C(n,k)n^{-k} ≤ (1)/(k!) ≤ ((e)/(k))^k. Union bound over n bins: P(max ≥ k) ≤ n(e/k)^k. Setting k = (3log n)/(loglog n) makes this o(1).
Lower bound: a second-moment argument shows some bin reaches (log n)/(loglog n)(1-o(1)) with high probability.
So the max load is (1+o(1))(log n)/(loglog n) — about 5 for n = 10⁶. ∎
Power of two choices: placing each ball in the lesser-loaded of two random bins drops this to (loglog n)/(log 2)+O(1) — about 5 as well at n=10⁶, but growing exponentially more slowly. This is the theory behind "two random choices" load balancers, and the gain from one to two choices is far larger than from two to three.
- Program
Compute D_n exactly for n ≤ 1000 and verify D_n = round(n!/e).
Solution
from math import factorial from fractions import Fraction D = [1, 0] for n in range(2, 1001): D.append((n-1)*(D[n-1] + D[n-2])) from decimal import Decimal, getcontext getcontext().prec = 3000 e = Decimal(1).exp() for n in range(1, 1001): assert D[n] == int((Decimal(factorial(n)) / e).quantize(Decimal(1)))Decimalwith 3000 digits of precision is needed because 1000! has 2568 digits and the rounding must be exact — afloatgives up at n = 170. This is a good illustration that "nearest integer to n!/e" is a genuine identity requiring exact arithmetic to verify, not a numerical approximation.The identity holds because the tail n!∑_{k>n}(-1)^k/k! has absolute value below 1/(n+1) < 1/2 for n ≥ 1.
- Program
Count permutations of n with exactly k fixed points, for n ≤ 10⁶, modulo 10⁹+7.
Solution
MOD = 10**9+7 def fixed_points(n, k, fact, ifact, D): if k > n: return 0 return fact[n] * ifact[k] % MOD * ifact[n-k] % MOD * D[n-k] % MOD # derangements by the (-1)^n recurrence, O(n), no inverses needed D = [1]*(n+1) for i in range(1, n+1): D[i] = (i*D[i-1] + (1 if i % 2 == 0 else -1)) % MODO(1) per query after O(n) precomputation. Verify that ∑_k fixed(n,k) ≡ n!, which exercises every k including k = n-1 (always 0, since fixing n-1 points forces the last).
The distribution fixed(n,k)/n! = \frac{D_{n-k}}{k!(n-k)!}·(n!)/(n!) → \frac{e^{-1}}{k!} is Poisson(1), and plotting it for n = 20 shows the convergence is already visually perfect.
- Program
Simulate balls into bins for n = 10⁶ and compare the empirical max load to the theoretical log n/loglog n; then simulate the two-choice variant.
Solution
import random, math def one_choice(n): bins = [0]*n for _ in range(n): bins[random.randrange(n)] += 1 return max(bins) def two_choice(n): bins = [0]*n for _ in range(n): i, j = random.randrange(n), random.randrange(n) if bins[j] < bins[i]: i = j bins[i] += 1 return max(bins) n = 10**6 print(one_choice(n), math.log(n)/math.log(math.log(n))) # ≈ 8 vs 5.2 print(two_choice(n), math.log(math.log(n))/math.log(2)) # ≈ 4 vs 3.8The one-choice max lands around 8 (the asymptotic constant approaches 1 slowly, and the o(1) is still substantial at 10⁶); the two-choice max is 4. Increasing n by a factor of 1000 raises the one-choice max by about 2 and the two-choice max by 0 or 1 — the double-logarithm in action.
The practical lesson for consistent hashing and load balancers: querying two backends and picking the less loaded gives an exponential improvement in tail load for one extra probe, and adding a third choice gives almost nothing more.
- Program
Count lattice paths from (0,0) to (n,n) that never touch the diagonal after the start, and verify the ballot theorem numerically.
Solution
from math import comb def strictly_above(p, q): # A gets p, B gets q, A strictly ahead throughout return (p - q) * comb(p+q, p) // (p + q) # brute-force verification from itertools import permutations def brute(p, q): cnt = 0 for arrangement in set(permutations([1]*p + [-1]*q)): s = 0; ok = True for v in arrangement: s += v if s <= 0: ok = False; break cnt += ok return cnt for p in range(1, 9): for q in range(p): assert strictly_above(p, q) == brute(p, q), (p, q)The formula (p-q)/(p+q)C(p+q,p) is the ballot number, and the brute force over all distinct arrangements confirms it for every small case. Note
s <= 0, nots < 0: "strictly ahead" means never tied, and confusing the two gives the Catalan-adjacent count instead.Setting q = p-1 gives (1)/(2p-1)C(2p-1,p) = C_{p-1}, the Catalan number — the bridge to Chapter 32.
- Program
Count the ways to distribute n identical items into k boxes with box i holding between l_i and u_i items.
Solution
from math import comb def bounded(n, lows, highs): k = len(lows) n -= sum(lows) # shift away the lower bounds caps = [h - l for h, l in zip(highs, lows)] if n < 0: return 0 total = 0 for mask in range(1 << k): # inclusion–exclusion over exceeded caps s, bits = n, 0 for i in range(k): if mask >> i & 1: s -= caps[i] + 1; bits += 1 if s < 0: continue total += (-1)**bits * comb(s + k - 1, k - 1) return totalTwo steps: substitute x_i' = x_i - l_i to remove the lower bounds (pure shift), then inclusion–exclusion over which upper bounds are violated. O(2^k k), practical to k ≈ 20.
For larger k with small caps, a DP over the running sum is O(kn) and better. As so often, the two methods are complementary — exponential in the number of constraints versus polynomial in the total — and the right one depends on which parameter is small.
- Program
Given 10⁵ queries (n,k), compute the probability that a random permutation of n has exactly k fixed points, to double precision, without big integers.
Solution
import math def prob_fixed(n, k): # P = C(n,k) D_{n-k} / n! = (1/k!) Σ_{j=0}^{n-k} (-1)^j / j! if k > n: return 0.0 s, term = 0.0, 1.0 for j in range(n-k+1): s += term term *= -1.0 / (j+1) if abs(term) < 1e-18: break # the series converges very fast return s / math.factorial(k) if k < 20 else math.exp(-math.lgamma(k+1))*sThe inner series converges to e^{-1} within 10^{-18} after about 20 terms regardless of n, so each query is effectively O(1). For k ≥ 20, k! overflows the useful range of a
double, so switch tolgamma.The whole computation never touches a number larger than about 1 — which is the general lesson for probability code: work with the ratio directly rather than computing a huge numerator and denominator and dividing.
- Program
Write a uniform random derangement generator in expected O(n) time (not by rejection).
Solution
import random def random_derangement(n): a = list(range(n)); mark = [False]*n i, u = n-1, n while u >= 2: if not mark[i]: while True: j = random.randrange(i) if not mark[j]: break a[i], a[j] = a[j], a[i] # with probability (u-1)·D_{u-2}/D_u, fix j out p = (u-1) * D[u-2] / D[u] if random.random() < p: mark[j] = True; u -= 1 u -= 1 i -= 1 return aThis is the Martínez–Panholzer–Prodinger algorithm. It follows the recurrence D_n = (n-1)(D_{n-1}+D_{n-2}) exactly, branching with the correct probability at each step, so the output is provably uniform over the D_n derangements.
Rejection sampling (shuffle until no fixed point) is simpler and accepts with probability → 1/e, so it needs 2.72 attempts on average — also expected O(n). The direct algorithm wins only when you need a guaranteed bound rather than an expectation, or when n is enormous and the constant matters. Verify uniformity for n = 5 (44 derangements) with a chi-square test over 10⁶ samples.
Catalan numbers and their bijections
One sequence counts balanced brackets, binary trees, triangulations, monotone lattice paths below the diagonal and a dozen other things - because they are all the same object seen from different angles. Recognising a Catalan structure often replaces a DP with a formula.
The things Catalan numbers count
| Object of size n | The bijection to balanced brackets |
|---|---|
| Balanced bracket sequences with n pairs | the canonical form |
| Monotone paths (0,0)→(n,n) staying weakly below y=x | R = (, U = ) |
| Binary trees with n internal nodes | preorder: node = ( left ) right |
| Full binary trees with n+1 leaves | same tree, counted by leaves |
| Triangulations of a convex (n+2)-gon | dual tree of the triangulation |
| Ways to parenthesise n+1 factors | the parse tree |
| Non-crossing chord pairings of 2n points | matched brackets = matched chords |
| Permutations of n avoiding the pattern 231 | stack-sortable permutations |
| Standard Young tableaux of shape 2× n | row 1 = ( positions |
| Dyck words: n up-steps, n down, never below 0 | identical |
The stack-sortable permutations row is the practically useful one: a sequence can be sorted by a single stack exactly when it avoids the pattern 231, and the number of such permutations of n is C_n. That is why "can this be produced by a stack?" problems are Catalan problems.
Two derivations of the formula
Reflection. Paths from (0,0) to (n,n) number C(2n,n). A bad path crosses above y = x, so it touches y = x+1; reflecting the prefix up to the first touch across that line gives a bijection with all paths to (n-1, n+1), of which there are C(2n, n-1). Subtracting,
Generating function. The convolution recurrence C_{n+1} = ∑ C_i C_{n-i} says C(x) = 1 + xC(x)² where C(x) = ∑ C_n x^n. Solving the quadratic and picking the branch that is finite at 0:
Expanding √{1-4x} by the binomial series recovers C_n = (1)/(n+1)C(2n,n). Chapter 35 develops this machinery in general.
The convolution recurrence a_{n+1} = ∑_i a_i a_{n-i} is the signature of a Catalan structure: it says "the object splits at a distinguished point into two independent smaller objects of the same kind". If your DP has that shape, the answer is Catalan and you can replace O(n²) with a closed form.
Relatives
"Count the valid parentheses strings of length 2n", "count BSTs with n nodes", and "count ways to triangulate a polygon" are the same question. The answer that lands is naming the bijection, not the formula: a BST is determined by which key is the root, splitting into independent left and right subtrees — that is exactly the Catalan convolution.
- Theory
Prove C_n = C(2n,n) - C(2n,n-1) by the reflection principle.
Solution
Encode a bracket string as a path:
(is a step right,)a step up. All strings with n of each give C(2n,n) paths from (0,0) to (n,n). A string is invalid iff some prefix has more)than(, i.e. the path strictly crosses y = x, i.e. it touches y = x+1.Reflect the portion of a bad path before its first touch of y = x+1 across that line. The start (0,0) maps to (-1,1), and the rest is unchanged, so bad paths correspond bijectively to all paths from (-1,1) to (n,n) — which number C(2n, n-1). The map is reversible (reflect back at the first touch), so it is a bijection. ∎
Hence C_n = C(2n,n) - C(2n,n-1), and factoring gives (1)/(n+1)C(2n,n).
- Theory
Prove the convolution recurrence C_{n+1} = ∑_{i=0}^{n}C_iC_{n-i} combinatorially.
Solution
Take a balanced string with n+1 pairs. Its first character is
(; let its matching)sit at position 2i+2. The content strictly inside that pair is a balanced string with i pairs; everything after the matching)is a balanced string with n-i pairs. Both are arbitrary and independent.So C_{n+1} = ∑_{i=0}^{n}C_i C_{n-i}. ∎
In tree language: a binary tree with n+1 nodes has a root, a left subtree with i nodes and a right subtree with n-i. Same recurrence, and the fact that these are the same recurrence is the bijection between bracket strings and trees.
- Theory
How many binary search trees can be built from n distinct keys? How many shapes of binary tree with n nodes?
Solution
Both are C_n, and it is the same count for a reason worth stating: once the shape is fixed, the BST property determines the key placement uniquely (in-order traversal must be sorted). So BSTs and shapes are in bijection.
Choosing which key is the root splits the remaining keys into those smaller and those larger, giving ∑_{i}C_{i}C_{n-1-i} — the Catalan convolution again.
Contrast with heap-ordered trees on n distinct keys with a fixed shape: exactly one arrangement per shape as well, but the number of binary heaps on a complete shape is a different count entirely (the hook length formula), which is why "how many valid heaps" is not a Catalan question.
- Theory
Show that the number of triangulations of a convex (n+2)-gon is C_n.
Solution
Fix an edge of the polygon, say v_0v_{n+1}. In any triangulation this edge lies in exactly one triangle, v_0 v_k v_{n+1} for some 1 ≤ k ≤ n. That triangle splits the polygon into a (k+1)-gon on one side and an (n-k+2)-gon on the other, each independently triangulated.
Writing T_m for the triangulations of an (m+2)-gon, we get T_n = ∑_{k=1}^{n} T_{k-1}T_{n-k} with T_0 = 1 — the Catalan recurrence. ∎
Every triangulation of an (n+2)-gon uses exactly n-1 diagonals and produces n triangles, regardless of the choices — a useful invariant, provable by induction on the same split.
- Theory
Prove that a permutation is stack-sortable iff it avoids the pattern 231, and that the count is C_n.
Solution
Necessity: suppose π contains indices i<j<k with π(k) < π(i) < π(j) (pattern 231). To output π(k) first it must be pushed and popped before π(i) is output; but π(i) is pushed before π(k) arrives, so it sits below π(k) in the stack and cannot be output later than it. Contradiction.
Sufficiency: if π avoids 231, the greedy algorithm (push, and pop whenever the stack top is the next needed value) succeeds — provable by induction on the position of the maximum.
Count: 231-avoiding permutations decompose at the position of n: everything before must be smaller than everything after (else a 231 appears), so |S_n(231)| = ∑_i |S_{i}(231)||S_{n-1-i}(231)| — Catalan. ∎
All six patterns of length 3 give the same count C_n (by symmetry and a non-trivial bijection for the 123/132 case), which is the starting point of the whole field of pattern-avoiding permutations.
- Theory
Derive C(x) = \frac{1-√{1-4x}}{2x} and use it to get the asymptotic C_n ~ 4^n/(n^{3/2}√π).
Solution
The convolution recurrence gives C(x) = 1 + xC(x)². Solving: xC² - C + 1 = 0, so C = \frac{1 ± √{1-4x}}{2x}. The + branch blows up as x\to0, so take the - branch, which tends to 1. ∎
Asymptotics: the singularity nearest the origin is the branch point at x = 1/4, so C_n ≈ 4^n times a subexponential correction. The √{1-4x} singularity type contributes n^{-3/2}/\Gamma(-1/2) = n^{-3/2}/(-2√π), giving
C_n ~ (4^n)/(n^{3/2)√π}.
Check at n=10: 4^{10}/(10^{1.5}√π) = 1048576/(31.6×1.772) = 18{,}718 against the true 16,796 — 11% high, converging as O(1/n). The singularity-analysis method is completely general: the location of the nearest singularity gives the exponential rate, its type gives the polynomial correction.
- Theory
Count the Dyck paths of semilength n with exactly k peaks (Narayana numbers) and verify they sum to C_n.
Solution
A peak is an occurrence of
(). The count isN(n,k) = (1)/(n)C(n,k)C(n,k-1).
Sketch: choose the k positions of the up-steps that begin peaks and the k-1 descents between them, then a cycle lemma argument divides by n.
Verification: ∑_{k=1}^{n}N(n,k) = C_n. For n=4: N = 1, 6, 6, 1 summing to 14 = C_4 ✓. The row is symmetric (N(n,k) = N(n,n+1-k)) because reversing and complementing a Dyck path exchanges peaks with valleys.
Narayana numbers refine Catalan the way binomials refine 2^n, and they appear as the h-vector of the associahedron — the polytope whose vertices are the triangulations of problem 4.
- Program
Compute C_n mod 10⁹+7 for n ≤ 10⁶, three ways, and check they agree.
Solution
MOD = 10**9+7 # (a) closed form, O(1) per query after factorial precompute def cat_closed(n): return fact[2*n] * ifact[n] % MOD * ifact[n+1] % MOD # (b) step recurrence C_{n+1} = C_n · 2(2n+1)/(n+2), O(n) total def cat_iter(N): c = [0]*(N+1); c[0] = 1 for n in range(N): c[n+1] = c[n] * 2 % MOD * (2*n+1) % MOD * pow(n+2, MOD-2, MOD) % MOD return c # (c) convolution, O(n²) — only for cross-checking small n def cat_conv(N): c = [0]*(N+1); c[0] = 1 for n in range(N): c[n+1] = sum(c[i]*c[n-i] for i in range(n+1)) % MOD return cMethod (a) is the practical choice: O(n) precompute, O(1) per query. Method (b) as written costs a modular inverse per step; batch them with the linear-inverse recurrence of Chapter 17 to make it genuinely O(n).
Cross-checking (c) against (a) for n ≤ 2000 is worthwhile — it validates the factorial tables and the n+1 versus n index in the closed form, which is the classic off-by-one here.
- Program
Generate all balanced bracket sequences of length 2n in lexicographic order, and rank/unrank them.
Solution
def gen(n): out = [] def rec(s, opened, closed): if len(s) == 2*n: out.append(''.join(s)); return if opened < n: s.append('('); rec(s, opened+1, closed); s.pop() if closed < opened: s.append(')'); rec(s, opened, closed+1); s.pop() rec([], 0, 0) return out def rank(s): # index among all balanced sequences n = len(s)//2; r = 0; bal = 0; rem = 2*n for i, ch in enumerate(s): rem -= 1 if ch == ')': # count the sequences that put '(' here instead r += ballot(rem, bal+1) bal -= 1 else: bal += 1 return r def ballot(steps, bal): # # of completions from height bal with `steps` left if bal < 0 or bal > steps or (steps - bal) % 2: return 0 k = (steps - bal)//2 return comb(steps, k) - comb(steps, k-1)Generation visits only valid prefixes, so it is O(C_n) outputs with O(1) amortised work each — the pruning conditions
opened < nandclosed < openedare exactly the Dyck path constraints.The ranking uses the ballot numbers to count completions from a given height, which is the same reflection formula as problem 1. This lets you pick a uniformly random balanced sequence by unranking a random index in [0, C_n) — far better than rejection sampling, which accepts only a Θ(n^{-3/2}) fraction.
- Program
Given a sequence, decide whether it is stack-sortable, and if so output the push/pop operations.
Solution
def stack_sort(a): st, out, ops = [], [], [] nxt = 1 # next value we must output i = 0 while nxt <= len(a): if st and st[-1] == nxt: out.append(st.pop()); ops.append('pop'); nxt += 1 elif i < len(a): st.append(a[i]); ops.append('push'); i += 1 else: return None # stuck: not stack-sortable return opsO(n) — each element is pushed and popped at most once. The greedy is optimal: whenever the stack top is the next needed value, popping it is forced (it will only get harder later), and otherwise pushing is the only legal move.
Failure occurs exactly when the input contains a 231 pattern (problem 5), so this routine doubles as a 231-pattern detector in linear time — much better than the O(n³) triple loop.
- Program
Count the ways to fully parenthesise a product of n matrices, then solve the matrix-chain DP and explain why the count is Catalan but the optimisation is not.
Solution
# count: C_{n-1} # optimise: classic O(n³) interval DP def matrix_chain(dims): # dims has n+1 entries n = len(dims) - 1 dp = [[0]*n for _ in range(n)] for length in range(2, n+1): for i in range(n-length+1): j = i + length - 1 dp[i][j] = min(dp[i][k] + dp[k+1][j] + dims[i]*dims[k+1]*dims[j+1] for k in range(i, j)) return dp[0][n-1]The number of parenthesisations is C_{n-1}, growing like 4^n/n^{1.5} — so brute force is hopeless past n ≈ 20. But the DP is O(n³) because the cost function has optimal substructure: the best way to compute a subchain does not depend on how it will be combined later.
That is the general lesson — an exponential count of structures does not imply an exponential optimisation, provided the objective decomposes along the same split the count does. Knuth's optimisation reduces this particular DP further to O(n²) using the monotonicity of the optimal split point.
- Program
Count the monotone lattice paths from (0,0) to (n,n) that stay weakly below the diagonal and avoid a given set of forbidden points.
Solution
# DP over the forbidden points, using the Catalan/ballot count between them def cat_paths(x1, y1, x2, y2): """paths (x1,y1)→(x2,y2), steps R/U, staying weakly below y = x""" if x2 < x1 or y2 < y1 or y2 > x2 or y1 > x1: return 0 a, b = x2-x1, y2-y1 return comb(a+b, b) - comb(a+b, b-1-(x1-y1)) # reflection, shifted def count(n, blocked): pts = sorted(blocked) + [(n, n)] f = [] for i, (x, y) in enumerate(pts): v = cat_paths(0, 0, x, y) for j, (px, py) in enumerate(pts[:i]): if px <= x and py <= y: v -= f[j] * cat_paths(px, py, x, y) f.append(v % MOD) return f[-1]Same first-blocked-point decomposition as Chapter 26 problem 12, but with the ballot count in place of the plain binomial. The reflection offset x_1 - y_1 accounts for starting at a point already below the diagonal.
O(m²) in the number of obstacles, independent of n — so n = 10⁹ with 20 obstacles is instant, where a grid DP would need 10^{18} cells.
- Program
Verify the Catalan bijections computationally: generate balanced strings, binary trees and triangulations for n ≤ 8 and check the counts match.
Solution
def trees(n): if n == 0: return [None] out = [] for i in range(n): for L in trees(i): for R in trees(n-1-i): out.append((L, R)) return out def triangulations(poly): # poly = tuple of vertex indices if len(poly) < 3: return [()] out = [] a, b = poly[0], poly[-1] for i in range(1, len(poly)-1): for L in triangulations(poly[:i+1]): for R in triangulations(poly[i:]): out.append(L + R + ((a, poly[i], b),)) return out for n in range(9): assert len(gen(n)) == len(trees(n)) == catalan(n) assert len(triangulations(tuple(range(n+2)))) == catalan(n)Three independent recursive enumerations landing on the same numbers is the most direct confirmation that the objects are in bijection. Adding memoisation makes each O(C_n) rather than exponentially wasteful.
Going further, you can build the bijections explicitly — tree to bracket string by preorder, bracket string to triangulation by the recursive split — and assert that composing them round-trips. That turns "the counts agree" into "here is the correspondence", which is what a bijective proof actually claims.
- Program
Compute the Motzkin numbers and the Schröder numbers, and compare their growth to Catalan.
Solution
# Motzkin: paths with U, D, F steps, never below 0, ending at 0 # M_n = M_{n-1} + Σ_{k} M_k M_{n-2-k} def motzkin(N): M = [0]*(N+1); M[0] = M[1] = 1 for n in range(2, N+1): M[n] = M[n-1] + sum(M[k]*M[n-2-k] for k in range(n-1)) return M # large Schröder: paths with U, D, and diagonal steps def schroder(N): S = [0]*(N+1); S[0] = 1 for n in range(1, N+1): S[n] = S[n-1] + sum(S[k]*S[n-1-k] for k in range(n)) return SGrowth rates: Catalan ~ 4^n n^{-3/2}, Motzkin ~ 3^n n^{-3/2}, Schröder ~ (3+2√2)^n n^{-3/2} ≈ 5.83^n. All three share the n^{-3/2} factor, because all three generating functions have a square-root branch point — the singularity type fixes the polynomial correction while its location fixes the exponential base.
That pattern is the payoff of the generating-function view: you can read off the asymptotics of a whole family of lattice-path counts without ever expanding a series.
Stirling numbers, Bell numbers and integer partitions
Three families that count set partitions, permutation cycles and integer partitions. They have no simple closed forms, which is exactly why the recurrences and their DP formulations are what you actually need.
Stirling numbers of the second kind
S(n,k) counts the partitions of an n-set into exactly k non-empty unlabelled blocks.
Stirling numbers of the first kind
c(n,k) = |s(n,k)| counts the permutations of n with exactly k cycles (the unsigned version).
The two kinds are inverse: ∑_k s(n,k)S(k,m) = [n=m]. They are the change-of-basis matrices between ordinary powers x^n and falling factorials x^{\underline n}, which is the cleanest way to remember which is which.
Bell numbers
Integer partitions
p(n) counts the ways to write n as an unordered sum of positive integers. Unlike compositions (which are 2^{n-1}), there is no closed form.
Compositions are ordered, partitions are not. Compositions of n into any number of parts: 2^{n-1}, a one-line stars-and-bars. Partitions: no closed form, and the fastest exact method is a recurrence with O(√n) terms. The whole difference is whether the parts are distinguishable — the same distinction that separates the rows of the twelvefold way.
"Count the ways to make change" is the partition generating function restricted to a coin set, and the DP is three lines: for coin: for amount: dp[amount] += dp[amount-coin]. The loop order matters and is the whole question — coin outer gives combinations (unordered), amount outer gives permutations (ordered). Being able to say which is which without testing is the signal.
- Theory
Prove the recurrence S(n,k) = kS(n-1,k) + S(n-1,k-1).
Solution
Condition on the block containing element n. Either it is alone in its own block — leaving a partition of the other n-1 elements into k-1 blocks, giving S(n-1,k-1) — or it joins one of the k blocks formed by the others, giving k · S(n-1,k). The cases are disjoint and exhaustive. ∎
Compare with the first-kind recurrence c(n,k) = (n-1)c(n-1,k) + c(n-1,k-1): element n either forms a singleton cycle, or is inserted into an existing cycle in one of n-1 positions. The structural difference — k blocks to join versus n-1 positions to insert — is exactly what separates set partitions from cycle structures.
- Theory
Prove S(n,2) = 2^{n-1}-1 and S(n,n-1) = C(n,2).
Solution
Two blocks: a partition into 2 labelled blocks is a choice of which elements go left — 2^n ways — minus the 2 that leave a block empty, then divided by 2 because the blocks are unlabelled: (2^n-2)/2 = 2^{n-1}-1. ∎
n-1 blocks: with n-1 blocks and n elements, exactly one block has two elements and the rest are singletons. So the partition is determined by which pair shares a block: C(n,2). ∎
These two, plus S(n,1)=S(n,n)=1, are the boundary values that anchor the DP table and are worth having as assertions in an implementation.
- Theory
Prove the identity x^n = ∑_k S(n,k)x^{\underline k}, where x^{\underline k} = x(x-1)···(x-k+1).
Solution
Both sides count functions from an n-set to an x-set, for every positive integer x. The left side is direct. On the right, classify a function by its image: if the image has exactly k elements, the function is determined by a partition of the domain into k blocks (S(n,k) ways) together with an injective assignment of blocks to codomain elements (x^{\underline k} ways).
The two sides agree for infinitely many x, and both are polynomials in x, so they are equal as polynomials. ∎
This makes S(n,k) the change-of-basis matrix from falling factorials to powers, and the first-kind numbers give the inverse — which is why ∑_k s(n,k)S(k,m) = [n=m]. It is also the reason Stirling numbers appear whenever you convert between E[X^n] and factorial moments in probability.
- Theory
Derive the Bell recurrence B_{n+1} = ∑_k C(n,k)B_k.
Solution
In a partition of {1,…,n+1}, look at the block containing n+1. If it contains k of the other n elements, choose them in C(n,k) ways, and partition the remaining n-k arbitrarily in B_{n-k} ways:
B_{n+1} = ∑_{k=0}^{n} C(n,k)B_{n-k} = ∑_{k=0}^{n}C(n,k)B_k (reindexing). ∎
This gives an O(n²) algorithm. The Bell triangle is an equivalent and simpler formulation: start a row with the last entry of the previous row, and each subsequent entry is the sum of the one to its left and the one above-left. The leftmost entries are the Bell numbers.
- Theory
How many ways are there to partition n distinct objects into k labelled non-empty groups? Into k unlabelled groups where some may be empty?
Solution
- Labelled, non-empty: k!S(n,k) — the surjection count.
- Unlabelled, possibly empty: ∑_{j=0}^{k}S(n,j) — sum over how many blocks are actually used.
- Labelled, possibly empty: k^n — every element chooses a group freely.
- Unlabelled, non-empty: S(n,k) by definition.
These are the four cells of the twelvefold way's "distinct balls" rows (Chapter 26), and having all four side by side is the fastest way to avoid the standard error of using S(n,k) where k!S(n,k) is meant. A quick check: for n=k=3 they give 6, 5, 27, 1 — all different, so the distinction is never cosmetic.
- Theory
Prove Euler's pentagonal number theorem statement for partitions: ∏_{k\ge1}(1-x^k) = ∑_{j=-∞}^{∞}(-1)^j x^{j(3j-1)/2}, and deduce the recurrence for p(n).
Solution
The product expands as ∑_{distinct-part partitions}(-1)^{#parts}x^{sum}. Franklin's involution pairs most partitions into distinct parts with one of opposite parity, cancelling them; the unpaired ones are exactly the staircase shapes with sum j(3j±1)/2, the generalised pentagonal numbers. ∎
Since (∑ p(n)x^n)∏(1-x^k) = 1, comparing coefficients of x^n for n ≥ 1 gives
p(n) = ∑_{j\ge1}(-1)^{j+1}[p(n-(j(3j-1))/(2))+p(n-(j(3j+1))/(2))].
The pentagonal numbers grow quadratically, so only O(√n) terms are non-zero — giving p(0..n) in O(n√n) instead of the O(n²) of the naive coin DP.
- Theory
Prove that the number of partitions of n into distinct parts equals the number into odd parts.
Solution
Generating functions: distinct parts give ∏_{k\ge1}(1+x^k); odd parts give ∏_{k odd}(1)/(1-x^k). Now
∏_k (1+x^k) = ∏_k \frac{1-x^{2k}}{1-x^k},
and the numerator cancels every even-index factor of the denominator, leaving ∏_{k odd}(1)/(1-x^k). ∎
Bijectively (Glaisher): given a partition into odd parts, merge equal parts repeatedly — write the multiplicity of each odd part in binary and split it into distinct powers of two times that part. The result has distinct parts, and the map reverses by factoring each distinct part as 2^a · odd.
This is the archetypal partition identity, and the two proofs illustrate the general choice: generating functions are mechanical, bijections are explanatory.
- Program
Compute the full Stirling tables of both kinds for n ≤ 5000 modulo 10⁹+7.
Solution
MOD = 10**9+7 def stirling2(N): S = [[0]*(N+1) for _ in range(N+1)] S[0][0] = 1 for n in range(1, N+1): for k in range(1, n+1): S[n][k] = (k*S[n-1][k] + S[n-1][k-1]) % MOD return S def stirling1(N): # unsigned c = [[0]*(N+1) for _ in range(N+1)] c[0][0] = 1 for n in range(1, N+1): for k in range(1, n+1): c[n][k] = ((n-1)*c[n-1][k] + c[n-1][k-1]) % MOD return cO(N²) time and memory — 25 million entries at N = 5000, about 200 MB as 32-bit ints. Keep only two rows if you need a single n.
Validate with the row sums: ∑_k c(n,k) ≡ n! and ∑_k S(n,k) ≡ B_n. For a single S(n,k) with huge n, use the explicit inclusion–exclusion formula instead: O(klog n) and no table at all.
- Program
Compute p(n) exactly for n ≤ 10⁵ using the pentagonal recurrence.
Solution
def partitions(N, mod=None): p = [0]*(N+1); p[0] = 1 for n in range(1, N+1): total = 0; k = 1 while True: g1 = k*(3*k-1)//2 if g1 > n: break sign = 1 if k % 2 else -1 total += sign * p[n-g1] g2 = k*(3*k+1)//2 if g2 <= n: total += sign * p[n-g2] k += 1 p[n] = total % mod if mod else total return pO(n√n) — about 3×10⁷ operations at n = 10⁵, a few seconds. Exact values grow enormous: p(10⁵) has 1,113 digits, so exact arithmetic dominates the cost unless you reduce modulo something.
Sanity checks: p(10) = 42, p(100) = 190{,}569{,}292, p(1000) = 24{,}061{,}467{,}864{,}032{,}622{,}473{,}692{,}149{,}727{,}991. Compare against the naive coin DP (O(n²)) for n ≤ 5000 to validate the sign pattern, which is where this implementation goes wrong if it goes wrong at all.
- Program
Solve the coin-change problem both ways and explain precisely why the loop order determines whether you count combinations or permutations.
Solution
# COMBINATIONS (unordered): coin loop OUTSIDE dp = [0]*(target+1); dp[0] = 1 for c in coins: for a in range(c, target+1): dp[a] += dp[a-c] # PERMUTATIONS (ordered): amount loop OUTSIDE dp = [0]*(target+1); dp[0] = 1 for a in range(1, target+1): for c in coins: if c <= a: dp[a] += dp[a-c]In the first, when coin c is processed,
dp[a]already accounts for every way using only coins earlier in the list plus any number of c. Each multiset is therefore built in exactly one canonical order — non-decreasing coin index — so it is counted once.In the second,
dp[a]is rebuilt from scratch considering every coin as the last one added, so 1+2 and 2+1 are counted separately.Test with coins {1,2} and target 3: the first gives 2 (1{+}1{+}1, 1{+}2), the second gives 3 (adding 2{+}1). That one test distinguishes them immediately and is worth keeping.
- Program
Count the ways to partition n into exactly k parts, and into parts of size at most m.
Solution
# p(n, k): partitions of n into exactly k parts # recurrence: either every part is ≥ 2 (subtract 1 from each), or some part is 1 def p_exact(N, K): dp = [[0]*(K+1) for _ in range(N+1)] dp[0][0] = 1 for n in range(1, N+1): for k in range(1, min(n, K)+1): dp[n][k] = dp[n-1][k-1] + dp[n-k][k] return dp # partitions into parts ≤ m: bounded coin DP def p_maxpart(N, m): dp = [0]*(N+1); dp[0] = 1 for c in range(1, m+1): for a in range(c, N+1): dp[a] += dp[a-c] return dpThe recurrence p(n,k) = p(n-1,k-1) + p(n-k,k) splits on whether the smallest part is 1 (remove it) or all parts exceed 1 (subtract 1 from every part). O(NK).
By conjugation (transposing the Young diagram), partitions of n into exactly k parts are in bijection with partitions of n into parts whose largest is exactly k — so the two functions above compute closely related quantities, and asserting ∑_k p(n,k) = p(n) validates both.
- Program
Generate all set partitions of an n-set in restricted growth string form, for n ≤ 12.
Solution
# A restricted growth string a[0..n-1] satisfies a[0]=0 and # a[i] <= 1 + max(a[0..i-1]). These are in bijection with set partitions. def rgs(n): a = [0]*n; out = [] def rec(i, m): if i == n: out.append(a[:]); return for v in range(m+2): a[i] = v rec(i+1, max(m, v)) rec(1, 0) return out assert len(rgs(n)) == bell(n)The restricted growth condition forces a canonical labelling — blocks are numbered by the order of their smallest element — so each partition appears exactly once and no deduplication is needed. That is the same "canonical form" idea as Chapter 5: an equivalence class is enumerated by choosing one representative by rule.
B_{12} = 4{,}213{,}597, so n=12 is about the practical limit for full enumeration; beyond that use the counting recurrences.
- Program
Compute a single S(n,k) mod p for n ≤ 10^{18} and small k.
Solution
MOD = 10**9+7 def stirling2_single(n, k): # S(n,k) = (1/k!) Σ_j (-1)^j C(k,j) (k-j)^n total = 0 for j in range(k+1): t = comb_mod(k, j) * pow(k-j, n, MOD) % MOD total = (total + (t if j % 2 == 0 else -t)) % MOD return total * pow(factorial_mod(k), MOD-2, MOD) % MODO(klog n) — independent of n except through the exponentiations, so n = 10^{18} is no harder than n = 100. Practical to k ≈ 10⁶.
Note that 0^n must be 0 for n ≥ 1 and 1 for n = 0 — Python's
pow(0, 0, m)returns 1, which is the convention this formula needs, but a hand-rolledpowerthat special-cases the exponent 0 must agree or S(0,0) comes out wrong. - Program
Verify Dobinski's formula numerically and compare its convergence to the exact Bell recurrence.
Solution
import math def bell_dobinski(n, terms=200): # B_n = (1/e) Σ_{j≥0} j^n / j! s = 0.0 for j in range(terms): s += math.exp(n*math.log(j) - math.lgamma(j+1)) if j else 0.0 return s / math.e def bell_exact(N): row = [1] for n in range(N): nxt = [row[-1]] for v in row: nxt.append(nxt[-1] + v) row = nxt return row[0]Dobinski's series has terms peaking near j ≈ n/ln n and decaying rapidly after, so a few hundred terms suffice for n ≤ 50. Beyond that the terms exceed the range of a
doubleand the log-space evaluation shown is essential.The exact Bell triangle is O(n²) and gives arbitrary precision; Dobinski gives 15 digits instantly for moderate n. Comparing them for n ≤ 30 shows agreement to full double precision, which is a satisfying check on a formula that looks like it should not converge at all — it sums an infinite series of positive terms to a specific integer.
Burnside's lemma and Polya enumeration
Counting objects "up to symmetry" is not division by the group size - some objects are fixed by non-trivial symmetries. Burnside's lemma is the correct version: average the number of fixed points over the group.
When configurations are considered identical under a group of symmetries (rotations of a necklace, rotations of a cube, colour swaps), the naive answer "total divided by |G|" is wrong. Division only works when the group acts freely — when no non-identity element fixes anything.
Proof: count the pairs (g, x) with g· x = x in two ways. By g it is ∑_g|Fix(g)|; by x it is ∑_x |Stab(x)|. The orbit–stabiliser theorem gives |Stab(x)| = |G|/|Orb(x)|, so the second sum is ∑_x |G|/|Orb(x)| = |G| · #orbits, because each orbit contributes |Orb| terms of 1/|Orb|. ∎
The standard example: necklaces
Count the necklaces of n beads in k colours, identified under rotation. The group is ℤ_n. A rotation by d decomposes the positions into gcd(d,n) cycles, and a colouring is fixed iff it is constant on each cycle — so |Fix(d)| = k^{gcd(d,n)}.
For n=4, k=2: \frac14(2^4 + 2^1 + 2^2 + 2^1) = (16+2+4+2)/(4) = 6. Naive division would give 16/4 = 4 — wrong, because the all-black, all-white and alternating necklaces are fixed by non-trivial rotations.
The whole computation reduces to: for each group element, how many cycles does it induce on the positions? If g has c(g) cycles then |Fix(g)| = k^{c(g)} for k-colourings with no further constraint. Finding the cycle counts is the entire problem; the averaging is arithmetic.
Pólya enumeration
Pólya's theorem refines Burnside to track how many beads of each colour are used. Define the cycle index
For the cyclic group: Z_{C_n} = \frac1n∑_{d|n}φ(d)x_d^{n/d}. For the dihedral group (rotations and reflections) add the reflection terms, which differ for odd and even n.
Cyclic C_n (1/n) Σ_{d|n} φ(d) · x_d^{n/d}
Dihedral D_n (1/2)·Z_{C_n} + reflections:
n odd : (1/2)· x₁ x₂^{(n−1)/2}
n even: (1/4)·( x₂^{n/2} + x₁² x₂^{(n−2)/2} )
Symmetric S_n (1/n!) Σ over cycle types, weight n!/∏(j^{cⱼ} cⱼ!)
Cube rotations (24 elements), acting on the 6 FACES:
identity ×1 x₁⁶
face turns ±90° ×6 x₁² x₄
face turns 180° ×3 x₁² x₂²
vertex turns ±120° ×8 x₃²
edge turns 180° ×6 x₂³"How many distinct dice can you make with 6 colours?" is a Burnside question on the cube's 24 rotations: (1)/(24)(6^6 + 6·6^3 + 3·6^4 + 8·6^2 + 6·6^3) = 2226. The insight to state is that the naive 6^6/24 is not even an integer — which immediately proves that division cannot be the right answer.
- Theory
Prove Burnside's lemma.
Solution
Count the set P = {(g,x) : g· x = x} two ways.
By g: |P| = ∑_{g}|Fix(g)|. By x: |P| = ∑_x |Stab(x)|.
The orbit–stabiliser theorem gives |Orb(x)|·|Stab(x)| = |G| (the cosets of the stabiliser correspond to orbit elements). So
∑_x|Stab(x)| = ∑_x (|G|)/(|Orb)(x)| = |G|∑_{orbits O}∑_{x∈ O}(1)/(|O|) = |G|·#orbits.
Equating gives the lemma. ∎ The key step is that each orbit contributes exactly 1 to the inner double sum, regardless of its size.
- Theory
Count the binary necklaces of length 6 up to rotation, by hand and by the formula.
Solution
\frac16∑_{d|6}φ(6/d)2^d = \frac16[φ(6)2^1 + φ(3)2^2 + φ(2)2^3 + φ(1)2^6] = \frac16[2·2 + 2·4 + 1·8 + 1·64] = (84)/(6) = 14.
By hand, grouped by the number of black beads: 0 black (1), 1 black (1), 2 black (3: adjacent, distance 2, opposite), 3 black (4), 4 black (3), 5 black (1), 6 black (1) — total 14 ✓.
Naive division gives 64/6 = 10.67, not even an integer — immediate proof that the orbits have different sizes.
- Theory
Count the binary bracelets of length 6 (rotations and reflections).
Solution
The group is D_6 with 12 elements. Rotations contribute the 84 from problem 2. Reflections for even n=6 come in two types:
- 3 reflections through opposite vertices: cycle type x_1^2x_2^2, so 2^4 = 16 fixed each — total 48.
- 3 reflections through opposite edges: cycle type x_2^3, so 2^3 = 8 fixed each — total 24.
(84 + 48 + 24)/(12) = (156)/(12) = 13.
One fewer than the 14 necklaces: exactly one necklace is chiral (distinct from its mirror image), namely the pattern with beads at positions {0,1,3}, and it merges with its reflection.
- Theory
How many ways to colour the faces of a cube with k colours, up to rotation?
Solution
Using the cube's face cycle index from the chapter:
(1)/(24)(k^6 + 6k^3 + 3k^4 + 8k^2 + 6k^3) = (k^6 + 3k^4 + 12k^3 + 8k^2)/(24).
- k=2: (64 + 48 + 96 + 32)/24 = 240/24 = 10.
- k=3: (729 + 243 + 324 + 72)/24 = 1368/24 = 57.
- k=6: (46656 + 3888 + 2592 + 288)/24 = 53424/24 = 2226.
That the polynomial always yields an integer is a non-trivial consistency check on the cycle index — if you miscount even one conjugacy class the divisibility fails, which makes it a good self-test.
- Theory
Use Pólya's theorem to count the necklaces of length 6 with exactly 3 black and 3 white beads, up to rotation.
Solution
Cycle index Z_{C_6} = \frac16(x_1^6 + x_2^3 + 2x_3^2 + 2x_6). Substitute x_j → b^j + w^j:
\frac16[(b+w)^6 + (b^2+w^2)^3 + 2(b^3+w^3)^2 + 2(b^6+w^6)].
Extract the coefficient of b^3w^3: from (b+w)^6 it is C(6,3)=20; from (b^2+w^2)^3 it is 0 (only even exponents); from 2(b^3+w^3)^2 it is 2·2 = 4; from 2(b^6+w^6) it is 0.
(20+0+4+0)/(6) = 4.
The four are: BBBWWW, BBWBWW, BBWWBW, BWBWBW. ✓ Pólya's advantage over plain Burnside is exactly this — one computation gives the count for every colour multiset at once.
- Theory
Derive the necklace-counting formula \frac1n∑_{d|n}φ(d)k^{n/d} and relate it to the number of aperiodic necklaces.
Solution
A rotation by j has gcd(j,n) cycles, so |Fix(j)| = k^{gcd(j,n)}. Grouping the j by their gcd: exactly φ(n/d) values of j have gcd(j,n)=d. Hence
N(n,k) = \frac1n∑_{d|n}φ(n/d)k^d = \frac1n∑_{d|n}φ(d)k^{n/d}. ∎
Aperiodic necklaces (Lyndon words) are those with trivial stabiliser. By Möbius inversion on ∑_{d|n} d· L(d) = k^n,
L(n) = \frac1n∑_{d|n}μ(d)k^{n/d}.
Each aperiodic necklace has an orbit of full size n, so n · L(n) counts the strings of period exactly n — and Lyndon words are the canonical representatives, generated in O(1) amortised time by Duval's algorithm.
- Theory
Count the distinct n× n binary matrices up to the 8 symmetries of the square, for n=3.
Solution
The group D_4 acts on the 9 cells. Cycle counts on cells:
- identity: 9 cycles → 2^9 = 512
- rotate ±90° (2 elements): centre fixed, 8 others in two 4-cycles → 3 cycles → 2^3 = 8 each
- rotate 180°: centre fixed, 4 pairs → 5 cycles → 2^5 = 32
- 2 axis reflections: 3 fixed cells, 3 pairs → 6 cycles → 2^6 = 64 each
- 2 diagonal reflections: 3 fixed cells, 3 pairs → 6 cycles → 2^6 = 64 each
(512 + 2(8) + 32 + 4(64))/(8) = (512+16+32+256)/(8) = (816)/(8) = 102.
The general method — write down each symmetry's cycle decomposition on the underlying cells, raise k to the cycle count, average — handles any board, any group, any number of colours.
- Program
Implement Burnside for necklaces and bracelets with n up to 10⁹ and k up to 10⁹, modulo 10⁹+7.
Solution
MOD = 10**9+7 def necklaces(n, k): total = 0 for d in divisors(n): total = (total + phi(n//d) * pow(k, d, MOD)) % MOD return total * pow(n, MOD-2, MOD) % MOD def bracelets(n, k): r = necklaces(n, k) * n % MOD # the rotation part, un-divided if n % 2: r = (r + n * pow(k, (n+1)//2, MOD)) % MOD else: r = (r + (n//2) * pow(k, n//2 + 1, MOD) + (n//2) * pow(k, n//2, MOD)) % MOD return r * pow(2*n, MOD-2, MOD) % MODCost is O(√n) to enumerate divisors plus O(log) per exponentiation, so n = 10⁹ is instant. Computing φ(n/d) needs the factorisation of n, obtained once.
Careful with the division by n: it is a modular inverse, which requires gcd(n, 10⁹+7)=1 — always true here since the modulus is prime and n < p, but it is exactly the step that breaks if the modulus is composite. Then you must keep the sum as an exact integer and divide before reducing.
- Program
Count the distinct colourings of a cube's faces, edges and vertices with k colours, by explicitly generating the rotation group.
Solution
from itertools import product def cube_rotations(): """generate the 24 rotations as permutations of the 8 vertices""" # vertices are (±1,±1,±1); a rotation is a signed permutation of axes # with determinant +1 verts = list(product([-1,1], repeat=3)) idx = {v:i for i,v in enumerate(verts)} out = [] for perm in permutations(range(3)): for signs in product([-1,1], repeat=3): det = sign_of(perm) * signs[0]*signs[1]*signs[2] if det != 1: continue p = [idx[tuple(signs[a]*v[perm[a]] for a in range(3))] for v in verts] out.append(p) return out # exactly 24 def count_colourings(perms, k): total = 0 for p in perms: total += k ** num_cycles(p) return total // len(perms)Generating the group explicitly and counting cycles is far more reliable than looking up a cycle index — it cannot be mis-transcribed, and the same code handles faces, edges or vertices just by changing what the rotations permute.
Results for k=2: 10 face colourings, 23 vertex colourings, 218 edge colourings. Assert the group has exactly 24 elements and that ∑ k^{c(g)} is divisible by 24 — both catch generation bugs immediately.
- Program
Count the distinct n× n grids up to the 8 symmetries of the square, for n ≤ 100 and k colours.
Solution
def grid_symmetry_count(n, k, MOD=10**9+7): def cyc(f): # cycles of a cell permutation given by f seen = [[False]*n for _ in range(n)]; c = 0 for i in range(n): for j in range(n): if seen[i][j]: continue c += 1 x, y = i, j while not seen[x][y]: seen[x][y] = True; x, y = f(x, y) return c maps = [lambda i,j: (i,j), lambda i,j: (j, n-1-i), lambda i,j: (n-1-i, n-1-j), lambda i,j: (n-1-j, i), lambda i,j: (i, n-1-j), lambda i,j: (n-1-i, j), lambda i,j: (j, i), lambda i,j: (n-1-j, n-1-i)] total = sum(pow(k, cyc(f), MOD) for f in maps) % MOD return total * pow(8, MOD-2, MOD) % MODO(8n²) — trivial at n=100. Computing cycle counts by traversal rather than by formula means the code is correct for both parities of n without special cases, which is where the closed forms get fiddly.
Cross-check n=3, k=2 against the hand computation in problem 7: it should print 102.
- Program
Implement Pólya enumeration for necklaces with a specified colour multiset, using polynomial multiplication.
Solution
# count necklaces of length n with exactly c_i beads of colour i from math import gcd, factorial def polya_multiset(n, counts): total = 0 for d in range(n): g = gcd(d, n) cyc_len = n // g # every cycle has this length # each colour count must be divisible by cyc_len if any(c % cyc_len for c in counts): continue per = [c // cyc_len for c in counts] # multinomial: distribute the g cycles among the colours m = factorial(g) for p in per: m //= factorial(p) total += m return total // nThe condition is that a rotation with cycles of length n/g can only fix a colouring whose every colour count is a multiple of that length — otherwise no assignment is constant on cycles. When it holds, the fixed colourings are the ways to assign colours to the g cycles, a multinomial.
Test: n=6 with counts (3,3) gives 4, matching problem 5 ✓. For many colours this is far cheaper than expanding the full cycle-index polynomial.
- Program
Generate all Lyndon words of length up to n over an alphabet of size k using Duval's algorithm, and verify the count against the Möbius formula.
Solution
def lyndon_words(n, k): """all Lyndon words of length ≤ n over {0..k-1}, in lexicographic order""" out = [] w = [-1] while w: w[-1] += 1 m = len(w) if m == n or True: out.append(tuple(w)) if m <= n else None while len(w) < n: # extend periodically w.append(w[len(w) - m]) while w and w[-1] == k - 1: # backtrack w.pop() return [x for x in out if len(x) <= n] def lyndon_count(n, k): return sum(mu(n//d) * k**d for d in divisors(n)) // nDuval's algorithm generates each Lyndon word in O(1) amortised time, in lexicographic order, using O(n) space. It is the standard way to enumerate necklace representatives without generating and deduplicating all k^n strings.
Verify |{Lyndon words of length n}| = \frac1n∑_{d|n}μ(n/d)k^d for all n ≤ 15, k ≤ 4. The concatenation of all Lyndon words of length dividing n, in lexicographic order, is also the lexicographically smallest de Bruijn sequence — a second, satisfying check.
- Program
Count the non-isomorphic graphs on n labelled-then-unlabelled vertices for n ≤ 8 using Burnside over S_n acting on edges.
Solution
from itertools import permutations from math import gcd def unlabelled_graphs(n): total = 0 for p in permutations(range(n)): # count cycles of p acting on the C(n,2) edges seen = set(); c = 0 for i in range(n): for j in range(i+1, n): if (i,j) in seen: continue c += 1 a, b = i, j while (min(a,b), max(a,b)) not in seen: seen.add((min(a,b), max(a,b))) a, b = p[a], p[b] total += 2 ** c return total // factorial(n)Results: 1, 2, 4, 11, 34, 156, 1044, 12346 for n = 1..8. The O(n!n²) cost is fine to n = 10; beyond that, group by cycle type rather than enumerating all n! permutations, since the edge-cycle count depends only on the type — that gives O(p(n)) terms and reaches n = 50.
This is the classical Pólya application. Note that counting unlabelled graphs is easy while deciding isomorphism of two given graphs is famously hard — a striking gap between enumeration and recognition.
- Program
Write a general Burnside solver: given a list of permutations of the ground set and a number of colours, return the orbit count. Test it on several groups.
Solution
def burnside(perms, k): """perms: list of permutations (as lists) of the same ground set""" n = len(perms[0]) assert all(sorted(p) == list(range(n)) for p in perms), "not permutations" total = 0 for p in perms: seen = [False]*n; c = 0 for i in range(n): if seen[i]: continue c += 1; j = i while not seen[j]: seen[j] = True; j = p[j] total += k ** c assert total % len(perms) == 0, "not a group, or perms are missing" return total // len(perms)The two assertions do real work. The first catches malformed input; the second catches the commonest error — supplying a set of symmetries that is not closed under composition. Burnside requires a genuine group, and if the identity is missing or a product is absent, the sum will not divide evenly.
Test cases: the cyclic group on n points (should match the necklace formula), the trivial group (should return k^n), and the full symmetric group on n points (should return C(n+k-1, n), the multiset count — a pleasing check, since "colourings up to all permutations" is exactly "multisets of colours").
Generating functions
Encode a sequence as the coefficients of a formal power series, and combinatorial operations become algebra: choice is addition, pairing is multiplication, sequences are 1/(1-x). Solve the algebra, read off the coefficients.
The ordinary generating function (OGF) of a sequence is A(x) = ∑_{n\ge0}a_n x^n. It is a formal series — convergence is irrelevant, x is a bookkeeping device, and the only operations needed are addition, multiplication and (formal) inversion.
The dictionary
This is the whole method: translate the combinatorial construction into an algebraic one.
| Combinatorial operation | OGF operation |
|---|---|
| disjoint union (choose A or B) | A(x) + B(x) |
| pair (one A and one B) | A(x)· B(x) |
| sequence of A's (a_0 = 0) | 1/(1-A(x)) |
| shift: a_{n-1} | xA(x) |
| partial sums ∑_{k≤ n}a_k | A(x)/(1-x) |
| weight by n: na_n | xA'(x) |
| convolution ∑ a_kb_{n-k} | A(x)B(x) |
The sequence rule is the workhorse: 1 + A + A² + A³ + ··· = (1)/(1-A) encodes "an ordered list of any number of A-objects", and it produces most of the classic results in one line.
Worked example: solving a recurrence
F_0 = 0, F_1 = 1, F_n = F_{n−1} + F_{n−2}
Multiply by xⁿ and sum over n ≥ 2:
Σ F_n xⁿ = x Σ F_{n−1}x^{n−1} + x² Σ F_{n−2}x^{n−2}
F(x) − x = x·F(x) + x²·F(x)
F(x) = x / (1 − x − x²)
Partial fractions with roots 1/φ and 1/ψ (φ,ψ = (1±√5)/2):
F(x) = (1/√5)·[ 1/(1−φx) − 1/(1−ψx) ]
⟹ F_n = (φⁿ − ψⁿ)/√5 — Binet's formula, derived not guessedAny linear recurrence with constant coefficients has a rational generating function P(x)/Q(x), and conversely. Partial fractions then give a closed form as a sum of n^j r^n terms, one per root of Q with multiplicity. That equivalence — rational GF ⟺ linear recurrence — is the single most useful structural fact in this chapter.
Exponential generating functions
For labelled structures use \hat{A}(x) = ∑ a_n x^n/n!. Multiplication then corresponds to splitting a labelled set into two parts:
"Count the ways to make change with unlimited coins" is ∏_i (1)/(1-x^{c_i)}, and the DP everyone writes is literally the polynomial multiplication of those factors done in place. Saying that connects the code to the mathematics, and it immediately explains why the coin loop must be outside — each factor is multiplied in exactly once.
- Theory
Find the generating function for a_n = 3a_{n-1} - 2a_{n-2} with a_0=1, a_1=3, and solve.
Solution
Multiply by x^n and sum from n=2:
A(x) - 1 - 3x = 3x(A(x)-1) - 2x²A(x), so A(x)(1-3x+2x²) = 1, giving A(x) = (1)/((1-x)(1-2x)).
Partial fractions: (1)/((1-x)(1-2x)) = (2)/(1-2x) - (1)/(1-x), so a_n = 2^{n+1} - 1.
Check: a_0 = 1 ✓, a_1 = 3 ✓, a_2 = 3·3-2·1 = 7 = 2³-1 ✓.
- Theory
Use generating functions to prove ∑_k C(m,k)C(n,r-k) = C(m+n,r).
Solution
The OGF of (C(m,k))_k is (1+x)^m and of (C(n,k))_k is (1+x)^n. Their product is (1+x)^{m+n}, whose x^r coefficient is C(m+n,r).
But the product of two OGFs has coefficients given by convolution, so the x^r coefficient is also ∑_k C(m,k)C(n,r-k). ∎
One line, versus the double-counting argument of Chapter 28 which needs a paragraph. That is the trade: generating functions are mechanical and always available; bijections explain why and generalise to refinements.
- Theory
Derive the generating function for the number of ways to make change with coins of denominations c_1,…,c_k.
Solution
For each coin denomination c, the choices are "use it 0, 1, 2, … times", contributing 1 + x^c + x^{2c} + ··· = (1)/(1-x^c).
The choices for different coins are independent and their amounts add, so multiply:
∏_{i=1}^{k}(1)/(1-x^{c_i)},
and the number of ways to make n is the coefficient of x^n. ∎
With c_i = i for all i ≥ 1 this becomes ∏_{k\ge1}(1)/(1-x^k), the partition generating function of Chapter 33. With a bounded supply of m_i coins of type i, the factor becomes \frac{1-x^{c_i(m_i+1)}}{1-x^{c_i}} — a finite geometric sum, which is exactly the bounded-knapsack DP.
- Theory
Show that (1)/((1-x)^k) = ∑_n C(n+k-1,k-1)x^n and interpret it.
Solution
By induction on k, or directly: (1)/((1-x)^k) is the product of k copies of (1)/(1-x) = ∑ x^n. The coefficient of x^n in that product counts the ways to write n = e_1+···+e_k with e_i ≥ 0 — which is C(n+k-1,k-1) by stars and bars. ∎
So the algebraic identity is stars and bars, and each of the two proofs of the binomial expansion (Newton's generalised binomial theorem, or the combinatorial count) proves the other.
This is the single most reused expansion in the chapter: whenever a generating function has a factor (1-x)^{-k}, that factor contributes a polynomial of degree k-1 in n to the coefficient formula.
- Theory
Find the EGF for derangements and extract D_n.
Solution
A permutation is a set of cycles; the EGF for a single cycle of length ≥ 1 is ∑_{k\ge1}((k-1)!x^k)/(k!) = ∑_{k\ge1}(x^k)/(k) = -ln(1-x). So permutations have EGF e^{-ln(1-x)} = (1)/(1-x) ✓ (since n!/n! = 1).
Derangements forbid 1-cycles, so the cycle EGF loses its x term: -ln(1-x) - x. Hence
\hat{D}(x) = e^{-ln(1-x)-x} = \frac{e^{-x}}{1-x}.
Expanding, \frac{e^{-x}}{1-x} = (∑_k((-1)^kx^k)/(k!))(∑_j x^j), so the coefficient of x^n is ∑_{k≤ n}((-1)^k)/(k!), and D_n = n!∑_{k≤ n}((-1)^k)/(k!) ✓ — Chapter 31's formula, derived rather than assumed.
- Theory
Prove that a sequence satisfies a linear recurrence with constant coefficients iff its OGF is rational.
Solution
(⟹) If a_n = ∑_{i=1}^{k}c_ia_{n-i} for n ≥ k, multiply by x^n and sum. Every term becomes c_ix^iA(x) plus finitely many correction terms from the initial conditions, giving A(x)(1 - ∑ c_ix^i) = P(x) with \deg P < k. So A = P/Q with Q(x) = 1-∑ c_ix^i.
(⟸) If A = P/Q with Q(0) ≠ 0, then AQ = P, and comparing coefficients of x^n for n > \deg P gives exactly the recurrence. ∎
Consequences: the denominator's roots determine the growth (a root of modulus r contributes r^{-n}), and a root of multiplicity m contributes a polynomial factor of degree m-1. This is why 1/(1-x)² gives n+1 rather than a constant.
- Theory
Use the "sequence" construction to count the binary strings with no two consecutive 1s.
Solution
Such a string is a sequence of blocks, each being
0or10, possibly followed by a trailing1. The block EGF — here an OGF, tracking length — is x + x², so sequences of blocks give (1)/(1-x-x²), and the optional trailing 1 multiplies by (1+x):A(x) = (1+x)/(1-x-x²).
The denominator is the Fibonacci denominator, so a_n = F_{n+2} ✓ — matching Chapter 26 problem 7. The (1+x) numerator is exactly what shifts the index by two.
This decomposition-into-blocks technique is completely general for "strings avoiding a pattern": describe the language as an unambiguous regular expression, translate | to +, concatenation to ×, and Kleene star to 1/(1-·). The result is the generating function, and it is rational exactly because the language is regular.
- Program
Implement formal power series arithmetic — add, multiply, invert, and compose — truncated to n terms.
Solution
MOD = 10**9+7 def mul(a, b, n): r = [0]*n for i, x in enumerate(a[:n]): if x: for j, y in enumerate(b[:n-i]): r[i+j] = (r[i+j] + x*y) % MOD return r def inv(a, n): """requires a[0] != 0; Newton iteration doubles the precision each step""" assert a[0] % MOD != 0 r = [pow(a[0], MOD-2, MOD)] k = 1 while k < n: k *= 2 # r ← r·(2 − a·r) mod x^k t = mul(a[:k], r, k) t = [(-v) % MOD for v in t]; t[0] = (t[0] + 2) % MOD r = mul(r, t, k) return r[:n]Naive multiplication is O(n²); with NTT (Chapter 51) it is O(nlog n), and Newton's iteration then makes inversion, \exp, log and square roots all O(nlog n) too, since each doubling step costs the same as one multiplication at the final size.
The requirement a_0 ≠ 0 for inversion is not a technicality: 1/x is not a power series. When you need to invert something vanishing at 0, factor out the x^k first and handle it as a shift.
- Program
Given a linear recurrence, find its generating function's numerator and denominator automatically, then compute a_n for huge n.
Solution
def gf_from_recurrence(c, init): """a_n = Σ c[i]·a_{n-1-i}; returns (P, Q) with A(x) = P(x)/Q(x)""" k = len(c) Q = [1] + [(-ci) % MOD for ci in c] # 1 − c1 x − c2 x² − ... P = mul(init, Q, k) # truncate to degree < k return P[:k], Q def nth_term(P, Q, n): """Bostan–Mori: O(k log k log n) with NTT, O(k² log n) naively""" while n: Qm = [(v if i % 2 == 0 else (-v) % MOD) for i, v in enumerate(Q)] U = mul(P, Qm, len(P)+len(Qm)) V = mul(Q, Qm, 2*len(Q)) P = U[n % 2::2] Q = V[0::2] n //= 2 return P[0] * pow(Q[0], MOD-2, MOD) % MODBostan–Mori is the modern replacement for matrix exponentiation: multiply numerator and denominator by Q(-x) so the denominator becomes even, then halve n by taking every other coefficient. Each step is two polynomial multiplications, giving O(klog klog n) against Kitamasa's O(klog k log n) with a better constant and matrix power's O(k³log n).
At k = 10⁴ and n = 10^{18} this is the only feasible method.
- Program
Count the ways to make change for n ≤ 10⁵ with a given coin set, and separately count the partitions into parts from a set.
Solution
def change_ways(n, coins, MOD=10**9+7): dp = [0]*(n+1); dp[0] = 1 for c in coins: # ONE factor 1/(1−x^c) at a time for a in range(c, n+1): dp[a] = (dp[a] + dp[a-c]) % MOD return dp[n]This is literally the polynomial product ∏ (1)/(1-x^{c_i)} evaluated in place: the inner loop multiplies the running series by one factor of (1)/(1-x^c), since B(x) = A(x)/(1-x^c) means b_a = a_a + b_{a-c}.
O(n·|coins|). Seeing the DP as a series product explains every variant at once: bounded supply multiplies by \frac{1-x^{c(m+1)}}{1-x^c} (a subtraction term appears), ordered compositions swap the loops, and "at most k coins total" adds a second dimension.
- Program
Use generating functions to count the number of ways to roll a sum of s with n dice of k faces.
Solution
# GF for one die: x + x² + ... + x^k = x(1−x^k)/(1−x) # For n dice: xⁿ(1−x^k)ⁿ/(1−x)ⁿ from math import comb def dice_ways(n, k, s): total = 0 for j in range((s - n)//k + 1): total += (-1)**j * comb(n, j) * comb(s - j*k - 1, n - 1) return totalExpanding (1-x^k)^n by the binomial theorem and (1-x)^{-n} by problem 4 gives the closed form directly — O(s/k) terms, versus O(ns) for the DP.
This is the same inclusion–exclusion as Chapter 30's bounded stars-and-bars, and it is a good illustration that generating functions and inclusion–exclusion are two views of one computation: the alternating signs come from expanding (1-x^k)^n.
Check: n=2, k=6, s=7 gives C(6,1) = 6 ✓.
- Program
Implement EGF operations (binomial convolution, \exp of a series) and use them to compute Bell numbers.
Solution
# Bell numbers: EGF = e^{e^x − 1} def bell_via_egf(N, MOD=10**9+7): fact = [1]*(N+1) for i in range(1, N+1): fact[i] = fact[i-1]*i % MOD ifact = [pow(fact[N], MOD-2, MOD)]*(N+1) for i in range(N, 0, -1): ifact[i-1] = ifact[i]*i % MOD # A(x) = e^x − 1 (no constant term, so exp is well defined) A = [0] + ifact[1:N+1] E = series_exp(A, N+1) # Newton iteration on exp return [E[n]*fact[n] % MOD for n in range(N+1)]O(Nlog N) with NTT-based \exp, versus O(N²) for the Bell triangle. Worth it above N ≈ 10⁴.
series_exprequires A(0)=0 — hence subtracting the 1 from e^x. It is implemented by Newton's iteration on log(E) - A = 0, which needsseries_log, which needsinv— the whole formal-power-series toolkit builds on the inversion of problem 8. - Program
Given a regular expression, compute the generating function counting the accepted strings by length.
Solution
# Build the DFA, then solve the linear system A_q(x) = [q accepting] + # x·Σ_{q→q'} A_{q'}(x) over the field of rational functions. def language_gf(dfa, start, accepting, alphabet, n): """returns the first n coefficients — the count of accepted strings""" # simple DP over (length, state) — equivalent to the linear system cur = {start: 1} out = [] for length in range(n): out.append(sum(v for q, v in cur.items() if q in accepting)) nxt = {} for q, v in cur.items(): for ch in alphabet: r = dfa[q][ch] nxt[r] = nxt.get(r, 0) + v cur = nxt return outThe DP is the generating function's coefficients computed one at a time; the closed form comes from solving the linear system by Cramer's rule, giving a rational function whose denominator is \det(I - xM) for the transition matrix M.
Consequence worth stating: every regular language has a rational generating function, hence its counting sequence satisfies a linear recurrence with constant coefficients — which is why "count strings of length n avoiding a pattern" is always solvable by matrix exponentiation in O(|Q|³log n).
- Program
Verify Binet's formula symbolically by comparing the power-series expansion of x/(1-x-x²) with the Fibonacci numbers.
Solution
from fractions import Fraction def series_div(P, Q, n): """coefficients of P/Q up to x^{n-1}, exact rationals""" r = [Fraction(0)]*n for i in range(n): num = (P[i] if i < len(P) else 0) - sum(Q[j]*r[i-j] for j in range(1, min(i, len(Q)-1)+1)) r[i] = Fraction(num, Q[0]) return r coeffs = series_div([0, 1], [1, -1, -1], 60) fib = [0, 1] for i in range(2, 60): fib.append(fib[-1] + fib[-2]) assert [int(c) for c in coeffs] == fibThe long division confirms x/(1-x-x²) is the Fibonacci OGF exactly, with rational arithmetic so there is no floating-point doubt.
Contrast with evaluating Binet's (φ^n - ψ^n)/√5 in floating point, which diverges from the true value around n = 71 (Chapter 3 problem 13). The generating function is exact where the closed form is not — a good reminder that a "closed form" is not automatically the better computational route.
Probability spaces, conditioning and Bayes
Conditioning is the whole subject. Most probability errors - the base-rate fallacy, the prosecutor's fallacy, the Monty Hall confusion - are the same error: conditioning on the wrong event, or forgetting to condition at all.
A probability space is a sample space Ω, a set of events, and a measure P with P(Ω)=1 and countable additivity on disjoint events. For finite discrete problems that reduces to: list the outcomes, assign weights summing to 1, and an event's probability is the sum of its outcomes' weights.
The single most important discipline is writing down Ω explicitly. Most paradoxes dissolve the moment the sample space is written correctly, and most errors come from an Ω that was never made precise.
Bayes and the base rate
A test is 99% accurate for a disease affecting 1 in 10,000. You test positive. The probability you have it is not 99% — it is about 1%.
Imagine 1,000,000 people.
100 have the disease → 99 test positive (99% sensitivity)
999,900 do not → 9,999 test positive (1% false positive)
Total positives = 99 + 9,999 = 10,098
P(disease | positive) = 99 / 10,098 ≈ 0.98%
The false positives from the huge healthy population swamp the true
positives from the tiny sick one. This is the BASE RATE FALLACY.When a Bayes computation feels slippery, convert to natural frequencies: pick a large population, propagate counts through the tree, and read the answer as a ratio of counts. It is algebraically identical and empirically far more reliable — studies find error rates drop from ~85% to ~20% with this reframing.
Two named fallacies
Prosecutor's fallacy: confusing P(evidence | innocent) with P(innocent | evidence). A DNA match with a 1-in-a-million false-positive rate in a city of 10 million yields about 10 innocent matches; the match alone gives roughly 1/11 probability of guilt, not 999999/1000000.
Simpson's paradox: an association can reverse when data is aggregated. A treatment better in every subgroup can appear worse overall if the subgroups have different sizes and different base rates. The fix is not statistical but causal — you must decide which variables to condition on, and that decision is not in the data.
Conditional independence
A and B are conditionally independent given C when P(A∩ B| C) = P(A| C)P(B| C). This neither implies nor is implied by ordinary independence, and the distinction is the whole content of naive Bayes classifiers, Markov chains, and Bayesian networks: they are precisely assertions about which conditional independences hold.
"A family has two children, at least one is a boy — probability both are boys?" The answer depends entirely on Ω and how the information arrived. If you learned it by asking "is at least one a boy?", the sample space is {BB, BG, GB} and the answer is 1/3. If you met one child at random and it was a boy, it is 1/2. Stating that the answer depends on the sampling procedure — and why — is the correct response.
- Theory
Solve Monty Hall rigorously with an explicit sample space, and state which assumption drives the answer.
Solution
Sample space: the car's location (uniform over 3 doors) and the host's choice. You pick door 1.
- Car behind 1 (prob 1/3): host opens 2 or 3, each with prob 1/2. Switching loses.
- Car behind 2 (prob 1/3): host must open 3. Switching wins.
- Car behind 3 (prob 1/3): host must open 2. Switching wins.
P(win | switch) = 2/3.
The critical assumption: the host knows where the car is and always opens a goat door. If the host opens a door at random and it happens to show a goat, then conditioning on that event gives 1/2 — switching is irrelevant. The puzzle's counter-intuitiveness lives entirely in this assumption, and stating it is the real answer.
- Theory
Compute the disease-test posterior for prevalences of 1/100, 1/1000 and 1/10⁶, with 99% sensitivity and specificity.
Solution
P(D|+) = (0.99p)/(0.99p + 0.01(1-p)).
p = 1/100 → 0.99·0.01 / (0.0099 + 0.0099) = 50.0% p = 1/1000 → 0.00099 / (0.00099 + 0.00999) = 9.0% p = 1/10⁶ → 9.9e-7 / (9.9e-7 + 0.01) = 0.0099%The posterior is essentially proportional to the prevalence when the prevalence is small. A "99% accurate" test for a rare condition is a machine for generating false positives, which is why screening programmes for rare diseases require confirmatory testing and why the same arithmetic governs alert fatigue in monitoring systems.
- Theory
Give three events that are pairwise independent but not mutually independent.
Solution
Toss two fair coins. Let A = "first is heads", B = "second is heads", C = "the two match".
Each has probability 1/2. Each pairwise intersection has probability 1/4, so all three pairs are independent. But A∩ B∩ C = A ∩ B has probability 1/4, while P(A)P(B)P(C) = 1/8. ∎
Indeed C is determined by A and B — maximally dependent on the pair, yet independent of each individually. This is why "pairwise independent" hash families are weaker than fully independent ones, and why the distinction matters for the concentration bounds of Chapter 38: Chebyshev needs only pairwise independence, Chernoff needs full independence.
- Theory
Two envelopes contain x and 2x. You open one and find A. "Switching gives \frac12(2A) + \frac12(A/2) = 1.25A > A, so always switch." Find the error.
Solution
The error is treating P(other = 2A | A) = 1/2 as if it held for every A. That requires a prior over x for which the posterior is always 1/2 either way — and no proper prior does this, because it would need P(x = A) = P(x = A/2) for all A, forcing a uniform distribution over an unbounded range, which cannot be normalised.
With any genuine prior, large observed values make "I hold the larger envelope" more likely, and the expected gain from switching becomes negative for large A — exactly cancelling the gain for small A. Overall E[gain] = 0 as symmetry demands.
The general lesson: a conditional probability computed without a prior is not a conditional probability. Improper priors produce contradictions of exactly this shape.
- Theory
State Bayes' theorem in odds form and use it to combine two independent pieces of evidence.
Solution
Odds form: posterior odds = prior odds × likelihood ratio,
(P(H| E))/(P(lnot H| E)) = (P(H))/(P(lnot H))·(P(E| H))/(P(E\midlnot H)).
The normalising constant cancels, which is why this form is so much easier to use. For evidence E_1, E_2 conditionally independent given H (and given lnot H), the likelihood ratios simply multiply:
posterior odds = prior odds× LR_1 × LR_2.
Taking logarithms turns this into addition — which is exactly what a naive Bayes classifier computes, and why its scores are sums of per-feature log-likelihood ratios. The conditional-independence assumption is usually false and the classifier usually works anyway, because the ranking survives even when the calibration does not.
- Theory
Construct a Simpson's paradox example with concrete numbers and explain which analysis is correct.
Solution
Treatment A Treatment B Small stones 81/87 = 93% 234/270 = 87% Large stones 192/263 = 73% 55/80 = 69% ───────────────────────────────────────────────── Overall 273/350 = 78% 289/350 = 83%A is better in both subgroups, worse overall. The cause: A was given mostly to large-stone cases (hard) and B mostly to small-stone cases (easy). Stone size is a confounder — it affects both treatment assignment and outcome.
Here the subgroup analysis is correct, because stone size is a common cause. But this is not a statistical rule: if the third variable were a consequence of treatment (a mediator) rather than a cause, conditioning on it would introduce bias instead of removing it.
Which analysis is right cannot be determined from the numbers alone — it requires a causal model. That is the real content of the paradox, and it is why A/B tests randomise assignment: randomisation guarantees no confounders exist.
- Theory
A random bit string of length n is generated. Given that it contains at least one 1, what is the probability the first bit is 1?
Solution
P(first = 1 | at least one 1) = (P(first=1))/(P(at least one 1)) = (1/2)/(1 - 2^{-n)} = \frac{2^{n-1}}{2^n - 1}.
The numerator simplifies because "first bit is 1" already implies "at least one 1". For n=1 this is 1 ✓; for large n it tends to 1/2 from above.
The instructive part is that it is slightly more than 1/2: conditioning on "not all zeros" removes one outcome, and that outcome had a 0 in the first position. Small conditioning effects like this are the source of many off-by-one probability errors, and writing the ratio explicitly is what catches them.
- Program
Write a Monte Carlo simulator for Monty Hall, including the variant with a host who opens a door at random.
Solution
import random def monty(switch, knowing_host=True, trials=10**6): wins = 0 for _ in range(trials): car = random.randrange(3) pick = random.randrange(3) if knowing_host: opts = [d for d in range(3) if d != pick and d != car] opened = random.choice(opts) else: opts = [d for d in range(3) if d != pick] opened = random.choice(opts) if opened == car: continue # invalid run: discard final = pick if not switch else next(d for d in range(3) if d != pick and d != opened) wins += (final == car) return wins / trials print(monty(True)) # ≈ 0.667 print(monty(False)) # ≈ 0.333 print(monty(True, False)) # ≈ 0.5 (of the non-discarded runs)The
continuein the ignorant-host branch is the crux: those runs are discarded, which is precisely what conditioning on "a goat was revealed" means. Note the denominator must then be the number of surviving runs, nottrials— the code above understates both variants, and fixing that is exactly the conditioning subtlety the problem is about.This is a good demonstration that a simulation encodes an assumption whether or not you notice: the two host models differ by three lines and give different answers.
- Program
Implement a Bayesian spam filter using log-odds, and show how a single unseen word can dominate the score.
Solution
import math def train(docs): from collections import Counter cnt = {True: Counter(), False: Counter()} tot = {True: 0, False: 0} for text, is_spam in docs: for w in set(text.split()): cnt[is_spam][w] += 1; tot[is_spam] += 1 return cnt, tot def score(text, cnt, tot, prior_odds, vocab, alpha=1.0): lo = math.log(prior_odds) V = len(vocab) for w in set(text.split()): # Laplace smoothing — WITHOUT it an unseen word gives log(0) = −∞ p_spam = (cnt[True][w] + alpha) / (tot[True] + alpha*V) p_ham = (cnt[False][w] + alpha) / (tot[False] + alpha*V) lo += math.log(p_spam / p_ham) return loWithout smoothing, a word appearing only in spam gives P(w\midham) = 0, a likelihood ratio of ∞, and one word decides the classification regardless of all other evidence. Laplace smoothing bounds each term's contribution and is not optional.
The conditional-independence assumption is plainly false for natural language ("New" and "York" are not independent), which systematically pushes scores toward the extremes — so naive Bayes is well-ranked but badly calibrated, and its probabilities should not be reported as probabilities.
- Program
Simulate the birthday problem and compare to the exact formula and the e^{-k²/2n} approximation.
Solution
import random, math def exact(k, n=365): p = 1.0 for i in range(k): p *= (n - i) / n return 1 - p def approx(k, n=365): return 1 - math.exp(-k*(k-1)/(2*n)) def simulate(k, trials=200000, n=365): hits = 0 for _ in range(trials): s = set() for _ in range(k): b = random.randrange(n) if b in s: hits += 1; break s.add(b) return hits / trials for k in (10, 23, 40, 60): print(k, round(exact(k),4), round(approx(k),4), round(simulate(k),4))At k=23 all three give ≈ 0.507 — the classic result that 23 people suffice for a better-than-even chance. The approximation comes from ∏(1-i/n) ≈ e^{-∑ i/n} and is accurate to about 1% in this range.
The general threshold k ≈ 1.177√n is the same square-root law as hash collisions (Chapter 6), and running the simulation for n = 2^{32} is how you convince someone that a 32-bit ID space is inadequate.
- Program
Build a small Bayesian network and compute exact posteriors by enumeration.
Solution
from itertools import product def enumerate_ask(query, evidence, nodes, parents, cpt): """nodes in topological order; cpt[node][parent_values] = P(node=True)""" hidden = [n for n in nodes if n != query and n not in evidence] dist = {} for qv in (True, False): total = 0.0 for hv in product([True, False], repeat=len(hidden)): assign = dict(evidence); assign[query] = qv assign.update(zip(hidden, hv)) p = 1.0 for n in nodes: pv = tuple(assign[par] for par in parents[n]) pt = cpt[n][pv] p *= pt if assign[n] else (1 - pt) total += p dist[qv] = total z = dist[True] + dist[False] return {k: v/z for k, v in dist.items()}O(2^{|hidden|}) — exact but exponential. The whole point of a Bayesian network is that the joint factorises as ∏_n P(n | parents(n)), encoding conditional independences that make the representation compact even when exact inference is not.
For larger networks use variable elimination (polynomial in the treewidth) or sampling. Exact inference in general Bayesian networks is #P-hard, which is the same hardness as the permanent in Chapter 30 — and for the same reason: both are sums over exponentially many configurations with no cancellation to exploit.
- Program
Demonstrate the prosecutor's fallacy numerically: given a DNA match probability and a population size, compute the posterior probability of guilt.
Solution
def posterior_guilt(match_prob, population, prior_guilt=None): if prior_guilt is None: prior_guilt = 1.0 / population p_match_given_guilty = 1.0 p_match_given_innocent = match_prob # odds form prior_odds = prior_guilt / (1 - prior_guilt) lr = p_match_given_guilty / p_match_given_innocent post_odds = prior_odds * lr return post_odds / (1 + post_odds) print(posterior_guilt(1e-6, 10**7)) # ≈ 0.0909 — about 1 in 11With a 1-in-a-million match probability and 10 million potential suspects, roughly 10 innocent people also match. Absent any other evidence, the match alone gives about 9% probability of guilt — not the 99.9999% the fallacy suggests.
The parameter that matters most is the prior, i.e. how the suspect was identified. A cold database trawl over 10 million profiles gives the calculation above; a match against a suspect already identified by independent evidence gives a much higher prior and a correspondingly higher posterior. Same likelihood ratio, entirely different conclusion — which is the point.
- Program
Write a rejection sampler for an arbitrary conditional distribution and use it to verify a hand-computed conditional probability.
Solution
def conditional_estimate(sample, condition, event, trials=10**6): kept = hit = 0 for _ in range(trials): x = sample() if not condition(x): continue # reject kept += 1 hit += event(x) return hit / kept, kept / trials # estimate, acceptance rateThis computes P(event\midcondition) by literally implementing the definition — generate from the unconditional distribution, discard the outcomes where the condition fails, and measure the frequency among the survivors.
The acceptance rate is the diagnostic: if it is tiny, the estimate has few effective samples and a large standard error ≈√{p(1-p)/kept}, and you need importance sampling or MCMC instead. Reporting the acceptance rate alongside the estimate is what turns this from a plausible number into a defensible one.
Apply it to the two-children problem with both sampling procedures from the chapter's Interview angle: the two give 1/3 and 1/2, and the code makes the difference between them explicit and undeniable.
- Program
Estimate π by Monte Carlo and determine how many samples are needed for 3 decimal places.
Solution
import random, math def estimate_pi(n): inside = sum(1 for _ in range(n) if random.random()**2 + random.random()**2 <= 1) return 4 * inside / nThe estimator is 4\hat{p} with p = π/4, so its standard error is 4√{p(1-p)/n} ≈ 1.64/√n. For 3 decimals (error < 5×10^{-4} with 95% confidence) you need 1.96×1.64/√n < 5×10^{-4}, i.e. n > 4×10⁷.
That is the general and rather brutal lesson about Monte Carlo: accuracy improves as 1/√n, so each additional decimal digit costs 100× more samples. Monte Carlo is the method of choice for high-dimensional integrals precisely because that 1/√n rate is independent of dimension — but for a one-dimensional problem like this, any deterministic quadrature rule crushes it.
- Program
Implement exact conditional probability computation over a small explicit sample space, and use it to resolve the two-children problem in both formulations.
Solution
from itertools import product def P(space, event, given=lambda o: True): num = sum(w for o, w in space if given(o) and event(o)) den = sum(w for o, w in space if given(o)) return num / den # Formulation 1: "at least one boy" learned as a fact about the family fam = [((a, b), 0.25) for a, b in product('BG', repeat=2)] print(P(fam, lambda o: o == ('B','B'), lambda o: 'B' in o)) # 1/3 # Formulation 2: you meet one child uniformly at random and it is a boy meet = [((a, b, i), 0.125) for a, b in product('BG', repeat=2) for i in (0,1)] print(P(meet, lambda o: o[0] == 'B' and o[1] == 'B', lambda o: o[o[2]] == 'B')) # 1/2The two sample spaces differ: the second includes which child you met, and that extra randomness changes the conditioning. Writing both explicitly settles an argument that verbal reasoning never does.
This tiny
Pfunction — sum the weights, divide — is worth keeping. Almost every discrete probability puzzle becomes unambiguous once you are forced to enumerate Ω and its weights in code.
Random variables and linearity of expectation
Linearity of expectation holds without independence, and that is the single most useful fact in probabilistic analysis. It turns problems that look intractable into a sum of trivial indicator probabilities.
A random variable is a function from the sample space to the reals. Its expectation is
Linearity holding without independence is what makes the whole technique work. To compute a complicated expectation, write X as a sum of simple indicators and add up their individual probabilities — the dependence between them is irrelevant.
The indicator method
1. Write X = Σ Xᵢ where each Xᵢ is an indicator (0/1).
2. Compute E[Xᵢ] = P(event i occurs) — usually one line.
3. Sum. Ignore all dependence between the Xᵢ.
Fixed points of a random permutation
Xᵢ = [σ(i)=i], P = 1/n, E[X] = n·(1/n) = 1
Number of inversions
X_{ij} = [i<j and σ(i)>σ(j)], P = 1/2, E = C(n,2)/2 = n(n−1)/4
Comparisons in randomised quicksort
X_{ij} = [ranks i,j are compared], P = 2/(j−i+1)
E = Σ 2/(j−i+1) = 2(n+1)H_n − 4n ≈ 1.39 n log₂ n
Empty bins, n balls into n bins
Xᵢ = [bin i empty], P = (1−1/n)ⁿ ≈ 1/e, E ≈ n/eWhen an expectation looks hard, ask: what am I counting? Then write that count as a sum of indicators, one per thing that might contribute. Almost every "expected number of X" problem in algorithm analysis collapses this way, and the dependence that makes the distribution intractable never enters the computation.
Conditional expectation
Conditioning on the first step is the standard way to set up expectation recurrences. For the expected number of coin flips to see the first head with P(H)=p: condition on the first flip, E = p·1 + (1-p)(1+E), giving E = 1/p.
The tail-sum formula
For a non-negative integer variable, E[X] = ∑_{k\ge1}P(X≥ k). This is often far easier than summing k· P(X=k), because "at least k" probabilities are frequently products while "exactly k" probabilities are differences.
Example: the expected maximum of n uniform draws from {1,…,m}. P(max ≥ k) = 1 - ((k-1)/m)^n, so E[max] = ∑_{k=1}^{m}(1 - ((k-1)/m)^n) = m - ∑_{j=0}^{m-1}(j/m)^n ≈ (mn)/(n+1).
"Expected number of comparisons in quicksort" is the canonical linearity question. The move that lands: ranks i and j are compared exactly when the first pivot chosen from the range [i..j] is one of them, which has probability 2/(j-i+1). No independence is needed, no recurrence is solved, and the answer follows from a double sum.
- Theory
Prove linearity of expectation and explain why independence is not required.
Solution
Working from the definition over the sample space:
E[X+Y] = ∑_{ω}(X(ω)+Y(ω))P(ω) = ∑_{ω}X(ω)P(ω) + ∑_{ω}Y(ω)P(ω) = E[X]+E[Y]. ∎
Independence never appears because the proof never factorises anything — it merely splits a sum of two terms. Contrast E[XY] = ∑ X(ω)Y(ω)P(ω), which does not split unless the joint distribution factorises.
The practical upshot: you may decompose any random variable into a sum however you like, no matter how tangled the pieces are, and the expectations still add.
- Theory
Compute the expected number of inversions in a random permutation of n.
Solution
For each pair i<j, let X_{ij} = [σ(i) > σ(j)]. By symmetry P(X_{ij}=1) = 1/2.
E[inv] = ∑_{i<j}\frac12 = (1)/(2)C(n,2) = (n(n-1))/(4).
The indicators are highly dependent (if σ(1)>σ(2) and σ(2)>σ(3) then σ(1)>σ(3) is forced) and it does not matter at all. ∎
Consequence: insertion sort runs in Θ(n + inv), so its expected cost on a random input is Θ(n²) with the constant 1/4 — matching the observed behaviour, and explaining why it is only competitive on nearly-sorted data where the inversion count is small.
- Theory
Derive the expected number of comparisons in randomised quicksort.
Solution
Let the elements be ranked 1..n and X_{ij} indicate that ranks i and j are ever compared (i < j).
They are compared iff, among the elements of rank i..j, the first one chosen as a pivot is i or j: if any middle element is chosen first, i and j land in different partitions and never meet again. All j-i+1 of them are equally likely to be picked first, so P(X_{ij}=1) = (2)/(j-i+1).
E = ∑_{i<j}(2)/(j-i+1) = ∑_{k=2}^{n}(n-k+1)\frac2k = 2(n+1)H_n - 4n ≈ 1.386nlog_2 n. ∎
No recurrence, no independence, three lines. The same argument gives the expected depth of a random BST, because a random BST's comparison structure is quicksort's.
- Theory
Prove the tail-sum formula E[X] = ∑_{k\ge1}P(X≥ k) for non-negative integer X.
Solution
Swap the order of summation:
∑_{k\ge1}P(X≥ k) = ∑_{k\ge1}∑_{m≥ k}P(X=m) = ∑_{m\ge1}P(X=m)∑_{k=1}^{m}1 = ∑_{m\ge1}mP(X=m) = E[X]. ∎
Equivalently, X = ∑_{k\ge1}[X≥ k] as a pointwise identity, and linearity finishes it — the indicator method applied to X itself.
The continuous analogue is E[X] = \int_0^{∞}P(X>t)dt, which is how expected latency is computed from a tail-latency curve: the area under the survival function is the mean.
- Theory
Compute the expected number of draws to collect all n coupon types.
Solution
Split the process into phases: phase i lasts from having i-1 distinct coupons to having i. In phase i the probability of a new coupon each draw is (n-i+1)/(n), so the phase length is geometric with mean (n)/(n-i+1).
E[T] = ∑_{i=1}^{n}(n)/(n-i+1) = n∑_{k=1}^{n}\frac1k = nH_n ≈ nln n + γn. ∎
For n = 50 that is about 225 draws — four and a half times the number of coupons. The variance is ≈ n²π²/6 (Chapter 11), so the standard deviation is 1.28n: individual runs vary a lot, which is why a single observation of a collection process tells you very little.
- Theory
n people throw their hats in a pile and each takes one at random. What is the expected number who get their own hat? Does it depend on n?
Solution
X_i = [person i gets their own hat] with P = 1/n. So E[X] = n·\frac1n = 1, independent of n.
This is the fixed-point count of Chapter 27, and the constancy is genuinely surprising: with a million people, exactly one person on average gets their own hat.
The variance is also 1 (Chapter 27 problem 3), and the distribution converges to Poisson(1) — so P(nobody)→ e^{-1}, matching the derangement result. Three different chapters, one fact.
- Theory
State and prove Jensen's inequality, and give a case where E[g(X)] ≠ g(E[X]) matters in practice.
Solution
Jensen: for convex g, E[g(X)] ≥ g(E[X]); for concave g, the inequality reverses.
Proof: convexity means g lies above each tangent line. Take the tangent at μ = E[X]: g(x) ≥ g(μ) + g'(μ)(x-μ) for all x. Taking expectations, the linear term vanishes and E[g(X)]≥ g(μ). ∎
In practice: average latency versus latency of the average load. Queueing delay is convex in utilisation (Chapter 8's Little's law discussion), so a service alternating between 40% and 100% utilisation has far worse mean latency than one steadily at 70% — even though the mean utilisation is identical. That is Jensen, and it is why smoothing load matters more than reducing average load.
Another: E[1/X] ≠ 1/E[X], so averaging rates and averaging times give different answers — the source of endless confusion in benchmark reporting.
- Program
Verify the quicksort comparison count empirically against 2(n+1)H_n - 4n.
Solution
import random def quicksort_count(a): cmps = 0 def rec(lo, hi): nonlocal cmps if hi - lo < 1: return p = random.randint(lo, hi) a[p], a[hi] = a[hi], a[p] pivot = a[hi]; i = lo for j in range(lo, hi): cmps += 1 if a[j] < pivot: a[i], a[j] = a[j], a[i]; i += 1 a[i], a[hi] = a[hi], a[i] rec(lo, i-1); rec(i+1, hi) rec(0, len(a)-1) return cmps n = 10000 H = sum(1.0/i for i in range(1, n+1)) avg = sum(quicksort_count(random.sample(range(10*n), n)) for _ in range(50))/50 print(avg, 2*(n+1)*H - 4*n)The two agree to well under 1% over 50 trials. Note the exact formula counts comparisons in the idealised model where the pivot is compared with every other element in its subarray — matching this Lomuto partition exactly, which is why the agreement is so close.
Hoare's partition scheme uses fewer comparisons (about 1.39nlog_2 n versus the same leading term but a smaller lower-order correction) and far fewer swaps, which is why real implementations use it despite Lomuto being easier to state.
- Program
Compute the expected number of distinct values when drawing k times with replacement from n, and verify by simulation.
Solution
import random def expected_distinct(n, k): return n * (1 - (1 - 1.0/n)**k) # indicator per value def simulate(n, k, trials=20000): return sum(len({random.randrange(n) for _ in range(k)}) for _ in range(trials)) / trialsDerivation: X_i = [value i appears], and P(X_i=0) = (1-1/n)^k, so E[distinct] = n(1-(1-1/n)^k). For k=n this is n(1-e^{-1}) ≈ 0.632n — the complement of the 37% empty bins.
This formula is the basis of cardinality estimation: observing the number of distinct values in a sample lets you invert the relation to estimate n. It is also why HyperLogLog exists — computing distinct counts exactly needs Θ(n) memory, while this kind of probabilistic relation gives 2% accuracy in a few kilobytes.
- Program
Solve by simulation and exactly: expected number of rolls of a fair die until all six faces appear, and the expected number until a 6 appears.
Solution
import random def all_faces(): seen, n = set(), 0 while len(seen) < 6: seen.add(random.randint(1,6)); n += 1 return n def until_six(): n = 0 while True: n += 1 if random.randint(1,6) == 6: return n print(sum(all_faces() for _ in range(10**6))/10**6) # ≈ 14.7 print(sum(until_six() for _ in range(10**6))/10**6) # ≈ 6.0All faces: coupon collector, 6H_6 = 6(1+\tfrac12+\tfrac13+\tfrac14+\tfrac15+\tfrac16) = 14.7.
First six: geometric with p=1/6, mean 1/p = 6. Condition on the first roll: E = \frac16(1) + \frac56(1+E), giving E = 6.
A worthwhile follow-up that catches people out: the expected number of rolls until two consecutive sixes is 42, not 12 — the states are "no six" and "one six", and the recurrence E_0 = 1+\frac56E_0+\frac16E_1, E_1 = 1+\frac56E_0 gives 42. Patterns are not independent of their own prefixes, which is Chapter 41's subject.
- Program
Estimate the expected length of the longest increasing subsequence of a random permutation and compare to 2√n.
Solution
import random, bisect, math def lis(a): tails = [] for x in a: i = bisect.bisect_left(tails, x) if i == len(tails): tails.append(x) else: tails[i] = x return len(tails) for n in (100, 1000, 10000, 100000): avg = sum(lis(random.sample(range(n), n)) for _ in range(200))/200 print(n, round(avg,2), round(2*math.sqrt(n),2))The measured means track 2√n closely from below — the Ulam–Hammersley constant, proved to be exactly 2 by Vershik–Kerov and Logan–Shepp, with fluctuations of order n^{1/6} following the Tracy–Widom distribution.
The LIS algorithm itself is O(nlog n) via patience sorting, and the number of piles is the LIS length. Note the connection to Chapter 6: Erdős–Szekeres guarantees a monotone subsequence of length √n in the worst case, and the random case is exactly twice that — the extremal bound is tight up to the constant.
- Program
Implement the "secretary problem" optimal strategy and verify the 1/e success probability.
Solution
import random, math def secretary(n, k): """reject the first k, then take the first candidate better than all seen""" perm = random.sample(range(n), n) best_seen = max(perm[:k]) if k else -1 for i in range(k, n): if perm[i] > best_seen: return perm[i] == n-1 return False n = 1000 for k in (200, 300, int(n/math.e), 400, 500): wins = sum(secretary(n, k) for _ in range(50000)) / 50000 print(k, round(wins, 4))The success probability peaks at k = n/e ≈ 368, giving ≈ 1/e = 0.368. The exact probability for a given k is (k)/(n)∑_{i=k+1}^{n}(1)/(i-1) ≈ (k)/(n)ln(n)/(k), maximised at k/n = 1/e.
The result is remarkable for being scale-free: whether there are 10 candidates or 10 million, observe 37% and then take the next record. The same threshold structure appears in online algorithms generally, where it is the basis of competitive-ratio bounds for secretary-type problems.
- Program
Compute the expected value of a complicated combinatorial quantity by both simulation and exact enumeration for small n, to validate an indicator argument.
Solution
from itertools import permutations import random def exact_expected(n, f): ps = list(permutations(range(n))) return sum(f(p) for p in ps) / len(ps) def simulated_expected(n, f, trials=200000): a = list(range(n)) tot = 0 for _ in range(trials): random.shuffle(a); tot += f(tuple(a)) return tot / trials # e.g. expected number of "descents" σ(i) > σ(i+1) descents = lambda p: sum(p[i] > p[i+1] for i in range(len(p)-1)) for n in range(2, 9): print(n, exact_expected(n, descents), (n-1)/2)The indicator argument gives E[descents] = (n-1)·\frac12, and the exact enumeration confirms it for n\le8. Having both an exact small-n computation and a simulation is the right validation pattern: the exact version catches systematic errors in the indicator probability, the simulation extends to n where enumeration is impossible.
The distribution of descents gives the Eulerian numbers, which satisfy A(n,k) = (k+1)A(n-1,k) + (n-k)A(n-1,k-1) — a third family alongside the two kinds of Stirling numbers, counting permutations by descents rather than by cycles or blocks.
- Program
Use linearity to estimate the number of triangles in a large graph by sampling, and analyse the estimator's accuracy.
Solution
import random def triangle_estimate(adj, n, samples=10**6): """sample random triples; scale up by the number of triples""" hits = 0 for _ in range(samples): a, b, c = random.sample(range(n), 3) if b in adj[a] and c in adj[b] and c in adj[a]: hits += 1 from math import comb return hits / samples * comb(n, 3)Unbiased by linearity: each triple contributes an indicator with mean T/C(n,3), so the scaled sample mean has expectation T.
But the variance is terrible: if T \ll C(n,3), almost every sample misses and the relative error is ≈√{C(n,3)/(T···amples)}. For a sparse graph with T = 10⁶ and n = 10⁵, you would need 10^{11} samples for 10% accuracy.
The practical fix is wedge sampling: sample a random path of length 2 and check whether it closes. The hit rate becomes 3T/W where W is the wedge count — orders of magnitude better, and it is the basis of real streaming triangle counters. Being unbiased is necessary but nowhere near sufficient; the variance decides whether an estimator is usable.
Variance and concentration: Markov, Chebyshev, Chernoff
An expectation alone says nothing about whether a random variable is ever near it. Three inequalities of increasing strength bound the deviation, and choosing the right one is a matter of what you know: non-negativity, variance, or independence.
Unlike expectation, variance does not add unconditionally. It adds when the variables are uncorrelated, which pairwise independence guarantees — and pairwise independence is much cheaper to arrange than full independence, which is why 2-universal hash families suffice for many analyses.
The three inequalities
The strength gap is enormous. For n fair coin flips, Markov gives nothing useful, Chebyshev bounds the probability of exceeding 0.6n heads by O(1/n), and Chernoff bounds it by e^{-Θ(n)}. At n = 1000: Chebyshev says ≤ 2.5%, Chernoff says ≤ 10^{-9}.
Choose by what you have. Only non-negativity? Markov. A variance? Chebyshev — and it needs only pairwise independence, so it survives with limited randomness. A sum of bounded independent terms? Chernoff or Hoeffding, and take the exponential bound. The commonest analysis error is applying Chernoff to dependent variables.
What concentration buys you
Randomised algorithms with high-probability guarantees. An algorithm correct with probability 2/3 can be run O(log(1/δ)) times with a majority vote; Chernoff says the majority is wrong with probability δ. That is why BPP amplification is cheap.
Load balancing. With n balls in n bins, Chernoff bounds the probability that any bin exceeds clog n/loglog n — a union bound over bins then gives the max-load result of Chapter 31.
Sampling. To estimate a proportion p to within ±ε with confidence 1-δ, Hoeffding gives n ≥ (ln(2/δ))/(2ε²) samples — independent of the population size, which is why polls of 1,000 people work for a country of any size.
Suppose an estimator is within ±ε of the truth with probability 2/3.
Run it k = O(log(1/δ)) times independently and take the MEDIAN.
The median is outside ±ε only if at least k/2 of the runs are —
but each fails independently with probability 1/3, so by Chernoff
P(≥ k/2 failures) ≤ exp(−k/36)
Choosing k = 36 ln(1/δ) gives failure probability ≤ δ.
Cost: a log(1/δ) factor. Gain: any constant success probability
becomes 1 − δ. This is how Count-Min, AMS and most streaming
sketches get their guarantees."How many samples do I need?" is a Hoeffding question and the answer is O(log(1/δ)/ε²) — note the 1/ε², so halving the error costs 4× the samples, while raising the confidence from 95% to 99.9% costs only a small constant factor. Knowing which parameter is expensive is the whole practical content.
- Theory
Prove Markov's and Chebyshev's inequalities.
Solution
Markov: for X\ge0 and a>0, consider Y = a·[X≥ a]. Then Y ≤ X pointwise (when the indicator is 1, Y = a ≤ X; when 0, Y = 0 ≤ X). Taking expectations, aP(X≥ a) ≤ E[X]. ∎
Chebyshev: apply Markov to (X-μ)² ≥ 0 with threshold k²σ²:
P(|X-μ|≥ kσ) = P((X-μ)²≥ k²σ²) ≤ (E[(X-μ)²])/(k²σ²) = (1)/(k²). ∎
Both are tight in the worst case: Markov is attained by a variable that is 0 or a; Chebyshev by one taking three values μ, μ± kσ with the right weights. Their weakness for real distributions is the price of assuming nothing else.
- Theory
For n fair coin flips, compare the bounds on P(heads ≥ 0.6n) from Chebyshev and Chernoff at n = 100, 1000, 10^4.
Solution
μ = n/2, σ = √n/2. Exceeding 0.6n means deviating by 0.1n = 0.2√nσ, so k = 0.2√n.
n Chebyshev 1/k² = 25/n Chernoff exp(−δ²μ/(2+δ)), δ=0.2 100 0.25 exp(−0.04·50/2.2) = 0.40 1000 0.025 exp(−0.04·500/2.2) = 1.1e-4 10⁴ 0.0025 exp(−0.04·5000/2.2) = 1.1e-40Chebyshev decays as 1/n, Chernoff as e^{-cn}. At n=100 Chebyshev is actually better (Chernoff's constant is loose for small deviations); by n=10^4 the gap is 38 orders of magnitude. The true probability at n=10^4 is about 10^{-89}, so even Chernoff is conservative — but it is in the right regime, which Chebyshev never is.
- Theory
Show that pairwise independence suffices for the variance to add, and give an example where it fails without it.
Solution
Var(∑ X_i) = ∑_iVar(X_i) + ∑_{i≠ j}Cov(X_i,X_j). Pairwise independence forces every covariance to 0, leaving the sum of variances. ∎ Note only uncorrelatedness is needed, which is weaker still.
Failure: let X_1 = X_2 = ··· = X_n all equal the same fair coin. Then Var(∑ X_i) = n²Var(X_1) = n²/4, not n/4 — off by a factor of n.
This is why the analysis of a hash-based sketch specifies the independence level of its hash family: 2-universal gives Chebyshev-strength bounds (enough for Count-Min), while 4-universal is needed for the second-moment estimator in AMS. More independence costs more space per hash function, so the exact requirement matters.
- Theory
Derive the Chernoff bound for the sum of independent Bernoulli variables.
Solution
Use the exponential moment (Chernoff's method). For t>0,
P(X≥ a) = P(e^{tX}≥ e^{ta}) ≤ \frac{E[e^{tX}]}{e^{ta}} by Markov.
By independence E[e^{tX}] = ∏_iE[e^{tX_i}] = ∏_i(1 + p_i(e^t-1)) ≤ ∏_ie^{p_i(e^t-1)} = e^{μ(e^t-1)}, using 1+x≤ e^x.
So P(X≥(1+δ)μ) ≤ \exp(μ(e^t-1) - t(1+δ)μ). Optimising over t gives t = ln(1+δ) and
P ≤ [\frac{e^{δ}}{(1+δ)^{1+δ}}]^{μ} ≤ \exp(-(δ²μ)/(2+δ)). ∎
Independence entered at exactly one step — factorising the MGF — and that is the assumption Chernoff cannot do without. Martingale versions (Azuma–Hoeffding) relax it to bounded differences, which is how concentration is proved for dependent processes.
- Theory
How many samples are needed to estimate a proportion within ±1% at 95% confidence? Does the population size matter?
Solution
By Hoeffding with X_i ∈ [0,1]: P(|\hat{p}-p|≥ε) ≤ 2e^{-2nε²}. Setting the right side to 0.05 with ε = 0.01:
n ≥ (ln(2/0.05))/(2(0.01)²) = (3.69)/(0.0002) ≈ 18{,}445.
The normal approximation is less conservative: n = (1.96/0.01)²· p(1-p) ≤ 9604 at p=1/2 — the familiar "about 10,000 for 1%", and about 1,000 for 3%.
Population size is irrelevant (for sampling with replacement, or when the population is much larger than the sample). This surprises people constantly: polling 1,000 people estimates a country of 300 million as accurately as a town of 30,000. The finite-population correction √{(N-n)/(N-1)} only bites when the sample is a substantial fraction of the population.
- Theory
Use Chernoff plus a union bound to prove the O(log n/loglog n) max-load result.
Solution
With n balls in n bins, bin i's load is Bin(n,1/n) with μ=1. Chernoff (or the direct bound P(load≥ k) ≤ C(n,k)n^{-k} ≤ 1/k!) gives
P(load_i ≥ k) ≤ (1)/(k!) ≤ ((e)/(k))^k.
Union bound over the n bins: P(max≥ k) ≤ n(e/k)^k. Choosing k = (clog n)/(loglog n) makes klog(k/e) > 2log n for c\ge3, so the bound is ≤ 1/n. ∎
The union bound is doing essential work: Chernoff bounds one bin, and the union bound converts that into a statement about the maximum over all bins at the cost of a factor n — which the exponential decay absorbs easily. This "Chernoff then union bound" pattern is the standard route to any "with high probability, every…" statement.
- Theory
Explain the median trick and why the mean would not work.
Solution
Given k independent estimators each within ±ε with probability 2/3, the median is outside ±ε only if at least half the estimators are. Each fails independently with probability 1/3, so the expected failure count is k/3 and Chernoff bounds P(≥ k/2 failures) ≤ e^{-k/36}. With k = 36ln(1/δ) the failure probability is ≤ δ. ∎
Why not the mean: a single wild outlier can drag the mean arbitrarily far, and the failure events are only bounded in probability, not in magnitude — an estimator that fails might return anything. The median is robust: it is wrong only if a majority is wrong, and each one's magnitude is irrelevant.
This is why Count-Min uses the minimum (its errors are one-sided), AMS uses the median of means, and hyperloglog uses a harmonic mean of many registers. The aggregation function is chosen to match the shape of the estimator's error, not by habit.
- Program
Empirically compare Markov, Chebyshev, Chernoff and the true probability for a binomial.
Solution
import math from math import comb, exp def true_tail(n, p, a): return sum(comb(n,k)*p**k*(1-p)**(n-k) for k in range(a, n+1)) def bounds(n, p, delta): mu = n*p; a = (1+delta)*mu markov = mu / a sigma = math.sqrt(n*p*(1-p)) k = (a - mu)/sigma cheb = 1/k**2 if k > 0 else 1.0 chern = exp(-delta*delta*mu/(2+delta)) return markov, cheb, chern, true_tail(n, p, math.ceil(a)) for n in (100, 1000, 10000): print(n, ["%.3g" % v for v in bounds(n, 0.5, 0.2)])The ordering is always Markov ≥ Chebyshev ≥ Chernoff ≥ truth, but the gaps grow dramatically with n. Markov is useless here because it ignores that X is a sum; Chebyshev captures the √n scale but not the exponential tail.
Plot log(bound) against n and the picture is stark: Markov is flat, Chebyshev is logarithmic, Chernoff and the truth are linear with different slopes. That slope difference is the constant-factor looseness in Chernoff, and it is why the tighter Kullback–Leibler form e^{-nD(p+\delta‖p)} is used when the constant matters.
- Program
Implement the median trick around a noisy estimator and measure the failure rate against the predicted bound.
Solution
import random, statistics def noisy_estimator(truth, eps): """within ±eps with probability 2/3, wild otherwise""" if random.random() < 2/3: return truth + random.uniform(-eps, eps) return truth + random.uniform(-100*eps, 100*eps) def median_of(k, truth, eps): return statistics.median(noisy_estimator(truth, eps) for _ in range(k)) for k in (1, 5, 11, 21, 41): fails = sum(abs(median_of(k, 10.0, 0.1) - 10.0) > 0.1 for _ in range(20000)) / 20000 print(k, round(fails, 5), round(math.exp(-k/36), 5))Measured failure rates fall roughly geometrically: about 0.33 at k=1, 0.21 at 5, 0.08 at 11, 0.017 at 21, 0.001 at 41. The Chernoff bound e^{-k/36} is very loose (0.97 at k=1) but has the right exponential shape.
Use an odd k so the median is a single sample rather than an average of two — averaging the middle pair reintroduces exactly the outlier sensitivity the median was chosen to avoid.
- Program
Implement a Count-Min sketch and verify its error guarantee empirically.
Solution
import random class CountMin: def __init__(self, eps, delta): self.w = int(math.ceil(math.e / eps)) # width = e/ε self.d = int(math.ceil(math.log(1/delta))) # depth = ln(1/δ) self.t = [[0]*self.w for _ in range(self.d)] self.seeds = [random.randrange(1<<30) for _ in range(self.d)] def add(self, key, c=1): for i, s in enumerate(self.seeds): self.t[i][hash((s, key)) % self.w] += c def query(self, key): return min(self.t[i][hash((s, key)) % self.w] for i, s in enumerate(self.seeds))Guarantee: the estimate never underestimates (counts only ever accumulate), and with probability 1-δ it overestimates by at most ε N where N is the total count.
Why: in one row, the expected collision mass at a key's cell is N/w = ε N/e. Markov gives P(error > ε N) ≤ 1/e for that row; the rows are independent, so all d rows exceeding it has probability e^{-d} ≤ δ — and the minimum takes the best row. That is Markov plus independence, not Chernoff: because the error is one-sided, the min is the right aggregator and a weak per-row bound suffices.
Verify by streaming a Zipf-distributed key stream and plotting the estimate against the truth: heavy hitters are essentially exact, tail keys are inflated.
- Program
Simulate a randomised algorithm with success probability 2/3 and measure how many repetitions are needed for 99.9% confidence.
Solution
import random def amplify(p_success, k, trials=100000): """majority vote over k independent runs""" wins = 0 for _ in range(trials): good = sum(random.random() < p_success for _ in range(k)) wins += (good > k // 2) return wins / trials for k in (1, 5, 11, 21, 31, 51): print(k, round(amplify(2/3, k), 5))Measured: 0.667, 0.868, 0.949, 0.986, 0.9958, 0.9996. Reaching 99.9% needs about k=35.
Chernoff predicts the failure probability decays as e^{-k/36} for p=2/3; the exact rate is e^{-kD(1/2‖1/3)} with D = 0.0589 nats, so k ≈ ln(1000)/0.0589 = 117 by the KL bound — still conservative against the measured 35, because the bound ignores the discreteness of the majority.
The practical point: any constant advantage over 1/2 amplifies to arbitrary confidence at logarithmic cost. That is the content of the complexity class BPP, and it is why a Monte Carlo algorithm with a 51% success rate is as good as one with 99%, up to a constant factor in runtime.
- Program
Verify that variance adds for pairwise-independent variables and fails for dependent ones, using a concrete hash family.
Solution
import random def pairwise_family(p): """h(x) = (a·x + b) mod p is 2-universal""" a = random.randrange(1, p); b = random.randrange(p) return lambda x: (a*x + b) % p def measure_variance(family_factory, n, trials=20000, p=101): tot = tot2 = 0 for _ in range(trials): h = family_factory(p) s = sum(1 for x in range(n) if h(x) < p//2) # indicator sum tot += s; tot2 += s*s mean = tot/trials return mean, tot2/trials - mean*meanFor the 2-universal family, the measured variance matches n· p(1-p) ≈ n/4 — the sum of individual variances, exactly as pairwise independence predicts.
Replace the family with h(x) = (a· 1 + b) mod p (ignoring x, so all indicators are identical) and the variance jumps to n²/4 — the failure case of problem 3, made measurable.
The experiment makes the abstract requirement concrete: "the hash family must be 2-universal" is not boilerplate, it is precisely the condition under which the variance calculation in the sketch's analysis is valid.
- Program
Compute the sample size needed for an A/B test to detect a given effect size, and simulate the false-positive and false-negative rates.
Solution
import random, math def sample_size(p0, mde, alpha=0.05, power=0.8): """two-proportion test, normal approximation""" z_a = 1.96 # two-sided α = 0.05 z_b = 0.84 # power = 0.8 p1 = p0 + mde pbar = (p0 + p1) / 2 num = (z_a*math.sqrt(2*pbar*(1-pbar)) + z_b*math.sqrt(p0*(1-p0)+p1*(1-p1)))**2 return math.ceil(num / mde**2) def simulate(n, p0, p1, trials=20000): sig = 0 for _ in range(trials): a = sum(random.random() < p0 for _ in range(n)) b = sum(random.random() < p1 for _ in range(n)) pa, pb = a/n, b/n pool = (a+b)/(2*n) se = math.sqrt(2*pool*(1-pool)/n) if se > 0 and abs(pb-pa)/se > 1.96: sig += 1 return sig/trialsWith p_0 = 0.10 and a minimum detectable effect of 1 percentage point, n ≈ 14{,}750 per arm. Simulating with p_1 = p_0 gives ≈ 5% significant (the false-positive rate, by construction); with p_1 = 0.11 it gives ≈ 80% (the power).
The 1/mde² scaling is the crucial practical fact: detecting a 0.5% lift instead of 1% needs four times the traffic. That is why small effects are expensive to measure and why peeking at results early — which inflates the false-positive rate far above 5% — is such a common and damaging mistake.
- Program
Implement a reservoir sampler and prove/verify uniformity, then extend it to weighted sampling.
Solution
import random, heapq def reservoir(stream, k): res = [] for i, x in enumerate(stream): if i < k: res.append(x) else: j = random.randrange(i+1) if j < k: res[j] = x return res def weighted_reservoir(stream, k): """A-Res: key = u^(1/w); keep the k largest keys""" h = [] for x, w in stream: key = random.random() ** (1.0/w) if len(h) < k: heapq.heappush(h, (key, x)) elif key > h[0][0]: heapq.heapreplace(h, (key, x)) return [x for _, x in h]Uniformity proof for the unweighted version: by induction, after i items each is in the reservoir with probability k/i. Item i+1 enters with probability k/(i+1); an existing item survives with probability 1 - (k)/(i+1)·\frac1k = (i)/(i+1), so its overall probability is (k)/(i)·(i)/(i+1) = (k)/(i+1). ∎
Verify empirically with a stream of 100 items and k=10 over 10⁶ runs: each item should appear in about 10% of reservoirs, and a chi-square test over the 100 counts should pass. That test catches the classic bug of using
randrange(i)instead ofrandrange(i+1), which biases against later items.
The distributions that show up in computer science
Six distributions cover almost everything a systems or algorithms engineer meets. Knowing each one's mean, variance and the situation that generates it is enough to recognise which one a problem is asking about.
| Distribution | Generated by | Mean | Variance |
|---|---|---|---|
| Bernoulli(p) | one trial | p | p(1-p) |
| Binomial(n,p) | n independent trials | np | np(1-p) |
| Geometric(p) | trials until first success | 1/p | (1-p)/p² |
| Poisson(λ) | rare events, fixed rate | λ | λ |
| Uniform{1..n} | a fair choice | (n+1)/2 | (n²-1)/12 |
| Exponential(λ) | waiting time, memoryless | 1/λ | 1/λ² |
| Normal(μ,σ²) | sums of many small effects | μ | σ² |
| Zipf / power law | preferential attachment | often ∞ | often ∞ |
The memoryless property
The geometric and exponential distributions are the only ones satisfying P(X > s+t | X > s) = P(X > t). A process that has waited s units is statistically identical to a fresh one.
This has a strong practical consequence: if request inter-arrival times are exponential, knowing that no request has arrived for 10 seconds tells you nothing about when the next will come. It also means "expected remaining wait" is constant — the basis of the Poisson process model used for queueing and for retry-timeout reasoning.
Heavy tails
The distributions above (except the last) have thin tails: deviations of many standard deviations are effectively impossible. Real systems frequently do not.
A power law P(X > x) \propto x^{-α} has infinite variance when α ≤ 2 and infinite mean when α ≤ 1. File sizes, request latencies, city populations, word frequencies and node degrees in scale-free networks all follow approximate power laws — and for them the sample mean does not converge usefully, the central limit theorem does not apply in the usual form, and "average latency" is a nearly meaningless statistic.
For heavy-tailed data, report percentiles, not means. A p99 latency is a statement about a specific customer experience; a mean latency dominated by a few enormous requests describes nobody. This is why SLOs are written as "99% of requests under 300 ms" rather than "mean under 50 ms".
Where each one appears
Hash table bucket load Binomial(n, 1/m) ≈ Poisson(n/m)
Attempts until a hash succeeds Geometric
Requests arriving per second Poisson (if independent arrivals)
Time between failures Exponential (constant hazard rate)
Sum of many independent costs Normal (CLT)
Skip list node height Geometric(1/2)
Treap priorities Uniform — that is the whole point
Number of retries until success Geometric
Cache miss run lengths often heavy-tailed, NOT geometric
File sizes / request sizes Power law / log-normal"What is the expected number of probes in open addressing at load factor α?" — successful search \frac1αln(1)/(1-α), unsuccessful (1)/(1-α). The shape to state is that it blows up as α\to1: at 90% load an unsuccessful search averages 10 probes, at 99% it averages 100. That is why resize thresholds sit at 0.7–0.75.
- Theory
Prove that the geometric distribution is memoryless, and that it is the only discrete one.
Solution
Memoryless: P(X>n) = (1-p)^n, so
P(X>s+t| X>s) = \frac{(1-p)^{s+t}}{(1-p)^s} = (1-p)^t = P(X>t). ∎
Uniqueness: let g(n) = P(X>n). Memorylessness says g(s+t) = g(s)g(t) for all non-negative integers. The only solutions with g(0)=1 and 0≤ g\le1 are g(n) = c^n, which is the geometric survival function with p = 1-c. ∎
The continuous analogue (Cauchy's functional equation with a monotonicity condition) singles out the exponential. This is why the exponential is the model for a constant-hazard process — assuming exponential inter-arrivals is exactly assuming the process has no memory.
- Theory
Derive the Poisson distribution as the limit of a binomial.
Solution
Take n→∞ with p = λ/n:
P(X=k) = C(n,k)((λ)/(n))^k(1-(λ)/(n))^{n-k}
= (n(n-1)···(n-k+1))/(n^k)·(λ^k)/(k!)·(1-(λ)/(n))^{n}(1-(λ)/(n))^{-k}.
As n→∞ the first factor → 1, the third → e^{-λ}, the fourth → 1, leaving \frac{λ^ke^{-λ}}{k!}. ∎
The approximation is good when n ≥ 20 and p ≤ 0.05. This is exactly the hash-table bucket-load situation: many keys, tiny probability each lands in a given bucket, product fixed at the load factor — hence Poisson, and hence the e^{-1} empty-bucket fraction at load 1.
- Theory
Derive the expected number of probes for successful and unsuccessful search in linear probing at load factor α.
Solution
For uniform hashing (idealised, no clustering):
- Unsuccessful: each probe hits an occupied slot with probability α, so the number of probes is geometric: E = (1)/(1-α).
- Successful: an element inserted when the load was i/m cost (1)/(1-i/m) probes; averaging over the n elements gives \frac1αln(1)/(1-α).
For linear probing, primary clustering makes it worse: \frac12(1+(1)/((1-α)²)) unsuccessful and \frac12(1+(1)/(1-α)) successful (Knuth).
α uniform unsucc. linear probing unsucc. 0.50 2.0 2.5 0.75 4.0 8.5 0.90 10.0 50.5 0.95 20.0 200.5Linear probing degrades far faster, which is the price of its excellent cache locality. Robin Hood hashing keeps the locality while bounding the variance of probe lengths, which is why it has become the common choice.
- Theory
State the central limit theorem precisely and explain when it fails.
Solution
CLT: for i.i.d. X_i with finite mean μ and finite variance σ²,
(∑ X_i - nμ)/(σ√n) \xrightarrow{d} N(0,1).
Failure modes:
- Infinite variance — a power law with α\le2 converges instead to a stable law with heavy tails, not a normal.
- Dependence — strongly correlated terms need mixing conditions; without them the sum may not concentrate at all.
- The tails — the CLT describes the centre; convergence at 4+ standard deviations is much slower (Berry–Esseen gives an O(1/√n) rate uniformly, which is useless for tail probabilities of order 10^{-6}).
That last point is the one that bites in practice: using a normal approximation to estimate a p99.99 latency from a skewed distribution can be off by orders of magnitude. Use the empirical quantile or an explicit tail model instead.
- Theory
Why is the exponential distribution the right model for time-to-failure with a constant hazard rate, and when is it wrong?
Solution
The hazard rate is h(t) = (f(t))/(1-F(t)) — the instantaneous failure rate given survival to t. Solving h(t) = λ constant gives 1-F(t) = e^{-λt}, the exponential. ∎
When it is wrong: most physical and software systems have a "bathtub" hazard curve — high early (infant mortality, config errors), low in the middle, rising late (wear-out, resource leaks). The exponential models only the flat middle.
The Weibull distribution generalises it with h(t) = (k)/(λ)(t/λ)^{k-1}: k<1 gives decreasing hazard (infant mortality), k=1 recovers the exponential, k>1 gives wear-out. Disk failure data fits k ≈ 0.5–0.8 — decreasing hazard, meaning an old disk that has survived is less likely to fail next month than a new one, which is the opposite of the naive intuition.
- Theory
A skip list gives each node a height from Geometric(1/2). Derive the expected height and the expected search cost.
Solution
Node height: P(h = k) = 2^{-k}, so E[h] = 2 and the expected total pointers is 2n.
Max height: P(some node has h ≥ k) ≤ n2^{-k}, so the max is log_2 n + O(1) with high probability.
Search cost: analyse the search path backwards from the target. At each step you either move left (probability 1/2, since the node's height did not increase) or up (probability 1/2). The number of steps to climb log_2 n levels is thus a negative binomial with mean 2log_2 n. ∎
So a skip list gives O(log n) expected search with a constant of 2, using 2n pointers on average — comparable to a balanced tree, with far simpler code and natural concurrency, which is why it is used in LevelDB, Redis and many lock-free structures.
- Theory
Show that the minimum of n independent Exponential(λ_i) variables is Exponential(∑λ_i), and interpret it.
Solution
P(min > t) = ∏_i P(X_i > t) = ∏_i e^{-λ_it} = e^{-(∑λ_i)t} — the survival function of an Exponential with rate ∑λ_i. ∎
Moreover P(X_j is the minimum) = (λ_j)/(∑_iλ_i), independently of the value of the minimum.
Interpretation: merging independent Poisson processes adds their rates, and each event belongs to source j with probability proportional to its rate. This is the foundation of discrete-event simulation (the next event is drawn from the merged process) and of the Gillespie algorithm in chemical kinetics.
Systems reading: n independent components each failing at rate λ gives a system failing at rate nλ — the MTBF divides by n. That is the arithmetic behind "more components means more failures", and it is why redundancy must be designed to tolerate failures rather than merely to reduce their rate.
- Program
Sample from each of the six distributions using only a uniform generator, via inverse transform sampling.
Solution
import random, math def bernoulli(p): return 1 if random.random() < p else 0 def geometric(p): return math.ceil(math.log(1-random.random())/math.log(1-p)) def exponential(lam):return -math.log(1-random.random())/lam def uniform_int(n): return int(random.random()*n) + 1 def binomial(n, p): return sum(bernoulli(p) for _ in range(n)) def poisson(lam): # Knuth's method L, k, p = math.exp(-lam), 0, 1.0 while True: k += 1; p *= random.random() if p <= L: return k - 1 def normal(mu, sigma): # Box–Muller u1, u2 = random.random(), random.random() return mu + sigma*math.sqrt(-2*math.log(1-u1))*math.cos(2*math.pi*u2)Inverse transform: if U\simUniform(0,1) then F^{-1}(U) has CDF F. It works whenever the CDF can be inverted in closed form — exponential and geometric qualify, the normal does not (hence Box–Muller).
Use
1-random.random()rather thanrandom.random()inside the logarithm: Python's generator can return exactly 0.0 but never 1.0, so this avoidslog(0). That single detail causes intermittent crashes in a surprising number of simulation codebases.Knuth's Poisson method is O(λ) — fine for small λ, but use the PTRS transformed-rejection method for λ > 30.
- Program
Simulate hash-table load and compare the bucket-size histogram to Poisson.
Solution
import random, math from collections import Counter def load_histogram(n, m, trials=100): hist = Counter() for _ in range(trials): buckets = [0]*m for _ in range(n): buckets[random.randrange(m)] += 1 hist.update(buckets) return hist n = m = 100000 h = load_histogram(n, m, trials=10) lam = 1.0 for k in range(8): got = h[k] / (10*m) exp = math.exp(-lam) * lam**k / math.factorial(k) print(k, round(got, 5), round(exp, 5))The agreement is excellent: 0.3679, 0.3679, 0.1839, 0.0613, 0.0153, … matching e^{-1}/k! to three decimals. That 36.8% of buckets are empty at load factor 1 is the practically important number — a "full" hash table wastes over a third of its slots.
Vary the load factor and the histogram becomes Poisson(α). At α=0.75, 47% of buckets are empty and 17% hold two or more — which is why the average chain length being under 1 does not mean collisions are rare.
- Program
Fit a power law to a real-looking dataset and compare mean-based and percentile-based summaries.
Solution
import random, math def pareto(alpha, xmin=1.0): return xmin * (1 - random.random()) ** (-1.0/alpha) data = sorted(pareto(1.5) for _ in range(10**6)) n = len(data) print("mean ", sum(data)/n) print("median ", data[n//2]) print("p95 ", data[int(0.95*n)]) print("p99 ", data[int(0.99*n)]) print("p99.9 ", data[int(0.999*n)]) print("max ", data[-1])With α=1.5 the mean is theoretically \frac{αx_{min}}{α-1} = 3, but the sample mean fluctuates wildly between runs — because the variance is infinite for α\le2, the sample mean is dominated by the largest observation and does not converge at the usual rate.
The median (1.59) and p95 (7.4) are stable across runs; the max varies by orders of magnitude. Run the script twice and compare: the percentiles agree, the mean and max do not.
This is the concrete argument for percentile-based SLOs. Estimate α by maximum likelihood, \hat{α} = 1 + n/∑ln(x_i/x_{min}), and if it comes out below 2, stop reporting means.
- Program
Implement a Poisson-process event simulator (M/M/1 queue) and verify Little's law.
Solution
import random, math, heapq def mm1(lam, mu, T): t = 0.0; queue = 0; events = [] heapq.heappush(events, (exponential(lam), 'arrive')) area = 0.0; last = 0.0; arrivals = 0; total_wait = 0.0 waiting = [] while events: t, kind = heapq.heappop(events) if t > T: break area += queue * (t - last); last = t if kind == 'arrive': arrivals += 1; waiting.append(t) if queue == 0: heapq.heappush(events, (t + exponential(mu), 'depart')) queue += 1 heapq.heappush(events, (t + exponential(lam), 'arrive')) else: total_wait += t - waiting.pop(0) queue -= 1 if queue > 0: heapq.heappush(events, (t + exponential(mu), 'depart')) L = area / T # time-average number in system W = total_wait / arrivals # average time in system return L, W, arrivals / TLittle's law L = λW should hold to within simulation noise, and it does — it is a remarkably general identity requiring only that the system is stable, with no assumption about the arrival or service distributions.
Theory for M/M/1 with ρ = λ/μ: L = (ρ)/(1-ρ) and W = (1)/(μ-λ). At ρ=0.9, L=9; at ρ=0.99, L=99. The simulation makes the blow-up visceral, and it is the quantitative form of "never run a queue near 100% utilisation".
- Program
Compare exponential, Weibull and log-normal fits to a latency dataset and decide which describes the tail.
Solution
import math def fit_exponential(data): return 1.0 / (sum(data)/len(data)) def fit_lognormal(data): logs = [math.log(x) for x in data] mu = sum(logs)/len(logs) sd = math.sqrt(sum((l-mu)**2 for l in logs)/len(logs)) return mu, sd def qq_points(data, quantile_fn): n = len(data); s = sorted(data) return [(quantile_fn((i+0.5)/n), s[i]) for i in range(n)]The diagnostic is a Q–Q plot: plot the sorted data against the fitted distribution's quantiles. A good fit is a straight line; systematic curvature at the top-right means the model underestimates the tail.
Real service latencies are almost never exponential — the Q–Q plot bends sharply upward, because the tail is heavier. Log-normal often fits the body well and still understates the extreme tail; a mixture (fast path plus slow path) or an explicit Pareto tail above some threshold usually fits best.
The engineering consequence: a capacity model built on an exponential assumption will systematically under-provision for the p99.9, and the error compounds through any queueing calculation built on it.
- Program
Implement a treap and verify that random priorities give expected O(log n) depth.
Solution
import random class Node: __slots__ = ('key','pri','l','r') def __init__(self, k): self.key=k; self.pri=random.random(); self.l=self.r=None def split(t, k): if not t: return None, None if t.key < k: a, b = split(t.r, k); t.r = a; return t, b a, b = split(t.l, k); t.l = b; return a, t def merge(a, b): if not a or not b: return a or b if a.pri > b.pri: a.r = merge(a.r, b); return a b.l = merge(a, b.l); return bA treap is a BST by key and a heap by priority. Because priorities are random and independent of the keys, the tree's shape is distributed exactly as a random BST built by inserting the keys in a uniformly random order — so the expected depth is 2ln n ≈ 1.39log_2 n and the maximum is 4.31ln n with high probability (Chapter 11).
Measure it: build a treap from 10⁵ sorted keys — the worst case for an unbalanced BST — and confirm the depth is still about 24 rather than 10⁵. That experiment is the whole argument for randomised balancing: the adversary controls the keys, but not the priorities.
- Program
Write a goodness-of-fit test (chi-square) and use it to validate a random number generator's uniformity and independence.
Solution
import random from collections import Counter def chi_square_uniform(samples, k): obs = Counter(int(x*k) for x in samples) n = len(samples); exp = n/k return sum((obs[i]-exp)**2/exp for i in range(k)), k-1 def serial_test(samples, k): """tests PAIRS — catches generators that are uniform but correlated""" obs = Counter((int(samples[i]*k), int(samples[i+1]*k)) for i in range(len(samples)-1)) n = len(samples)-1; exp = n/(k*k) return sum((obs[(i,j)]-exp)**2/exp for i in range(k) for j in range(k)), k*k-1The uniformity test alone is weak — a generator emitting 0, 1/k, 2/k, … in strict rotation passes it perfectly. The serial test on consecutive pairs catches that immediately, and testing triples catches the classic lattice structure of linear congruential generators (Marsaglia's theorem: LCG outputs in d dimensions lie on relatively few hyperplanes).
Run both on
random.random()(Mersenne Twister — passes) and on a hand-rolled LCG with a poor multiplier (passes uniformity, fails the serial test in 3 dimensions). For anything security-related usesecretsor a CSPRNG: Mersenne Twister's full state is recoverable from 624 consecutive outputs, so it is fine for simulation and disqualifying for tokens.
Expected running time of randomised algorithms
Randomisation buys you a guarantee against the adversary rather than against the input. The analysis is almost always linearity of expectation plus one clever indicator, and the distinction between Las Vegas and Monte Carlo decides what the guarantee actually is.
The crucial framing: a randomised algorithm's guarantee holds for every input, with the probability taken over the algorithm's own coin flips. A deterministic algorithm with good average-case behaviour guarantees nothing on the input an adversary picks. That difference is why randomised quicksort is used and deterministic quicksort with a fixed pivot rule is not.
The standard analyses
QUICKSORT E[comparisons] = 2(n+1)H_n − 4n ≈ 1.39 n log₂n
indicator: ranks i,j compared ⟺ first pivot in [i..j] is i or j → 2/(j−i+1)
QUICKSELECT E[comparisons] ≈ 3.4n (linear, not n log n)
recurrence T(n) = n + (1/n)Σ max-side sizes → T(n) ≤ 4n
TREAP / RANDOM BST E[depth] = 2 ln n ≈ 1.39 log₂n, max ≈ 4.31 ln n
indicator: node i is an ancestor of j ⟺ i has the highest priority in [i..j]
SKIP LIST E[search] = 2 log₂ n, E[space] = 2n pointers
each level halves; the backward path is a sequence of fair coin flips
COUPON COLLECTOR E[draws] = nH_n ≈ n ln n
phase i is geometric with p = (n−i+1)/nRandomisation removes the adversary, it does not remove the worst case. Quicksort still has an O(n²) execution path — it just has probability n^{-Θ(1)} of being taken, and the adversary cannot steer you into it. If you need a hard bound rather than a probabilistic one, you need a deterministic algorithm (median-of-medians) and you pay a constant factor for it.
Karger's min cut: the clean Monte Carlo example
Repeatedly contract a uniformly random edge until two vertices remain; output the resulting cut. A fixed minimum cut of size k survives a contraction with probability at least 1 - (k)/(|E|) ≥ 1 - (2)/(|V|), since every vertex has degree at least k. Multiplying over the n-2 contractions:
Two things are worth noticing. First, a success probability as low as 1/C(n,2) is still polynomially small, so polynomially many repetitions amplify it — that is the whole trick. Second, the argument also proves a combinatorial fact for free: a graph has at most C(n,2) distinct minimum cuts, because each is found with probability at least 1/C(n,2) and they are disjoint events.
"Why randomise quicksort?" — not for the average case, which sorted-input quicksort also has over random inputs, but because the guarantee then holds per input. A deterministic median-of-three pivot has known adversarial inputs (the Antiquicksort construction) that force Θ(n²), and those inputs have been used as denial-of-service attacks against real systems.
- Theory
Prove that randomised quickselect runs in expected O(n) time.
Solution
Call a pivot "good" if it falls in the middle half of the range, which happens with probability 1/2. A good pivot reduces the array to at most 3n/4.
Let T(n) be the expected comparisons. Since we expect 2 pivot choices before a good one, and each costs at most n comparisons:
T(n) ≤ 2n + T(3n/4), so T(n) ≤ 2n(1 + \tfrac34 + \tfrac9{16} + ···) = 8n. ∎
The exact analysis gives ≈ 3.4n for finding the median. The key structural point is that the recursion has a single branch: unlike quicksort, only one side is explored, so the geometric series converges to a constant times n rather than picking up a log n factor (Chapter 10's root-heavy case).
- Theory
Show that the expected depth of a node in a random BST is 2ln n, using an ancestor indicator.
Solution
Let A_{ij} indicate that node of rank i is an ancestor of node of rank j. In a random BST (equivalently, a treap with random priorities), i is an ancestor of j exactly when i has the highest priority among ranks [min(i,j), max(i,j)] — because whichever of that range is inserted first becomes their common ancestor, and it separates them only if it is strictly between.
So P(A_{ij}) = (1)/(|i-j|+1), and
E[depth(j)] = ∑_{i≠ j}(1)/(|i-j|+1) = ∑_{d=1}^{j-1}(1)/(d+1) + ∑_{d=1}^{n-j}(1)/(d+1) ≈ 2ln n. ∎
Averaging over j gives the expected total path length ≈ 2nln n — the same quantity as quicksort's comparison count, because the BST's structure is quicksort's recursion tree.
- Theory
Analyse Karger's algorithm and derive the bound on the number of distinct minimum cuts.
Solution
Fix a minimum cut C of size k. Every vertex has degree ≥ k (else its own singleton cut would be smaller), so |E| ≥ nk/2 and a uniformly random edge lies in C with probability ≤ (k)/(nk/2) = \frac2n.
After n-i contractions there are i vertices left, so the survival probability is
∏_{i=n}^{3}(1-\frac2i) = ∏_{i=3}^{n}(i-2)/(i) = (2)/(n(n-1)) = (1)/(\binom n2). ∎
Corollary: for each distinct minimum cut, the algorithm outputs it with probability ≥ 1/\binom n2. These are disjoint events, so their probabilities sum to at most 1, giving at most \binom n2 distinct minimum cuts. That bound is tight — the cycle C_n achieves it — and it is a purely combinatorial theorem proved by analysing a randomised algorithm, which is a striking use of the probabilistic method.
- Theory
Explain Freivalds' algorithm for verifying AB = C in O(n²) and prove its error bound.
Solution
Pick a uniformly random vector r ∈ {0,1}^n and check A(Br) = Cr. Each product is a matrix–vector multiply, so the cost is O(n²) rather than the O(n^{2.37}) of recomputing AB.
If AB = C, the test always passes. If AB ≠ C, let D = AB-C ≠ 0 with some d_{ij}\ne0. Condition on all coordinates of r except r_j: the i-th entry of Dr is d_{ij}r_j + (fixed), which is zero for at most one of the two values of r_j. So the test fails to detect the error with probability ≤ 1/2. ∎
Repeat k times: error probability ≤ 2^{-k}, cost O(kn²). With k=100 the failure probability is below the chance of undetected hardware error, at a cost far below one matrix multiplication.
This is the archetype of a randomised verifier: checking is cheaper than computing, which is the intuition behind the class NP and behind probabilistically checkable proofs.
- Theory
Compare Las Vegas and Monte Carlo formally, and state when each can be converted to the other.
Solution
- Las Vegas → Monte Carlo: always possible. Run for 2E[T] steps and give up; by Markov the failure probability is ≤ 1/2. Amplify by repetition.
- Monte Carlo → Las Vegas: possible only if answers are efficiently verifiable. Run until a verified-correct answer appears; the expected number of runs is 1/p. Miller–Rabin cannot be converted this way (a "probably prime" verdict is not checkable in polynomial time without a certificate), but Karger's min cut can be — sort of, since verifying a cut is minimum is itself the original problem.
The asymmetry matters in practice: a Las Vegas algorithm's output needs no trust, only patience. A Monte Carlo algorithm's output needs an error budget that must be tracked through the rest of the system — and error budgets compound across calls, which is why a routine used a billion times needs a per-call error rate far below what feels intuitively safe.
- Theory
Derive the expected number of comparisons in randomised quicksort with duplicate keys using three-way partitioning.
Solution
With k distinct values each appearing with multiplicity, three-way partitioning (Dutch national flag) separates <, =, > and recurses only on the outer two.
The key observation: two elements of equal value are compared at most once — when one of them is the pivot — and then never separated again. So the analysis reduces to the distinct case on the k distinct values, weighted by multiplicities.
The result is O(nlog k) rather than O(nlog n), and in the extreme where all elements are equal it is O(n). ∎
This matters: two-way partitioning on an array of all-equal keys is Θ(n²) with the naive "stop on equal" rule and Θ(nlog n) with the "swap on equal" rule. The all-equal input is a real and easily-triggered adversarial case, and three-way partitioning is the standard defence.
- Theory
Analyse the expected time for randomised binary search on a skip list, and compare to a balanced tree.
Solution
Trace the search path backwards from the target. At each node on the path, the search either came from the left at the same level (probability 1/2, the node's height stopped there) or from above (probability 1/2). So the reversed path is a sequence of coin flips, and we need log_2 n "up" moves.
The number of flips until log_2 n heads is negative binomial with mean 2log_2 n. ∎
Versus a balanced tree: a red–black tree guarantees ≤ 2log_2 n comparisons in the worst case, so the skip list's expectation merely matches the tree's guarantee. What the skip list buys is simplicity (no rotations), better concurrency (updates are local pointer swaps, amenable to lock-free implementation), and cache-friendly range scans at the bottom level.
What it costs: 2n pointers on average versus n, and no worst-case bound — a pathological run of coin flips can produce a tall, useless tower, though only with probability n^{-Θ(1)}.
- Program
Implement randomised and deterministic quicksort, and construct an adversarial input that defeats the deterministic version.
Solution
# The "antiquicksort" adversary: decide comparison results lazily so that # whatever pivot the algorithm picks turns out to be near-extremal. class Adversary: def __init__(self, n): self.val = [0]*n # gas: unassigned self.candidate = 0 self.nsolid = 0 def compare(self, i, j): # if either is "gas", make it the current candidate pivot for x in (i, j): if self.val[x] == 0: self.val[x] = self.nsolid; self.candidate = x return self.val[i] - self.val[j]This is McIlroy's Antiquicksort. It never commits to an array; it answers comparisons so that the chosen pivot always turns out to be the smallest remaining element, forcing Θ(n²). Running it against a fixed-pivot or median-of-three quicksort produces a concrete array that takes quadratic time — and such arrays were used to attack real systems, prompting the switch to introsort in the C++ standard library.
Against randomised quicksort the adversary is powerless: it cannot predict which pivot will be chosen, so no fixed input is bad with more than negligible probability.
- Program
Implement Karger's contraction algorithm with union-find and measure the empirical success rate against 1/\binom n2.
Solution
import random def karger(n, edges): dsu = DSU(n); remaining = n e = edges[:] random.shuffle(e) for u, v in e: if remaining == 2: break if dsu.find(u) != dsu.find(v): dsu.unite(u, v); remaining -= 1 return sum(1 for u, v in edges if dsu.find(u) != dsu.find(v)) def karger_repeated(n, edges, trials): return min(karger(n, edges) for _ in range(trials))Shuffling the edge list once and scanning is equivalent to repeatedly contracting a uniformly random remaining edge — a neat simplification that avoids maintaining edge multiplicities.
Measure: for a graph with a known min cut, run 10⁴ single trials and count successes. For n=10 the predicted rate is 1/45 = 2.2%, and the measured rate is typically higher (the bound is worst-case, attained by the cycle).
Karger–Stein improves this to O(n²log³n) by observing that early contractions are safe and only the last few are risky: recurse twice on graphs contracted down to n/√2 vertices. The success probability rises to Ω(1/log n) per run.
- Program
Implement Freivalds' verification and compare its runtime to recomputing the product, for n=2000.
Solution
import random def freivalds(A, B, C, k=20): n = len(A) for _ in range(k): r = [random.randint(0,1) for _ in range(n)] Br = matvec(B, r) ABr = matvec(A, Br) Cr = matvec(C, r) if ABr != Cr: return False return TrueAt n=2000: computing AB is 8×10⁹ multiply-adds; 20 rounds of Freivalds is 20×3×4×10⁶ = 2.4×10⁸ — about 30× faster, with an error probability below 10^{-6}.
Over a finite field the analysis is cleaner: choosing r uniformly from 𝔽_p^n gives error probability 1/p per round by the Schwartz–Zippel lemma, so a single round with a 64-bit prime suffices.
Over floating point, be careful: exact equality is the wrong test, and a tolerance reintroduces the possibility of masking a genuine small error. Verification of numerical computations needs a different framework (interval arithmetic or a residual bound), which is Chapter 66's territory.
- Program
Empirically measure the distribution of quicksort's running time and show the tail is polynomially, not exponentially, small.
Solution
import random def qs_comparisons(n, trials=20000): out = [] for _ in range(trials): a = random.sample(range(10*n), n) out.append(quicksort_count(a)) return sorted(out) n = 1000 data = qs_comparisons(n) import math H = sum(1.0/i for i in range(1, n+1)) mean = 2*(n+1)*H - 4*n for q in (0.5, 0.9, 0.99, 0.999, 1.0): print(q, data[min(len(data)-1, int(q*len(data)))] / mean)Typical ratios to the mean: p50 ≈ 0.99, p90 ≈ 1.13, p99 ≈ 1.30, p99.9 ≈ 1.45, max ≈ 1.6. The distribution is concentrated but with a visible right tail.
The theory says P(T > cnln n) = n^{-Θ(c)} — polynomial decay, not exponential, because a single unlucky pivot early on has a lasting effect. That is why quicksort's worst case is a genuine (if rare) risk and why introsort exists: track the recursion depth, and if it exceeds 2log_2 n, switch to heapsort for a hard O(nlog n) ceiling.
- Program
Implement a randomised algorithm for approximate median selection using sampling, and bound its error.
Solution
import random, math def approx_median(a, eps=0.01, delta=0.01): n = len(a) k = math.ceil(math.log(2/delta) / (2*eps*eps)) # Hoeffding sample = [random.choice(a) for _ in range(k)] sample.sort() return sample[k//2]By Hoeffding, the sample median's rank in the full array is within ±ε n of n/2 with probability 1-δ. For ε=δ=0.01 that is k ≈ 26{,}500 samples — independent of n.
So an approximate median of a billion elements costs 26,500 samples plus a sort of them: milliseconds, versus a full O(n) pass. When an approximate answer suffices — choosing a partition pivot, picking a histogram bucket boundary, estimating a p50 — sampling is dramatically cheaper.
The same principle underlies approximate quantile sketches (t-digest, KLL), which maintain ε-approximate quantiles in O(\frac1εlog\frac1ε) space over an unbounded stream.
- Program
Implement a Bloom filter, derive the optimal number of hash functions, and measure the false-positive rate.
Solution
import math class Bloom: def __init__(self, n, fp_rate): self.m = math.ceil(-n*math.log(fp_rate)/(math.log(2)**2)) self.k = max(1, round(self.m/n * math.log(2))) self.bits = bytearray((self.m + 7)//8) def _idx(self, item): h1, h2 = hash((0, item)), hash((1, item)) return [(h1 + i*h2) % self.m for i in range(self.k)] # Kirsch–Mitzenmacher def add(self, item): for i in self._idx(item): self.bits[i>>3] |= 1 << (i&7) def __contains__(self, item): return all(self.bits[i>>3] >> (i&7) & 1 for i in self._idx(item))Analysis: after n insertions with k hashes into m bits, a given bit is 0 with probability (1-1/m)^{kn}≈ e^{-kn/m}. A false positive needs all k bits set: p ≈ (1-e^{-kn/m})^k. Minimising over k gives k^* = (m)/(n)ln 2 and p = 2^{-k^*}, i.e. m = -(nln p)/((ln 2)²) — about 9.6 bits per element for a 1% false-positive rate, regardless of how large the elements are.
The Kirsch–Mitzenmacher trick — deriving k hashes as h_1 + ih_2 from two — provably preserves the asymptotic false-positive rate while computing only two hashes. Measure the empirical rate against the formula; agreement to within a few percent confirms both the sizing and the hash quality.
- Program
Compare randomised and deterministic selection on adversarial input, measuring both time and worst-case behaviour.
Solution
import random, time def bench(fn, arrays, k): t = time.perf_counter() for a in arrays: fn(a[:], k) return time.perf_counter() - t n = 200000 random_arr = [random.sample(range(10*n), n) for _ in range(20)] sorted_arr = [list(range(n)) for _ in range(20)] organ_pipe = [list(range(0,n,2)) + list(range(n-1,0,-2)) for _ in range(20)] for name, arrs in (("random", random_arr), ("sorted", sorted_arr), ("organ-pipe", organ_pipe)): print(name, "randomised", round(bench(quickselect_random, arrs, n//2), 3), "first-elem", round(bench(quickselect_first, arrs, n//2), 3), "median-of-medians", round(bench(mom_select, arrs, n//2), 3))Expected shape of the results: randomised quickselect is fast and stable across all three input families; the fixed-pivot version is fast on random input and quadratic on sorted input; median-of-medians is uniformly slower by 5–10× but never degrades.
That table is the entire argument for randomisation in one experiment. The right production choice is usually introselect: randomised quickselect with a recursion-depth counter that falls back to median-of-medians, giving the good constant almost always and the hard bound in the rare bad case.
Markov chains, random walks and hitting times
A process with no memory beyond its current state is a matrix, and everything you want to know - long-run behaviour, expected hitting times, absorption probabilities - is a linear algebra question about that matrix.
A Markov chain on a finite state space has transition matrix P with P_{ij} = P(next = j | now = i), rows summing to 1. The Markov property is that the future depends on the present only — the path taken to get here is irrelevant.
Hitting and absorption
Almost all contest and interview questions about Markov chains reduce to one of two linear systems, obtained by conditioning on the first step.
EXPECTED HITTING TIME to a target set T:
h_i = 0 for i ∈ T
h_i = 1 + Σ_j P_ij · h_j otherwise
— "one step, then whatever the expected cost is from there"
ABSORPTION PROBABILITY of reaching set A before set B:
a_i = 1 for i ∈ A, a_i = 0 for i ∈ B
a_i = Σ_j P_ij · a_j otherwise
— no "+1": we are tracking probability, not cost
Both are linear systems in |S| unknowns: solve by Gaussian elimination
in O(|S|³), or by iteration if the chain mixes fast.Write the recurrence by conditioning on the first step. Expected cost gets a +1 (or +c_i for a weighted step); probability does not. Then check whether the system is triangular — many contest problems have a DAG-shaped state graph, in which case it solves in O(|S| + |E|) by topological order instead of O(|S|³).
Random walks on graphs
A uniform random walk on an undirected connected graph is a reversible Markov chain, and its stationary distribution has a beautiful closed form:
The electrical-network analogy is not a metaphor: harmonic functions on a graph satisfy the same equations as voltages, and effective resistance genuinely equals commute time divided by 2|E|. It gives immediate intuition — adding edges reduces resistance, hence reduces commute time (Rayleigh's monotonicity law).
The 1D random walk
"Expected number of coin flips to see HTH" is a Markov chain on prefix states, and the answer (10) differs from HTT (8) even though both patterns have probability 1/8 — because HTH can overlap itself. Recognising that self-overlap is what breaks the naive symmetry, and that the state space is the KMP automaton of the pattern, is the whole answer.
- Theory
Solve the gambler's ruin: starting with k of N, fair bets, find the ruin probability and expected duration.
Solution
Let a_k = P(reach N before 0). Conditioning on the first bet: a_k = \tfrac12a_{k-1} + \tfrac12a_{k+1}, with a_0=0, a_N=1. The recurrence says a_{k+1}-a_k = a_k - a_{k-1}, so the differences are constant and a_k = k/N. ∎
For the duration, h_k = 1 + \tfrac12h_{k-1}+\tfrac12h_{k+1} with h_0=h_N=0. The homogeneous solution is linear and a particular solution is -k², giving h_k = k(N-k). ∎
Consequence for a gambler with 100 units against a house with 10,000: the probability of doubling before ruin is 100/10100 ≈ 1%, and the expected number of bets is 100×10000 = 10^6. Even a fair game ruins the smaller bankroll almost surely — before any house edge is applied.
- Theory
Find the stationary distribution of a random walk on an undirected graph and verify detailed balance.
Solution
Guess π_v = \deg(v)/(2|E|). Check detailed balance for an edge (u,v):
π_uP_{uv} = (\deg(u))/(2|E|)·(1)/(\deg(u)) = (1)/(2|E|) = (\deg(v))/(2|E|)·(1)/(\deg(v)) = π_vP_{vu} ✓.
Detailed balance implies stationarity: summing over u, ∑_uπ_uP_{uv} = ∑_uπ_vP_{vu} = π_v. ∎
So a random walk spends time at each vertex proportional to its degree, and the expected return time is 2|E|/\deg(v). On a bipartite graph the chain is periodic (period 2) so π_n does not converge — the fix is a lazy walk that stays put with probability 1/2, which preserves π and kills the periodicity.
- Theory
Compute the expected number of coin flips to first see HH, HT, THH, and explain why they differ.
Solution
States are the longest prefix of the pattern currently matched.
HT: E_∅ = 1 + \tfrac12E_H + \tfrac12E_∅, E_H = 1 + \tfrac12E_H + \tfrac12·0. So E_H = 2 and E_∅ = 4.
HH: E_H = 1 + \tfrac12·0 + \tfrac12E_∅, E_∅ = 1+\tfrac12E_H+\tfrac12E_∅. Solving gives E_∅ = 6.
THH: E = 8.
Why HH > HT: after a failed HH attempt (H then T) you fall back to the empty state, losing all progress. After a failed HT attempt (H then H) you are still in state H — no progress lost. Self-overlap costs you.
The general formula is Conway's leading-number algorithm: E = ∑_{k} 2^k[the length-k prefix equals the length-k suffix]. For HH: prefixes/suffixes match at k=1,2, giving 2+4=6 ✓. This is the same autocorrelation structure as the KMP failure function.
- Theory
Two players pick patterns HHT and HTT; who is more likely to see theirs first? Explain non-transitivity.
Solution
By Conway's algorithm, P(A before B) = \frac{L_{BB} - L_{BA}}{L_{AA} - L_{AB}} in odds form, where L_{XY} is the correlation sum of X against Y.
For A = HHT, B = HTT: A wins with probability 2/3.
Penney's game is non-transitive: for any pattern of length ≥ 3 that player 1 picks, player 2 can choose one that beats it. The winning response to b_1b_2b_3 is \bar{b_2}b_1b_2 — prepend the opposite of the second bit.
This is genuinely counter-intuitive: all length-3 patterns are equally likely in a fixed window, yet the race between them is not symmetric, because the loser's partial progress carries over differently depending on the overlap structure. Playing second is a decisive advantage, which is why the game is a classic hustle.
- Theory
State the conditions for a unique stationary distribution and explain what goes wrong without each.
Solution
- Irreducible (every state reaches every other): without it the chain has separate closed communicating classes, each with its own stationary distribution, and any convex combination is stationary — so it is not unique.
- Aperiodic (gcd of return-time lengths is 1): without it, e.g. a walk on a bipartite graph, π_n oscillates and never converges, even though a unique stationary π exists.
- Positive recurrent (automatic for finite chains): on infinite state spaces a chain can be recurrent but with infinite expected return time, giving no stationary distribution at all — the symmetric walk on ℤ is the standard example.
The practical fix for periodicity is the lazy chain P' = \frac12(I+P), which has the same stationary distribution and is always aperiodic. PageRank's damping factor plays both roles at once: it makes the web graph irreducible (teleport anywhere) and aperiodic, guaranteeing convergence from any start.
- Theory
Derive the cover time of a random walk on a path and on a complete graph.
Solution
Complete graph K_n: from any vertex, each step lands on a uniformly random other vertex, so covering is exactly the coupon collector problem: E[cover] = (n-1)H_{n-1} ≈ nln n.
Path P_n: to cover, the walk must reach both endpoints. The expected time to travel distance d on a line is Θ(d²) (gambler's ruin), so the cover time is Θ(n²).
Worst case: the lollipop graph — a clique of n/2 vertices with a path of n/2 attached — has cover time Θ(n³). The walk gets absorbed into the clique (high degree, high stationary mass) and takes Θ(n²) attempts to escape down the path, each attempt costing Θ(n). Matthews' bound shows Θ(n³) is the maximum over all graphs.
Cover time bounds matter for randomised graph exploration with O(log n) memory — the setting where a random walk is the only option because you cannot store a visited set.
- Theory
Explain PageRank as a Markov chain and why the damping factor is needed.
Solution
PageRank is the stationary distribution of a random surfer: with probability d follow a uniformly random outlink, with probability 1-d teleport to a uniformly random page.
π = dπP + (1-d)/(n)\mathbf{1}, typically with d = 0.85.
Why damping is needed: the raw web graph is neither irreducible (many pages are unreachable) nor aperiodic, and it has dangling nodes with no outlinks, whose rows sum to 0 rather than 1. Teleportation repairs all three at once and additionally guarantees a spectral gap: |λ_2| ≤ d, so power iteration converges geometrically at rate 0.85 — about 50 iterations for 10^{-4} accuracy, regardless of graph size.
The choice of d is a trade-off: smaller d converges faster but weights the link structure less; d=0.85 was chosen empirically and has stuck.
- Program
Implement stationary-distribution computation by power iteration and by solving the linear system; compare.
Solution
def power_iteration(P, iters=10000, tol=1e-12): n = len(P); pi = [1.0/n]*n for _ in range(iters): nxt = [sum(pi[i]*P[i][j] for i in range(n)) for j in range(n)] if max(abs(a-b) for a, b in zip(pi, nxt)) < tol: return nxt pi = nxt return pi def solve_stationary(P): """(P^T − I)π = 0 with Σπ = 1 — replace one equation with the constraint""" n = len(P) A = [[P[j][i] - (1 if i == j else 0) for j in range(n)] for i in range(n)] A[-1] = [1.0]*n b = [0.0]*(n-1) + [1.0] return gaussian_solve(A, b)Power iteration is O(n²) per step (or O(|E|) for a sparse chain) and converges at rate |λ_2|^k; the direct solve is O(n³) but exact.
Use power iteration when the chain is sparse and mixes well — PageRank on billions of nodes has no other option. Use the direct solve for small dense chains, or when |λ_2| is near 1 (a bottlenecked chain) and iteration would take forever. Computing λ_2 first tells you which situation you are in.
- Program
Compute expected hitting times for an arbitrary chain by solving the linear system, and verify by simulation.
Solution
def hitting_times(P, targets): n = len(P); idx = [i for i in range(n) if i not in targets] m = len(idx); pos = {v: k for k, v in enumerate(idx)} A = [[0.0]*m for _ in range(m)]; b = [1.0]*m for k, i in enumerate(idx): A[k][k] = 1.0 for j in range(n): if j in targets: continue A[k][pos[j]] -= P[i][j] h = gaussian_solve(A, b) out = [0.0]*n for k, i in enumerate(idx): out[i] = h[k] return out def simulate_hitting(P, start, targets, trials=100000): import random total = 0 for _ in range(trials): s, steps = start, 0 while s not in targets: r = random.random(); acc = 0.0 for j, p in enumerate(P[s]): acc += p if r < acc: s = j; break steps += 1 total += steps return total / trialsThe system is (I - Q)h = \mathbf{1} where Q is P restricted to the transient states. It has a unique solution iff every transient state can reach a target — otherwise I-Q is singular and the expected hitting time is genuinely infinite, which a solver will report as a near-zero pivot.
Simulating and comparing is the right validation: it catches sign errors and the classic mistake of putting the +1 on the wrong side.
- Program
Solve the "expected flips to see a given pattern" problem for arbitrary patterns using a KMP automaton.
Solution
def expected_flips(pattern, alphabet='HT'): m = len(pattern) # KMP failure function → automaton on prefix-length states fail = [0]*m k = 0 for i in range(1, m): while k and pattern[i] != pattern[k]: k = fail[k-1] if pattern[i] == pattern[k]: k += 1 fail[i] = k def go(state, ch): while state and pattern[state] != ch: state = fail[state-1] return state + 1 if pattern[state] == ch else 0 # solve h[state] = 1 + Σ_ch (1/|A|) h[go(state,ch)], h[m] = 0 n = m + 1 A = [[0.0]*n for _ in range(n)]; b = [0.0]*n for s in range(m): A[s][s] = 1.0; b[s] = 1.0 for ch in alphabet: A[s][go(s, ch)] -= 1.0/len(alphabet) A[m][m] = 1.0 return gaussian_solve(A, b)[0]The state is "how much of the pattern is currently matched", and the KMP failure function computes the correct fallback on a mismatch — which is precisely the "progress retained" that makes HH and HT differ.
Check against Conway's formula E = ∑_k |A|^k[prefix_k = suffix_k]: for HH it gives 2+4=6, for HT just 4, for HTH 2+8=10. Agreement across all patterns up to length 6 is a strong test of both the automaton and the linear solve.
- Program
Simulate random walks on ℤ, ℤ², ℤ³ and measure the return probability, confirming Pólya's theorem.
Solution
import random def walk_returns(dim, steps, trials): dirs = [] for d in range(dim): for s in (1, -1): v = [0]*dim; v[d] = s; dirs.append(tuple(v)) returns = 0 for _ in range(trials): pos = tuple([0]*dim) for _ in range(steps): d = random.choice(dirs) pos = tuple(p+q for p, q in zip(pos, d)) if all(c == 0 for c in pos): returns += 1; break return returns / trials for dim in (1, 2, 3): print(dim, walk_returns(dim, 10**6, 2000))Measured return rates within 10⁶ steps: ℤ ≈ 0.999, ℤ² ≈ 0.99, ℤ³ ≈ 0.34.
Pólya's theorem: the walk is recurrent (returns with probability 1) in dimensions 1 and 2, transient in 3 and above, with P(return) ≈ 0.3405 in ℤ³. The 1D and 2D cases return with probability 1 but with infinite expected return time — so the finite step budget makes 2D look slightly transient, and increasing the budget pushes it toward 1 only as 1/log(steps).
That slow convergence is itself the lesson: recurrence with infinite expected return time is qualitatively different from recurrence with finite expected return time, and simulation alone cannot distinguish the 2D case from transience.
- Program
Implement PageRank on a real graph and study the effect of the damping factor on convergence.
Solution
def pagerank(adj, n, d=0.85, iters=100, tol=1e-10): out = [len(adj[v]) for v in range(n)] pr = [1.0/n]*n for it in range(iters): nxt = [(1-d)/n]*n dangling = sum(pr[v] for v in range(n) if out[v] == 0) for v in range(n): if out[v]: share = d * pr[v] / out[v] for w in adj[v]: nxt[w] += share for w in range(n): nxt[w] += d * dangling / n # redistribute dangling mass diff = sum(abs(a-b) for a, b in zip(pr, nxt)) pr = nxt if diff < tol: return pr, it return pr, itersThe dangling-node handling is essential and frequently omitted: a page with no outlinks would otherwise leak probability mass every iteration, and the vector would decay toward zero rather than converging to a distribution. Redistributing that mass uniformly is equivalent to giving dangling nodes links to everywhere.
Measure iterations to convergence against d: d=0.5 takes ~15 iterations, d=0.85 ~50, d=0.99 ~1000 — matching log(tol)/log(d) exactly, since |λ_2|≤ d. That plot is the clearest demonstration of a spectral gap controlling a real computation's cost.
- Program
Implement Metropolis–Hastings to sample from a distribution known only up to a constant, and diagnose its mixing.
Solution
import random, math def metropolis(target_unnormalised, propose, x0, n): x = x0; out = [] accepted = 0 for _ in range(n): y = propose(x) a = target_unnormalised(y) / target_unnormalised(x) if random.random() < min(1.0, a): x = y; accepted += 1 out.append(x) return out, accepted/nWhy it works: the acceptance rule enforces detailed balance with respect to the target, so the target is stationary — and crucially, the normalising constant cancels in the ratio, which is what makes the method usable for posteriors and Gibbs distributions where the constant is intractable.
Diagnostics that matter: the acceptance rate (aim for ~0.23 for high-dimensional random-walk proposals, ~0.44 in one dimension); the autocorrelation time (how many samples until they are effectively independent); and running several chains from different starts to check they agree (the Gelman–Rubin statistic).
A chain that has not mixed produces samples that look fine and are systematically wrong — which is why every MCMC result needs these diagnostics reported alongside it, not merely the posterior summary.
- Program
Model a board game (Snakes and Ladders) as an absorbing Markov chain and compute the expected number of turns exactly.
Solution
def snakes_and_ladders(jumps, size=100): """jumps[i] = destination if square i has a snake/ladder, else i""" n = size + 1 P = [[0.0]*n for _ in range(n)] for i in range(size): for d in range(1, 7): j = i + d if j > size: j = i # overshoot: stay put else: j = jumps.get(j, j) P[i][j] += 1.0/6 P[size][size] = 1.0 return hitting_times(P, {size})[0]For the standard board this gives about 39.2 turns — an exact number obtained from a 101×101 linear solve, no simulation needed.
Two modelling decisions deserve care and are where implementations differ: the overshoot rule (stay put, versus bounce back), and whether a ladder landing on a snake's head chains onward (it should not, on a standard board, but the data structure must encode that). Both change the answer by a turn or two, and both are invisible unless you write them down explicitly.
The general pattern — turn a game into a transition matrix, solve one linear system — handles any game with bounded state and no player choice. Adding choices makes it a Markov decision process, solved by value iteration instead.
Probability and expectation dynamic programming
Contest problems asking for a probability or an expected value are DPs over states, with two twists: the answer is usually required as a fraction modulo a prime, and cyclic dependencies turn the DP into a linear system rather than a recursion.
The setup is always the same. Define a state, write the transition by conditioning on the next random event, and choose an evaluation order. What varies is whether the state graph is acyclic (a plain DP) or has cycles (a linear system, Chapter 41).
Answers as fractions modulo a prime
Contest answers are stated as P· Q^{-1} mod (10⁹+7). This is exact rational arithmetic, not an approximation (Chapter 17), and it means the entire DP runs in ℤ_p with no floating point at all.
MOD = 10**9 + 7
inv6 = pow(6, MOD-2, MOD)
# dp[i] = expected rolls to go from total i to at least n
dp = [0]*(n+1)
for i in range(n-1, -1, -1):
s = 0
for d in range(1, 7):
s = (s + dp[min(i+d, n)]) % MOD # min: clamp at the target
dp[i] = (1 + s*inv6) % MOD # one roll, then the rest
print(dp[0])If the state graph is acyclic, evaluate in topological order — usually just a loop in the right direction. If it has cycles (the process can return to a state it has left), the equations are simultaneous and you must either solve a linear system, or find the algebraic structure that breaks the cycle. Failing to notice a cycle produces a DP that silently reads uninitialised values.
Breaking self-loops algebraically
The commonest cycle is a self-loop: with probability q nothing happens and you stay in state s. That does not need a linear solver:
The 1/(1-q) factor is the expected number of attempts before something happens — a geometric waiting time absorbed into the equation. Recognising this turns many apparently cyclic DPs back into acyclic ones.
Linearity as a DP simplifier
Before building a state machine, ask whether linearity of expectation splits the problem. "Expected number of X" almost always decomposes into a sum of per-item probabilities, each of which may be far easier than the joint process.
Example: the expected number of distinct values remaining after a random deletion process is ∑_i P(item i survives) — and each survival probability is a one-line computation, whereas the joint distribution over surviving sets is exponential.
"Expected number of steps until the process stops" is a backward DP with a +1 per step. The follow-up that separates candidates: is the state graph acyclic? If a state can be revisited, the recursion diverges and you need Gaussian elimination or the self-loop trick — noticing that before writing code is the whole skill.
- Theory
Set up the DP for the expected number of dice rolls until the running total reaches n, and give the asymptotic answer.
Solution
dp[i] = 1 + \frac16∑_{d=1}^{6}dp[min(i+d, n)], with dp[n] = 0, evaluated from i=n-1 down to 0. The state graph is acyclic because the total strictly increases.
Asymptotically, each roll advances the total by 3.5 on average, so by Wald's identity E[rolls] ≈ n/3.5 = 2n/7.
More precisely, renewal theory gives E[rolls] = (n)/(3.5) + (E[D²])/(2E[D]²) - \frac12 + o(1) = (2n)/(7) + 0.7 + o(1). The constant comes from the overshoot at the boundary, and matching it against the DP output for large n is a satisfying check on both.
- Theory
Explain when an expectation DP requires solving a linear system rather than a recursion, with an example.
Solution
A recursion works iff the state graph is a DAG — every transition must move to a state that is evaluated earlier in some fixed order.
Example needing a system: a token on a cycle of n nodes moves left or right with probability 1/2, and you want the expected time to reach node 0. Every state can reach every other, so the equations
h_i = 1 + \tfrac12h_{i-1} + \tfrac12h_{i+1}
are genuinely simultaneous. Solving gives h_i = i(n-i).
Example not needing one: the dice problem above — the total only increases, so a single backward loop suffices.
The diagnostic is mechanical: build the state graph and run a topological sort (Chapter 5). If it succeeds, loop; if it reports a cycle, solve. For |S| ≤ 500 Gaussian elimination at O(|S|³) is fine; beyond that you need structure or iteration.
- Theory
Derive the self-loop elimination formula and apply it to: a machine that succeeds with probability p, and on failure retries from the same state.
Solution
With dp = 1 + (1-p)dp + p·0, rearranging gives dpp = 1, so dp = 1/p — the geometric mean, as expected.
The general form: if state s has a self-loop with probability q and other transitions p_{s→ s'} summing to 1-q, then
dp[s] = \frac{c_s + ∑_{s'≠ s}p_{s→ s'}dp[s']}{1-q}.
The division by 1-q is a modular inverse when working mod a prime, and it fails if q ≡ 1 — which happens exactly when the state is absorbing and unreachable from the target, i.e. the expectation is infinite. Guarding that division with a check is what turns a silent wrong answer into a correct "impossible".
- Theory
A fair coin is flipped until n consecutive heads appear. Derive the expected number of flips.
Solution
Let E_k be the expected additional flips given k consecutive heads so far. Then
E_k = 1 + \tfrac12E_{k+1} + \tfrac12E_0 for k < n, and E_n = 0.
Working downward and solving gives the clean closed form
E_0 = 2^{n+1} - 2.
Check: n=1 gives 2 ✓, n=2 gives 6 ✓ (matching Chapter 41), n=3 gives 14.
The generalisation to a p-biased coin is E = (1-p^n)/(p^n(1-p)), and to a k-sided die needing n consecutive identical faces it is (k^n-1)/(k-1). The exponential growth is the point: waiting for 20 consecutive heads takes about two million flips.
- Theory
Show how linearity of expectation avoids a state-space DP for "expected number of surviving elements".
Solution
Suppose each of n elements is independently deleted in each of T rounds with probability q. The state space of surviving sets has size 2^n — hopeless.
But E[survivors] = ∑_i P(i survives all T rounds) = n(1-q)^T, a one-line computation. ∎
This works even when the deletions are dependent (linearity needs no independence), provided each individual survival probability is computable. That is the crucial advantage: the joint distribution can be intractable while every marginal is easy.
The technique fails when the question is about something other than a sum — "probability that all survive" or "expected maximum" do not decompose, and those genuinely need the joint structure or a different tool (the tail-sum formula for maxima, Chapter 37).
- Theory
Derive the expected value of the maximum of n independent Uniform{1..m} draws, and give the DP-free method.
Solution
By the tail-sum formula, E[max] = ∑_{k=1}^{m}P(max ≥ k), and P(max ≥ k) = 1 - ((k-1)/(m))^n.
E[max] = m - (1)/(m^n)∑_{j=0}^{m-1}j^n.
For large m this tends to (mn)/(n+1) — so 10 draws from 1..100 average about 91.
The power sum ∑ j^n is computable in O(n) by Lagrange interpolation (Chapter 9 problem 13), so the whole expectation costs O(n) regardless of m — versus O(m) for the naive sum and O(m^n) for enumerating outcomes.
Note this is the tail-sum formula doing the same job the indicator method does for sums: converting a hard joint quantity into an easy marginal one.
- Theory
Explain why a probability DP must be normalised and how a common bug produces probabilities exceeding 1.
Solution
Every transition distribution must satisfy ∑_{s'}P(s→ s') = 1. The classic bug is a forward DP that pushes probability along overlapping events — counting the same outcome through two different paths — so the totals exceed 1.
Example: computing "probability that at least one of A or B happens" as P(A)+P(B) inside a DP transition. Correct is P(A)+P(B)-P(A∩ B), or a state machine where the cases are disjoint by construction.
The defence: after every layer of a forward DP, assert that the probabilities sum to 1 (modulo floating-point tolerance, or exactly if working with fractions). Modulo a prime you cannot check this by inspection, so run the same DP with
Fractionon a small instance and assert the sum. That single check catches nearly every transition-modelling error. - Program
Compute the expected number of dice rolls to reach a total of at least n, modulo 10⁹+7, for n ≤ 10⁶.
Solution
MOD = 10**9+7 def expected_rolls(n): inv6 = pow(6, MOD-2, MOD) dp = [0]*(n+7) for i in range(n-1, -1, -1): s = 0 for d in range(1, 7): s += dp[i+d] if i+d < n else 0 dp[i] = (1 + s % MOD * inv6) % MOD return dp[0]O(6n) — instant at n = 10⁶. The sliding window of six terms can be maintained incrementally for a 6× constant-factor win if needed.
Validate against the asymptotic 2n/7 + 0.7 by running the same DP in floating point: at n = 10⁶ it should give 285{,}714.99, and 2×10⁶/7 + 0.7 = 285{,}715.4. Checking a modular answer against a float version of the same recurrence is the standard way to gain confidence in code whose output is an opaque 9-digit residue.
- Program
Solve a probability DP with cycles by Gaussian elimination modulo a prime.
Solution
MOD = 10**9+7 def gauss_mod(A, b): n = len(A) for c in range(n): piv = next(r for r in range(c, n) if A[r][c] % MOD) A[c], A[piv] = A[piv], A[c]; b[c], b[piv] = b[piv], b[c] inv = pow(A[c][c], MOD-2, MOD) A[c] = [v*inv % MOD for v in A[c]]; b[c] = b[c]*inv % MOD for r in range(n): if r != c and A[r][c]: f = A[r][c] A[r] = [(A[r][k] - f*A[c][k]) % MOD for k in range(n)] b[r] = (b[r] - f*b[c]) % MOD return bO(n³) — practical to n ≈ 500 in Python, 2000 in C++. Modular arithmetic means no pivoting for numerical stability is needed; you only pivot to find a non-zero entry.
If
next(...)raisesStopIteration, the column is entirely zero and the system is singular — which for a hitting-time system means some state cannot reach the target, so the expectation is infinite. Catching that and reporting it, rather than crashing, is what a complete implementation does. - Program
A game: you have n lives and each round lose one with probability p. Compute the probability distribution of the round at which you die, and the expected survival time.
Solution
MOD = 10**9+7 def survival(n, p_num, p_den): p = p_num * pow(p_den, MOD-2, MOD) % MOD q = (1 - p) % MOD # dying at exactly round r = negative binomial: C(r-1, n-1) p^n q^{r-n} # expected time = n/p (sum of n geometric waits) return n * pow(p, MOD-2, MOD) % MOD # distribution, for reporting: def death_round_prob(n, r, p): from math import comb return comb(r-1, n-1) * p**n * (1-p)**(r-n)The time to lose each life is Geometric(p) with mean 1/p, and the n waits are independent, so by linearity E[survival] = n/p — no DP required at all.
The full distribution is negative binomial: P(die at round r) = C(r-1,n-1)p^n(1-p)^{r-n}, since the last round must be a death and the other n-1 deaths fall anywhere among the first r-1 rounds.
This is the pattern worth internalising: recognise the standard distribution before writing a DP. A DP over (lives, round) would be O(nR); the closed form is O(1).
- Program
Implement a DP for the expected score in a game where you may stop or continue (an optimal-stopping problem).
Solution
# Roll a die; keep rolling to accumulate, but rolling a 1 loses everything. # Should you roll again with a current bank of b? def expected_optimal(max_bank): dp = [0.0]*(max_bank + 7) for b in range(max_bank, -1, -1): # value of stopping stop = b # value of rolling: 1/6 chance of losing all, else gain 2..6 roll = sum(dp[min(b+d, max_bank)] for d in range(2, 7)) / 6.0 dp[b] = max(stop, roll) return dp[0]The
maxis what makes this a Markov decision process rather than a plain Markov chain: at each state you choose the action maximising the value, and the DP computes the optimal policy along with the value.For this game the optimal policy is a threshold — keep rolling while the bank is below about 20, then stop — and it comes out of the DP without being assumed. Reading the policy off (
roll > stop) is often more interesting than the value itself.Note the state graph is acyclic here only because the bank strictly increases on a non-losing roll. If an action could return you to the same state, this becomes value iteration: repeat the update until the values converge, which is guaranteed by the contraction-mapping argument when there is discounting or a guaranteed terminal state.
- Program
Verify a modular-arithmetic expectation DP against an exact rational version on small inputs.
Solution
from fractions import Fraction MOD = 10**9+7 def dp_fraction(n): dp = [Fraction(0)]*(n+7) for i in range(n-1, -1, -1): dp[i] = 1 + sum(dp[i+d] if i+d < n else 0 for d in range(1,7)) * Fraction(1,6) return dp[0] def dp_modular(n): inv6 = pow(6, MOD-2, MOD) dp = [0]*(n+7) for i in range(n-1, -1, -1): s = sum(dp[i+d] if i+d < n else 0 for d in range(1,7)) dp[i] = (1 + s*inv6) % MOD return dp[0] for n in range(1, 60): f = dp_fraction(n) assert dp_modular(n) == f.numerator % MOD * pow(f.denominator, MOD-2, MOD) % MODThis is the essential validation technique for modular-rational contest code. The residue
314159265is unreadable; the fraction1234/567is checkable by hand for small n, and the assertion ties the two together.Fractionbecomes slow as denominators grow (they can reach 6^n), so cap the cross-check at n ≈ 60. That is more than enough to catch a wrong inverse, a missing +1, or an off-by-one in the boundary. - Program
Compute the probability that a random binary string of length n contains a given pattern, using a KMP-automaton DP.
Solution
MOD = 10**9+7 def prob_contains(n, pattern): m = len(pattern) fail = kmp_failure(pattern) def go(s, ch): while s and pattern[s] != ch: s = fail[s-1] return s + 1 if pattern[s] == ch else 0 inv2 = pow(2, MOD-2, MOD) dp = [0]*(m+1); dp[0] = 1 # not yet matched matched = 0 for _ in range(n): nxt = [0]*(m+1) for s in range(m): if not dp[s]: continue for ch in '01': t = go(s, ch) if t == m: matched = (matched + dp[s]*inv2) % MOD else: nxt[t] = (nxt[t] + dp[s]*inv2) % MOD dp = nxt return matchedO(nm). The automaton state is "length of the longest prefix of the pattern that is a suffix of what we have read" — and once the full pattern is matched we accumulate into an absorbing total rather than continuing.
For large n, note the transition is a fixed linear map on the m-dimensional state vector, so matrix exponentiation (Chapter 48) gives O(m³log n) — the standard way to handle n = 10^{18}. That is the general recipe for "probability a long random string avoids a pattern".
- Program
Solve an expected-value problem on a DAG of states with weighted transitions, and detect when the expectation is infinite.
Solution
from collections import deque def expected_on_dag(n, trans, target): """trans[s] = list of (prob, next, cost); returns dp[s] or None if infinite""" indeg = [0]*n for s in range(n): for _, t, _ in trans[s]: indeg[t] += 1 # topological order restricted to states that can REACH target reach = [False]*n; reach[target] = True rev = [[] for _ in range(n)] for s in range(n): for _, t, _ in trans[s]: rev[t].append(s) q = deque([target]) while q: v = q.popleft() for u in rev[v]: if not reach[u]: reach[u] = True; q.append(u) if not all(reach[s] for s in range(n) if trans[s]): return None # some state cannot reach target # ... then evaluate in reverse topological orderThe reachability check is the part that is usually omitted and matters most: a state from which the target is unreachable has infinite expected hitting time, and a DP that ignores this returns a finite number computed from uninitialised values.
Even with full reachability, a cycle among the transient states makes the system simultaneous rather than topological — so the complete algorithm is: check reachability, run a topological sort, and fall back to Gaussian elimination if the sort reports a cycle. Those three steps cover every case, and skipping any of them produces a solution that works on the samples and fails on the tests.
Vectors, matrices and the product as composition
A matrix is a linear map written in coordinates, and matrix multiplication is function composition. Every identity in this part - associativity, the transpose rule, why AB differs from BA - follows from that one reading.
A linear map T satisfies T(ax+by) = aT(x)+bT(y). Fix a basis and T is determined by where it sends the basis vectors — and those images, written as columns, are the matrix.
Reading Ax as a combination of columns is the single most useful mental shift. The column space is then obviously the set of reachable outputs, the rank is the number of independent columns, and Ax = b is solvable exactly when b lies in the span of the columns.
The four fundamental subspaces
Rank–nullity is the statement that a linear map cannot lose and keep the same dimension twice. It immediately answers most solvability questions: Ax=b has a unique solution iff r = n, infinitely many iff r < n and b is in the column space, and none otherwise.
Matrices you will actually meet
| Matrix | Meaning | Key property |
|---|---|---|
| Adjacency A | graph edges | (A^k)_{ij} = walks of length k |
| Transition P | Markov chain | rows sum to 1; πP = π |
| Rotation | \begin{smallmatrix}\cosθ & -\sinθ\\ \sinθ & \cosθ\end{smallmatrix} | orthogonal, \det = 1 |
| Projection P | drop onto a subspace | P² = P |
| Permutation | reorder coordinates | orthogonal, \det = ±1 |
| Laplacian D-A | graph structure | eigenvalue 0; matrix-tree theorem |
| Companion | a linear recurrence | char. poly = the recurrence |
(A^k)_{ij} counts walks of length k from i to j, because matrix multiplication sums over intermediate vertices — exactly the "last step came from somewhere" decomposition. Change the semiring from (+,×) to (min,+) and the same product computes shortest walks (Chapter 21). Recognising a DP transition as a matrix product is what unlocks O(log n) exponentiation.
Implementation notes
// naive i-j-k: B is traversed by COLUMN — a cache miss per access
for (i) for (j) { s = 0; for (k) s += A[i][k]*B[k][j]; C[i][j] = s; }
// i-k-j: both A and B are traversed by ROW — sequential, vectorisable
for (i) for (k) { a = A[i][k]; for (j) C[i][j] += a * B[k][j]; }
// The two do the SAME n³ multiplications. On a 1024×1024 matrix the
// second is typically 3–8× faster purely from memory locality.
// Blocked/tiled versions add another 2–3×; BLAS adds more still."Multiply two matrices" is a warm-up; the real questions are the follow-ups. Loop order and cache behaviour (above), Strassen's O(n^{2.807}) and why it is rarely used (Chapter 10), and the observation that A(BC) versus (AB)C can differ by orders of magnitude in cost for non-square shapes — which is the matrix-chain DP.
- Theory
Prove that matrix multiplication is associative but not commutative, using the composition interpretation.
Solution
Associativity: matrices represent functions, and function composition is associative — ((f\circ g)\circ h)(x) = f(g(h(x))) = (f\circ(g\circ h))(x) by definition, with no computation. Since the correspondence between matrices and linear maps is a bijection preserving composition, (AB)C = A(BC). ∎
Non-commutativity: take a 90° rotation R and a reflection F in the x-axis. RF reflects then rotates; FR rotates then reflects. Applying each to (1,0) gives (0,1) and (0,-1) respectively. ∎
The composition view also explains (AB)^T = B^TA^T: transposition reverses the direction of the map, so it reverses the order of composition — the same reason (f\circ g)^{-1} = g^{-1}\circ f^{-1}.
- Theory
Prove that (A^k)_{ij} counts walks of length k from i to j in a graph with adjacency matrix A.
Solution
Induction on k. Base k=1: A_{ij} is 1 iff there is an edge, i.e. one walk of length 1. ∎
Step: a walk of length k+1 from i to j is a walk of length k from i to some m, followed by an edge m→ j. These decompositions are in bijection with walks, and summing over m,
(A^{k+1})_{ij} = ∑_m (A^k)_{im}A_{mj},
which is exactly the definition of the matrix product. ∎
Applications: the number of triangles is \frac16tr(A³) (each triangle is counted once per starting vertex and direction); reachability is (I+A)^{n-1} over booleans; and "shortest path with exactly k edges" is the same product in the (min,+) semiring.
- Theory
State and prove the rank–nullity theorem.
Solution
Rank–nullity: for A: ℝ^n→ℝ^m, \dim C(A) + \dim N(A) = n.
Proof: take a basis v_1,…,v_k of N(A) and extend it to a basis v_1,…,v_k,w_1,…,w_{n-k} of ℝ^n. Claim: Aw_1,…,Aw_{n-k} is a basis of C(A).
Spanning: any Ax expands in the basis, and the v_i terms vanish. Independence: if ∑ c_iAw_i = 0 then ∑ c_iw_i ∈ N(A), so it is a combination of the v_j — but the full set is a basis, forcing all c_i = 0. ∎
Consequence: a square matrix is invertible iff its null space is trivial iff its rank is full — three equivalent conditions that get used interchangeably.
- Theory
Show that rank(AB) ≤ min(rank(A), rank(B)) and give a case of strict inequality.
Solution
C(AB) ⊆ C(A), since every column of AB is a combination of columns of A. So rank(AB)\lerank(A).
Also N(B)⊆ N(AB), so by rank–nullity rank(AB)\lerank(B). ∎
Strict: take A = \begin{smallmatrix}0&1\\0&0\end{smallmatrix} and B = A. Both have rank 1, but AB = 0 has rank 0.
Sylvester's rank inequality gives the lower bound rank(AB) ≥ rank(A)+rank(B)-n, which together with the upper bound pins the rank of a product between two computable numbers — useful when reasoning about the rank of a factorised representation, as in low-rank approximation.
- Theory
Explain why a projection satisfies P²=P and derive the least-squares projection matrix.
Solution
Projecting an already-projected vector changes nothing, so P(Px) = Px for all x, i.e. P²=P — the definition of idempotent.
Least squares: to project b onto C(A), we need b - A\hat{x} \perp C(A), i.e. A^T(b - A\hat{x}) = 0. This gives the normal equations A^TA\hat{x} = A^Tb, so \hat{x} = (A^TA)^{-1}A^Tb and the projection is
P = A(A^TA)^{-1}A^T.
Check P² = A(A^TA)^{-1}A^TA(A^TA)^{-1}A^T = A(A^TA)^{-1}A^T = P ✓.
Practical warning: forming A^TA squares the condition number, so the normal equations are numerically poor. Use QR (via Householder reflections) or the SVD instead — the same answer, far better conditioning.
- Theory
Prove that similar matrices B = S^{-1}AS represent the same linear map in different bases, and list what is preserved.
Solution
If S has the new basis vectors as columns, then S converts new-basis coordinates to old. So "convert to old, apply A, convert back" is S^{-1}AS — the same map, expressed in the new basis. ∎
Preserved by similarity: rank, determinant, trace, characteristic polynomial, eigenvalues (with multiplicity), minimal polynomial, and Jordan form.
Not preserved: the individual entries, the singular values, the norm, and orthogonality of eigenvectors (unless S is orthogonal).
This is why "diagonalise" is a strategy: find a basis in which the map is a diagonal matrix, do the easy computation there, and convert back. A^n = SD^nS^{-1} is the archetype, and it is Chapter 47.
- Theory
Explain why loop order changes matrix-multiply performance by several times, in terms of cache lines.
Solution
A cache line holds 64 bytes = 8 doubles. In row-major storage, consecutive elements of a row share cache lines; consecutive elements of a column are n doubles apart.
In the
i-j-korder, the inner loop readsB[k][j]for successive k — striding down a column, so each access touches a new cache line. For a 1024×1024 matrix of doubles (8 MB, far larger than L2), that is a miss on nearly every access.In the
i-k-jorder, the inner loop readsB[k][j]and writesC[i][j]for successive j — both sequential, so one miss per 8 accesses, and the compiler can vectorise.Blocking improves it further: process b× b tiles chosen so three tiles fit in L1, reducing the memory traffic from Θ(n³) to Θ(n³/√M) — the external-memory bound of Chapter 13, achieved.
- Program
Implement matrix multiplication in three loop orders and one blocked version; measure all four at n=1024.
Solution
void mul_ikj(const double* A, const double* B, double* C, int n){ for (int i = 0; i < n; i++) for (int k = 0; k < n; k++){ double a = A[i*n+k]; for (int j = 0; j < n; j++) C[i*n+j] += a * B[k*n+j]; } } void mul_blocked(const double* A, const double* B, double* C, int n, int bs){ for (int ii = 0; ii < n; ii += bs) for (int kk = 0; kk < n; kk += bs) for (int jj = 0; jj < n; jj += bs) for (int i = ii; i < min(ii+bs,n); i++) for (int k = kk; k < min(kk+bs,n); k++){ double a = A[i*n+k]; for (int j = jj; j < min(jj+bs,n); j++) C[i*n+j] += a*B[k*n+j]; } }Typical timings at n=1024 with
-O2: ijk ≈ 4.0 s, jki ≈ 5.5 s, ikj ≈ 0.9 s, blocked (bs=64) ≈ 0.4 s, OpenBLAS ≈ 0.03 s.All perform exactly 2n³ = 2.1×10⁹ floating-point operations. The 130× spread between the worst and BLAS comes entirely from memory hierarchy, vectorisation and instruction scheduling — a vivid demonstration that the RAM model's uniform-cost assumption (Chapter 8) is a fiction at this scale.
- Program
Count triangles, paths of length k, and closed walks in a graph using adjacency-matrix powers.
Solution
import numpy as np def triangles(A): A3 = np.linalg.matrix_power(A, 3) return int(np.trace(A3)) // 6 # each triangle: 3 starts × 2 directions def walks_of_length(A, k): return np.linalg.matrix_power(A, k) # (i,j) entry = walk count def closed_walks(A, k): return int(np.trace(np.linalg.matrix_power(A, k)))Triangle counting this way is O(n^{ω}) with fast matrix multiplication — better than the O(m^{1.5}) combinatorial algorithm only for dense graphs. It is also the standard way to see that triangle detection is no harder than matrix multiplication.
Watch for overflow: walk counts grow like λ_1^k, so for a graph with max degree 10 and k=20 the entries reach 10^{20} — beyond both
int64and the exact range of adouble. Use Python integers, or compute modulo a prime if only a residue is needed.Note that A^k counts walks, not paths: vertices may repeat. Counting simple paths of length k is #W[1]-hard and needs colour-coding or inclusion–exclusion instead.
- Program
Implement a general matrix class over a modular ring with multiplication, power and identity, reusable for DP acceleration.
Solution
MOD = 10**9+7 class Mat: __slots__ = ('n','a') def __init__(self, a): self.a = a; self.n = len(a) @staticmethod def identity(n): return Mat([[int(i==j) for j in range(n)] for i in range(n)]) def __mul__(self, o): n = self.n A, B = self.a, o.a C = [[0]*n for _ in range(n)] for i in range(n): Ai = A[i]; Ci = C[i] for k in range(n): if Ai[k]: aik = Ai[k]; Bk = B[k] for j in range(n): Ci[j] = (Ci[j] + aik*Bk[j]) % MOD return Mat(C) def __pow__(self, e): r = Mat.identity(self.n); b = self while e: if e & 1: r = r * b b = b * b; e >>= 1 return rThe
if Ai[k]guard is a large win for the sparse transition matrices typical of DP acceleration — many DP matrices are mostly zero. Thei-k-jloop order and hoistingAi[k]out of the inner loop both matter even in Python.This class plus
__pow__is all you need for Chapter 48: express the DP transition as a matrix, raise it to the power n, and read the answer off. Keeping it generic (parameterise the modulus, or the multiply/add operations) lets the same code serve (min,+) shortest paths and boolean reachability. - Program
Implement Gaussian elimination to compute rank and a basis for the null space, over the rationals.
Solution
from fractions import Fraction def rref(M): M = [[Fraction(x) for x in row] for row in M] rows, cols = len(M), len(M[0]) pivots = [] r = 0 for c in range(cols): piv = next((i for i in range(r, rows) if M[i][c] != 0), None) if piv is None: continue M[r], M[piv] = M[piv], M[r] inv = M[r][c] M[r] = [v / inv for v in M[r]] for i in range(rows): if i != r and M[i][c] != 0: f = M[i][c] M[i] = [a - f*b for a, b in zip(M[i], M[r])] pivots.append(c); r += 1 if r == rows: break return M, pivots def null_space(M): R, pivots = rref(M) cols = len(M[0]); free = [c for c in range(cols) if c not in pivots] basis = [] for f in free: v = [Fraction(0)]*cols; v[f] = Fraction(1) for r, p in enumerate(pivots): v[p] = -R[r][f] basis.append(v) return basis # dim = cols − rank, by rank–nullityExact rational arithmetic avoids every numerical pitfall, at the cost of coefficient growth — entries can become large fractions even when the input is small integers. For big systems use modular arithmetic plus CRT (Chapter 22), or fraction-free Bareiss elimination which keeps everything integral.
Verify with rank–nullity:
len(pivots) + len(null_space(M)) == colsmust hold for every input, which is a one-line invariant that catches most implementation errors. - Program
Use a matrix formulation to answer "number of paths of length exactly k between all pairs" for k ≤ 10^{18}.
Solution
MOD = 10**9+7 def paths_exactly(adj, n, k): A = Mat([[adj[i][j] % MOD for j in range(n)] for i in range(n)]) return (A ** k).aO(n³log k). At n=100 and k=10^{18}: 10⁶×60 = 6×10⁷ modular multiplications — a few seconds in C++.
Variants obtained by changing the semiring, with no change to the exponentiation code:
- (min,+): shortest walk with exactly k edges.
- (OR,AND) on booleans: is there any walk of length k.
- (max,×): most reliable path with exactly k hops.
That is the payoff of Chapter 21's monoid framing: one exponentiation routine, four different problems, distinguished only by the multiply and add operations passed in.
- Program
Implement matrix inversion by Gauss–Jordan and verify with AA^{-1}=I, over both floats and a modular field.
Solution
def inverse_mod(A, mod): n = len(A) M = [row[:] + [int(i==j) for j in range(n)] for i, row in enumerate(A)] for c in range(n): piv = next((r for r in range(c, n) if M[r][c] % mod), None) if piv is None: return None # singular M[c], M[piv] = M[piv], M[c] inv = pow(M[c][c], mod-2, mod) M[c] = [v*inv % mod for v in M[c]] for r in range(n): if r != c and M[r][c]: f = M[r][c] M[r] = [(M[r][k] - f*M[c][k]) % mod for k in range(2*n)] return [row[n:] for row in M]Modular inversion is exact — no pivoting for stability is needed, only to find a non-zero entry, and the verification AA^{-1} ≡ I holds bit-for-bit.
Over floats, that verification never holds exactly, and the right check is ‖AA^{-1}-I‖ ≤ c· n·ε·κ(A) where κ is the condition number. More importantly: do not compute an inverse to solve Ax=b — LU factorisation followed by two triangular solves is faster and numerically better. Explicit inverses are for when you genuinely need the matrix itself.
- Program
Detect whether a set of vectors is linearly independent over ℚ, over 𝔽_p, and over 𝔽_2, and note where the answers differ.
Solution
def independent_mod(vectors, p): rows = [v[:] for v in vectors] rank = 0; n = len(rows[0]) for c in range(n): piv = next((r for r in range(rank, len(rows)) if rows[r][c] % p), None) if piv is None: continue rows[rank], rows[piv] = rows[piv], rows[rank] inv = pow(rows[rank][c], p-2, p) rows[rank] = [v*inv % p for v in rows[rank]] for r in range(len(rows)): if r != rank and rows[r][c]: f = rows[r][c] rows[r] = [(a - f*b) % p for a, b in zip(rows[r], rows[rank])] rank += 1 return rank == len(vectors)The answers genuinely differ. The vectors (1,0) and (1,2) are independent over ℚ but dependent over 𝔽_2, where 2 ≡ 0 makes the second equal the first.
So "rank" is not a property of the integer matrix alone — it depends on the field. Computing a rank modulo a random prime gives the rational rank with high probability (it can only drop, and it drops only for primes dividing certain minors), which is the basis of fast randomised rank algorithms.
Over 𝔽_2 the whole computation becomes bitwise XOR on machine words, giving a 64× speed-up — the subject of Chapter 46.
Gaussian elimination, rank and solution spaces
One algorithm answers every question about a linear system: does a solution exist, is it unique, what is the whole solution set, and what is the rank. The variations - over the rationals, modulo a prime, over GF(2) - differ only in the arithmetic.
Gaussian elimination reduces a matrix to row-echelon form using three elementary row operations: swap two rows, scale a row by a non-zero constant, add a multiple of one row to another. All three are invertible and preserve the solution set — that is why the algorithm is correct.
The full solution set
When solutions exist, the set is {x_p + v : v∈ N(A)} — one particular solution plus the null space. Reading it off the RREF:
Columns with a leading 1 are PIVOT columns; the rest are FREE.
[1 0 −2 | 3] pivots: x₁, x₂ free: x₃
[0 1 4 | 5]
x₃ = t (free)
x₂ = 5 − 4t
x₁ = 3 + 2t
general solution = (3,5,0) + t·(2,−4,1)
──────── ──────────
particular null-space basis vectorNumber of solutions over a field of size q: 0 if inconsistent, q^{n-r} otherwise. Over ℝ that is 1 when r=n and infinite otherwise; over 𝔽_2 it is 2^{n-r}, a finite count you can report — which is exactly what XOR-equation contest problems ask for.
Numerical stability
Over the reals, the naive algorithm is unstable. The fix is partial pivoting: at each step choose the row with the largest absolute value in the pivot column. Without it, a tiny pivot divides through and amplifies rounding error catastrophically.
The condition number κ(A) = ‖A‖‖A^{-1}‖ bounds the damage: a relative input error ε can become a relative output error κ(A)ε. A matrix with κ = 10^{10} loses 10 digits, which is most of a double.
Modulo a prime none of this applies — the arithmetic is exact, and pivoting is needed only to find a non-zero entry.
LU decomposition
Elimination implicitly factors PA = LU with L unit-lower-triangular and U upper-triangular. Storing L and U lets you solve Ax=b for many different b in O(n²) each after one O(n³) factorisation — which is why libraries factor once and reuse.
"Solve a system of linear equations" expects Gaussian elimination at O(n³). The follow-ups worth pre-empting: many right-hand sides → LU once; the matrix is sparse → iterative methods (conjugate gradient) at O(nnz·√κ); exact answers needed → modular arithmetic plus CRT; only the rank matters → elimination without back-substitution.
- Theory
Solve completely and describe the solution set: x+2y+3z=6, 2x+4y+6z=12, x+y+z=3.
Solution
Row 2 is exactly twice row 1, so it carries no information. Eliminating:
[1 2 3 | 6] [1 2 3 | 6] [1 0 −1 | 0] [2 4 6 |12] → [0 0 0 | 0] → [0 1 2 | 3] [1 1 1 | 3] [0 −1 −2 |−3] [0 0 0 | 0]Rank 2, three unknowns, consistent. So the solution set is one-dimensional:
z = t, y = 3-2t, x = t — that is (0,3,0) + t(1,-2,1).
Check t=1: (1,1,1) gives 1+2+3=6 ✓ and 1+1+1=3 ✓.
- Theory
Prove that elementary row operations preserve the solution set.
Solution
Each operation is left-multiplication by an invertible matrix E: a swap by a permutation matrix, a scaling by a diagonal matrix with one non-unit entry c ≠ 0, and a row addition by I + ce_{ij}.
Then Ax = b \iff EAx = Eb, because E is invertible so the implication runs both ways. ∎
The requirement c\ne0 for scaling is what makes E invertible — scaling a row by 0 destroys information and is not an elementary operation, a distinction that matters when someone "simplifies" a system by zeroing a row.
The same argument shows the row space and null space are preserved, though the column space generally is not.
- Theory
How many solutions does a system of m equations in n unknowns over 𝔽_2 have, in terms of the rank?
Solution
If the system is inconsistent (an all-zero row with a non-zero right-hand side): 0.
Otherwise the solution set is a coset of the null space, which has dimension n-r by rank–nullity. Over 𝔽_2 a d-dimensional space has 2^d elements, so the count is 2^{n-r}. ∎
This is the answer to the standard contest problem "count the assignments satisfying a set of XOR constraints": eliminate, check consistency, report 2^{n-r} mod p. Over 𝔽_q the same reasoning gives q^{n-r}.
- Theory
Explain partial pivoting and construct a system where its absence gives a completely wrong answer.
Solution
Partial pivoting: before eliminating column c, swap in the row with the largest |a_{rc}| among the remaining rows. This bounds the multipliers by 1, so the elimination cannot amplify errors.
The chapter's example makes the failure concrete: with a pivot of 10^{-4} and 3-digit arithmetic, the multiplier is 10^4, which swamps the second row's original entries. The computed y is right to 3 digits but the back-substitution divides a cancellation error by 10^{-4}, magnifying it 10,000×.
With pivoting the multiplier is 10^{-4} and nothing is amplified. ∎
Complete pivoting (search the whole remaining submatrix) is more stable still but costs O(n³) comparisons and is almost never worth it — partial pivoting is stable in practice for essentially all matrices, though contrived counterexamples with exponential growth factors exist.
- Theory
Define the condition number and explain what κ(A) = 10^{12} means for a
double-precision solve.Solution
κ(A) = ‖A‖·‖A^{-1}‖, and for the 2-norm it equals σ_{max}/σ_{min}, the ratio of largest to smallest singular value.
The bound is: a relative perturbation ε in the data can produce a relative error κ(A)ε in the solution.
A
doublehas ε ≈ 2.2×10^{-16} (about 16 digits). With κ = 10^{12} the solution may have relative error 10^{-4} — only 4 correct digits, and no amount of pivoting recovers them, because the information is genuinely absent from the input.Diagnosing this matters: a "wrong" answer from a well-implemented solver on an ill-conditioned system is not a bug. The fixes are regularisation (Tikhonov), higher precision, or reformulating the problem — and the first step is always to compute κ and find out which situation you are in.
- Theory
Derive LU decomposition from Gaussian elimination and explain the benefit for multiple right-hand sides.
Solution
Each elimination step is left-multiplication by a unit-lower-triangular matrix E_k. After all steps, E_{n-1}··· E_1A = U, so A = (E_{n-1}··· E_1)^{-1}U = LU where L is unit-lower-triangular (the product of inverses of unit-lower-triangular matrices is one). With pivoting, PA = LU. ∎
The multipliers can be stored in the zeroed-out positions of A, so LU needs no extra memory.
Benefit: factorise once at O(n³/3), then each new right-hand side costs two triangular solves at O(n²). For n=1000 and 100 right-hand sides: 3.3×10⁸ + 100×10⁶ versus 100×3.3×10⁸ if you re-eliminated each time — a 75× saving.
This is also why computing A^{-1} explicitly is wasteful: it is n triangular solves (one per unit vector) plus the factorisation, and then using it costs another O(n²) per solve with worse accuracy.
- Theory
Given a system with more equations than unknowns and no exact solution, define the least-squares solution and state when it is unique.
Solution
The least-squares solution minimises ‖Ax-b‖_2, and by the projection argument of Chapter 43 it satisfies the normal equations A^TAx = A^Tb.
It is unique iff A has full column rank (r = n), which makes A^TA invertible. Otherwise the minimisers form an affine set, and the conventional choice is the minimum-norm one, given by the pseudoinverse A^+b.
Never form A^TA in floating point: κ(A^TA) = κ(A)², so a system with κ=10⁶ (perfectly solvable) becomes 10^{12} (4 digits left). Use QR: A = QR gives Rx = Q^Tb, a triangular solve with the original condition number. For rank-deficient problems, use the SVD.
- Program
Implement Gaussian elimination with partial pivoting over floats, returning the solution and an estimate of the condition number.
Solution
def solve(A, b): n = len(A) M = [row[:] + [b[i]] for i, row in enumerate(A)] growth = 0.0 for c in range(n): piv = max(range(c, n), key=lambda r: abs(M[r][c])) if abs(M[piv][c]) < 1e-14: raise ValueError("singular or nearly so") M[c], M[piv] = M[piv], M[c] for r in range(c+1, n): f = M[r][c] / M[c][c] for k in range(c, n+1): M[r][k] -= f * M[c][k] growth = max(growth, abs(f)) x = [0.0]*n for i in range(n-1, -1, -1): x[i] = (M[i][n] - sum(M[i][j]*x[j] for j in range(i+1, n))) / M[i][i] # crude condition estimate: ratio of largest to smallest pivot piv_mags = [abs(M[i][i]) for i in range(n)] return x, max(piv_mags) / min(piv_mags)The pivot-ratio estimate is crude but useful — it is cheap and catches the common case. A proper estimate uses the LAPACK
geconapproach (estimate ‖A^{-1}‖ by a few solves with cleverly chosen right-hand sides) at O(n²).Test on the Hilbert matrix H_{ij} = 1/(i+j-1), whose condition number grows like e^{3.5n}: at n=12, κ ≈ 10^{16} and the computed solution is meaningless despite the code being correct. That experiment separates "my algorithm is wrong" from "my problem is ill-posed".
- Program
Solve a linear system modulo a prime and count the solutions.
Solution
MOD = 10**9+7 def solve_mod(A, b, p=MOD): n = len(A); m = len(A[0]) M = [row[:] + [b[i]] for i, row in enumerate(A)] where = [-1]*m r = 0 for c in range(m): piv = next((i for i in range(r, n) if M[i][c] % p), None) if piv is None: continue M[r], M[piv] = M[piv], M[r] inv = pow(M[r][c], p-2, p) M[r] = [v*inv % p for v in M[r]] for i in range(n): if i != r and M[i][c]: f = M[i][c] M[i] = [(M[i][k] - f*M[r][k]) % p for k in range(m+1)] where[c] = r; r += 1 for i in range(r, n): if M[i][m] % p: return None, 0 # inconsistent x = [0]*m for c in range(m): if where[c] != -1: x[c] = M[where[c]][m] % p return x, pow(p, m - r, p) # number of solutionsExact throughout — no pivoting for stability, no condition number, no rounding. The
wherearray records which row is the pivot for each column, which makes both the particular solution and the free-variable identification straightforward.The returned count p^{m-r} is the answer to "how many solutions" problems. For p=2 this becomes 2^{m-r} and the whole computation can use bitsets (Chapter 46).
- Program
Implement LU decomposition with partial pivoting and use it to solve 100 systems sharing a matrix.
Solution
def lu_decompose(A): n = len(A); LU = [row[:] for row in A]; piv = list(range(n)) for c in range(n): p = max(range(c, n), key=lambda r: abs(LU[r][c])) if p != c: LU[c], LU[p] = LU[p], LU[c]; piv[c], piv[p] = piv[p], piv[c] for r in range(c+1, n): LU[r][c] /= LU[c][c] # store the multiplier in L for k in range(c+1, n): LU[r][k] -= LU[r][c]*LU[c][k] return LU, piv def lu_solve(LU, piv, b): n = len(LU) y = [b[piv[i]] for i in range(n)] for i in range(n): # forward: Ly = Pb for j in range(i): y[i] -= LU[i][j]*y[j] for i in range(n-1, -1, -1): # backward: Ux = y for j in range(i+1, n): y[i] -= LU[i][j]*y[j] y[i] /= LU[i][i] return yStoring L's multipliers in the strictly-lower part of the same array is the standard in-place trick: L's diagonal is known to be all ones, so no space is wasted.
Measure it: one factorisation at n=500 plus 100 solves takes about as long as 1.3 full eliminations, versus 100 for the naive approach. That ratio — O(n³) once plus O(n²) each — is the reason every numerical library exposes a factorisation object rather than only a
solvefunction. - Program
Given a set of XOR equations over 𝔽_2, decide satisfiability and count solutions, using bitset elimination.
Solution
def xor_solve(equations, nvars): """equations: list of (mask, rhs) where mask is a bitmask of variables""" rows = [(m, r) for m, r in equations] basis = {} # pivot bit → (mask, rhs) rank = 0 for m, r in rows: for b in sorted(basis, reverse=True): if m >> b & 1: m ^= basis[b][0]; r ^= basis[b][1] if m: p = m.bit_length() - 1 basis[p] = (m, r); rank += 1 elif r: return None # 0 = 1: inconsistent return pow(2, nvars - rank)Each equation is a single machine word (for n ≤ 64) or a bitset, and elimination is XOR — so the whole O(n³) becomes O(n³/64). At n=2000 that is 1.25×10⁸ word operations, about a second.
The consistency check is the
elif rbranch: a row that reduces to an all-zero mask with a non-zero right-hand side is the equation 0 = 1. Omitting it returns a solution count for an unsatisfiable system — the single most common bug in XOR-elimination code. - Program
Implement exact rational Gaussian elimination and compare its coefficient growth to the Bareiss fraction-free algorithm.
Solution
def bareiss(M): """fraction-free elimination: all intermediates stay INTEGERS""" n = len(M); M = [row[:] for row in M] prev = 1 for k in range(n-1): if M[k][k] == 0: swap = next((i for i in range(k+1, n) if M[i][k]), None) if swap is None: return 0 M[k], M[swap] = M[swap], M[k] for i in range(k+1, n): for j in range(k+1, n): M[i][j] = (M[i][j]*M[k][k] - M[i][k]*M[k][j]) // prev M[i][k] = 0 prev = M[k][k] return M[n-1][n-1] # this IS the determinantThe magic of Bareiss is that the division by
previs always exact — each intermediate is a minor determinant of the original matrix, hence an integer. So there are no fractions at all, and by Hadamard's bound the entries stay within O(nlog(n‖A‖)) bits.Rational elimination on the same input produces fractions whose numerators and denominators can grow exponentially in intermediate steps even though the final answer is small — measure the bit length of the largest intermediate under both to see the difference, which is typically an order of magnitude at n=20.
- Program
Detect whether a large sparse system is better solved directly or iteratively, and implement conjugate gradient for the symmetric positive-definite case.
Solution
def conjugate_gradient(matvec, b, n, tol=1e-10, maxit=None): x = [0.0]*n r = b[:]; p = r[:] rs = sum(v*v for v in r) for it in range(maxit or n): Ap = matvec(p) alpha = rs / sum(pi*api for pi, api in zip(p, Ap)) x = [xi + alpha*pi for xi, pi in zip(x, p)] r = [ri - alpha*api for ri, api in zip(r, Ap)] rs_new = sum(v*v for v in r) if rs_new**0.5 < tol: return x, it p = [ri + (rs_new/rs)*pi for ri, pi in zip(r, p)] rs = rs_new return x, maxitCG needs only matrix–vector products, so it costs O(nnz) per iteration and converges in O(√κlog(1/ε)) iterations — for a well-conditioned sparse system, far cheaper than O(n³).
The decision rule: direct methods suffer fill-in — elimination turns zeros into non-zeros, so a sparse matrix's LU factors may be dense. Estimate the fill (a symbolic factorisation with a good ordering such as AMD does this cheaply); if the factors stay sparse, go direct for exactness and multiple right-hand sides. If they fill in, go iterative — and precondition, because √κ is the term that decides whether it converges in 50 iterations or 50,000.
Determinants, inverses and Cramer's rule
The determinant is the signed volume scaling factor of a linear map. That one interpretation explains multiplicativity, why zero means singular, and why it appears in change of variables, in counting spanning trees, and in geometric orientation tests.
The last two make the computation obvious: run Gaussian elimination, track the sign flips from swaps, and multiply the diagonal. O(n³) instead of the n! of the Leibniz formula — a difference of 10⁶ versus 10^{158} at n=100.
Cofactors, adjugate and Cramer
Cramer's rule is theoretically illuminating and computationally terrible: it needs n+1 determinants, so O(n⁴) against Gaussian elimination's O(n³), and it is numerically far less stable. Use it to prove things (the solution is a rational function of the entries, so it is continuous where \det\ne0), never to compute them. The same applies to A^{-1} = adj(A)/\det(A).
Where determinants actually earn their keep
| Use | Statement |
|---|---|
| Orientation test | \det\begin{smallmatrix}b_x-a_x & c_x-a_x\\ b_y-a_y & c_y-a_y\end{smallmatrix} = twice the signed area (Chapter 54) |
| Matrix-tree theorem | spanning trees = any cofactor of the Laplacian |
| Lindström–Gessel–Viennot | non-intersecting lattice paths = \det of the path-count matrix |
| Change of variables | \int f(u)du = \int f(g(x))|\det Jg|dx |
| Characteristic polynomial | \det(A - λI) = 0 defines the eigenvalues |
| Resultant | two polynomials share a root iff the Sylvester determinant is 0 |
| Cayley–Menger | simplex volume from pairwise distances |
The matrix-tree theorem
Let L = D - A be the graph Laplacian (degree matrix minus adjacency). Then every cofactor of L equals the number of spanning trees. Delete any row and column and take the determinant.
The theorem is remarkable for turning an exponential-looking count into an O(n³) computation. For K_n it gives Cayley's formula n^{n-2} (Chapter 60), and for a weighted graph the same determinant computes the sum over spanning trees of the product of edge weights.
# Bareiss: every intermediate is a minor determinant, hence an integer
def det_bareiss(M):
n = len(M); M = [r[:] for r in M]; sign = 1; prev = 1
for k in range(n-1):
if M[k][k] == 0:
s = next((i for i in range(k+1,n) if M[i][k]), None)
if s is None: return 0
M[k], M[s] = M[s], M[k]; sign = -sign
for i in range(k+1, n):
for j in range(k+1, n):
M[i][j] = (M[i][j]*M[k][k] - M[i][k]*M[k][j]) // prev # exact
prev = M[k][k]
return sign * M[n-1][n-1]"Compute a determinant" expects Gaussian elimination, O(n³), not cofactor expansion at O(n!). The useful follow-up is exactness: over the integers use Bareiss or modular arithmetic plus CRT (Chapter 22), because floating point loses the property that matters most — whether the answer is exactly zero.
- Theory
Prove \det(AB) = \det(A)\det(B) using the volume interpretation.
Solution
\det(M) is the factor by which M scales signed volume: the unit cube maps to the parallelepiped spanned by M's columns, whose signed volume is \det(M).
Applying B then A scales volume first by \det(B) then by \det(A), so the composite scales by the product. Since AB is the matrix of that composite, \det(AB) = \det(A)\det(B). ∎
Consequences that follow immediately: \det(A^{-1}) = 1/\det(A) (inverse maps must undo the scaling); similar matrices have equal determinants (\det(S^{-1}AS) = \det(S)^{-1}\det(A)\det(S)); and a singular matrix has \det = 0 because it flattens the cube into a lower dimension, giving zero volume.
- Theory
Compute the Vandermonde determinant \det(x_i^{j-1}) and state its significance.
Solution
\det V = ∏_{i<j}(x_j - x_i).
Proof: as a polynomial in the x_i, the determinant vanishes whenever two variables coincide (two equal rows), so it is divisible by every (x_j-x_i). Comparing degrees — both sides have total degree \binom n2 — and comparing the coefficient of x_2x_3²··· x_n^{n-1} fixes the constant at 1. ∎
Significance: the determinant is non-zero iff the x_i are distinct, which proves that polynomial interpolation through n distinct points has a unique solution of degree < n — the foundation of Lagrange interpolation (Chapter 49), Reed–Solomon codes, and Shamir secret sharing.
- Theory
State the matrix-tree theorem and use it to derive Cayley's formula.
Solution
Matrix-tree: for a connected graph with Laplacian L = D-A, the number of spanning trees equals any cofactor of L — delete row i and column i and take the determinant, for any i.
For K_n: L = nI - J where J is all-ones. Deleting one row and column gives an (n-1)×(n-1) matrix nI - J, whose eigenvalues are n-(n-1) = 1 (once, on the all-ones vector) and n (with multiplicity n-2).
So the determinant is 1· n^{n-2} = n^{n-2} — Cayley's formula. ∎
The weighted version replaces the adjacency entries by edge weights and computes ∑_{trees}∏_{e∈ T}w_e, which is how one samples spanning trees and computes network reliability polynomials.
- Theory
Prove Cramer's rule and explain why it is impractical.
Solution
Let X_i be the identity with its i-th column replaced by x. Then AX_i is A with its i-th column replaced by Ax = b, i.e. AX_i = A_i. Expanding \det(X_i) along the i-th column gives x_i, so
\det(A)x_i = \det(A_i), hence x_i = \det(A_i)/\det(A). ∎
Impractical because: it needs n+1 determinants at O(n³) each, so O(n⁴) versus O(n³) for elimination. Numerically it is worse still — each determinant is computed independently, so errors do not cancel, and the division by \det(A) is catastrophic for ill-conditioned systems.
Where it is useful: n = 2 or 3 with symbolic entries, and theoretical arguments — it shows the solution is a rational function of the data, hence smooth where the determinant is non-zero, which is what implicit-function-theorem arguments need.
- Theory
State the Lindström–Gessel–Viennot lemma and apply it to count non-intersecting lattice paths.
Solution
LGV: given sources a_1,…,a_n and sinks b_1,…,b_n in a DAG, if every pair of paths from a_i→ b_j and a_k→ b_l with i<k, j>l must intersect, then the number of n-tuples of non-intersecting paths a_i→ b_i equals \det(M) where M_{ij} is the number of paths a_i→ b_j.
Idea of proof: expand the determinant as a signed sum over permutations. Intersecting path systems pair off via a sign-reversing involution — swap the tails after the first intersection — cancelling all terms except the non-intersecting identity-permutation ones. ∎
Application: for sources (0,-i) and sinks (m,-i), the counts are binomials and the determinant gives the number of families of non-crossing paths — which enumerate plane partitions, semistandard Young tableaux, and rhombus tilings of a hexagon.
- Theory
Explain why a floating-point determinant is a poor singularity test, and give the right test.
Solution
Two problems. First, \det(cA) = c^n\det(A), so the determinant's magnitude carries no scale-free information — a well-conditioned matrix scaled by 10^{-2} at n=100 has determinant 10^{-200}, which underflows to zero. Second, rounding means the computed determinant of a genuinely singular matrix is almost never exactly zero.
The right test is the condition number, or equivalently the smallest singular value σ_{min}: the matrix is numerically singular when σ_{min} \lesssim εσ_{max}. Both are scale-invariant in the right way.
Over the integers or modulo a prime the determinant is a valid test, because the arithmetic is exact — which is why exact methods (Bareiss, modular plus CRT) are used whenever the yes/no answer matters, as in geometric predicates (Chapter 54).
- Theory
Prove Hadamard's inequality |\det A| ≤ ∏_i ‖a_i‖_2 and use it to bound the size of an integer determinant.
Solution
The determinant is the volume of the parallelepiped spanned by the rows. A parallelepiped's volume is at most the product of its edge lengths, with equality iff the edges are mutually orthogonal. ∎ (Formally: apply Gram–Schmidt; each orthogonalised vector has norm at most the original.)
Bound for integer matrices: if every entry satisfies |a_{ij}|≤ B then each row has norm ≤ B√n, so
|\det A| ≤ (B√n)^n = B^n n^{n/2}.
At n=50, B=100 that is about 10^{142} — which tells you exactly how many CRT primes are needed for an exact modular computation (Chapter 22 problem 13), and it is what makes that method rigorous rather than merely probable.
- Program
Compute determinants three ways — float elimination, Bareiss, modular plus CRT — and compare on ill-conditioned input.
Solution
def det_float(M): n = len(M); A = [r[:] for r in M]; d = 1.0 for c in range(n): p = max(range(c,n), key=lambda r: abs(A[r][c])) if abs(A[p][c]) < 1e-15: return 0.0 if p != c: A[c], A[p] = A[p], A[c]; d = -d d *= A[c][c] for r in range(c+1, n): f = A[r][c]/A[c][c] for k in range(c, n): A[r][k] -= f*A[c][k] return d def det_mod(M, p): n = len(M); A = [[x % p for x in r] for r in M]; d = 1 for c in range(n): piv = next((r for r in range(c,n) if A[r][c]), None) if piv is None: return 0 if piv != c: A[c], A[piv] = A[piv], A[c]; d = -d d = d * A[c][c] % p inv = pow(A[c][c], p-2, p) for r in range(c+1, n): f = A[r][c]*inv % p A[r] = [(A[r][k] - f*A[c][k]) % p for k in range(n)] return d % pOn a matrix of random 3-digit integers at n=30, the true determinant has ~70 digits. Bareiss and CRT give it exactly; the float version has only 16 significant digits and its relative error can exceed 1 for ill-conditioned inputs.
On the Hilbert matrix the float determinant underflows to 0 by n=15 while the exact value is a tiny but non-zero rational — which is precisely the "is it singular?" failure mode of problem 6.
- Program
Count spanning trees of a graph using the matrix-tree theorem, modulo a prime.
Solution
MOD = 10**9+7 def spanning_trees(n, edges): L = [[0]*n for _ in range(n)] for u, v in edges: L[u][u] += 1; L[v][v] += 1 L[u][v] -= 1; L[v][u] -= 1 # delete the last row and column, take the determinant mod p M = [[L[i][j] % MOD for j in range(n-1)] for i in range(n-1)] return det_mod(M, MOD)O(n³) — so n ≤ 500 is comfortable, which would be an astronomically large count to enumerate directly (a complete graph on 500 vertices has 500^{498} spanning trees).
Validate on K_n: the answer must be n^{n-2} mod p. Validate on a tree: exactly 1. Validate on a cycle C_n: exactly n (remove any one edge). Those three cases exercise the sign conventions and the row/column deletion.
For weighted spanning-tree sums, put -w_{uv} off-diagonal and ∑ w on the diagonal; the same determinant then computes ∑_T∏_{e∈ T}w_e.
- Program
Implement a robust 2D orientation predicate using exact arithmetic, and demonstrate a float version failing.
Solution
def orient_float(a, b, c): return (b[0]-a[0])*(c[1]-a[1]) - (b[1]-a[1])*(c[0]-a[0]) def orient_exact(a, b, c): from fractions import Fraction return orient_float(*[[Fraction(v) for v in p] for p in (a,b,c)]) # a failing case: three nearly-collinear points with large coordinates a = (0.5, 0.5); b = (12.0, 12.0); c = (24.0, 24.0) # perturb c by one ulp and the float sign becomes essentially randomThe determinant (b-a)×(c-a) is a difference of two products of similar magnitude — a classic catastrophic-cancellation shape. For nearly-collinear points the true value is far below the rounding error of the products, so the computed sign is arbitrary.
Consequences: a convex hull algorithm can produce a non-convex output, or crash; a Delaunay triangulation can loop forever. These are real, documented failures.
The fix: Shewchuk's adaptive predicates — compute a cheap float estimate together with an error bound, and fall back to exact arithmetic only when the estimate is within that bound. Exactness where it matters, float speed almost always. This is Chapter 54's territory and it is not optional for production geometry code.
- Program
Implement LGV to count families of non-intersecting lattice paths.
Solution
from math import comb def lgv_paths(sources, sinks): """sources/sinks as (x,y); monotone R/U paths""" n = len(sources) def paths(a, b): dx, dy = b[0]-a[0], b[1]-a[1] return comb(dx+dy, dx) if dx >= 0 and dy >= 0 else 0 M = [[paths(sources[i], sinks[j]) for j in range(n)] for i in range(n)] return det_bareiss(M)The whole algorithm is: build the path-count matrix with binomials, take its determinant. O(n³) plus the binomials, for a quantity whose direct enumeration is astronomically large.
The hypothesis matters: LGV requires that any "crossing" pairing forces an intersection, which holds for the standard staircase arrangement of sources and sinks but not for arbitrary placements. If the sources and sinks are interleaved wrongly, the determinant computes a signed count and can even be negative — a good diagnostic that the configuration violates the hypothesis.
Test on a 2×2 case computable by hand, and on the known plane-partition counts, before trusting it.
- Program
Compute the characteristic polynomial of a matrix and use it to find eigenvalues for small n.
Solution
# Faddeev–LeVerrier: computes char poly coefficients in O(n⁴) with only # matrix multiplications and traces — no symbolic determinant needed. def char_poly(A): n = len(A) M = [[0.0]*n for _ in range(n)] c = [1.0] + [0.0]*n for k in range(1, n+1): M = mat_add(mat_mul(A, M), scalar_mat(c[k-1], n)) AM = mat_mul(A, M) c[k] = -trace(AM)/k return c # p(λ) = λⁿ + c₁λ^{n−1} + ... + cₙO(n⁴). For small n this is fine and gives exact coefficients over the rationals; for large n it is numerically unstable and nobody uses it — eigenvalues are computed by the QR algorithm at O(n³) instead, without ever forming the characteristic polynomial.
Why not root-find the polynomial: polynomial roots are extremely ill-conditioned functions of the coefficients (Wilkinson's polynomial is the classic example — perturbing one coefficient by 2^{-23} moves roots by 10), so going via the characteristic polynomial destroys accuracy even when the eigenvalues themselves are well-conditioned.
Over a finite field, though, the characteristic polynomial is exact and genuinely useful — it is how one computes the order of a linear recurrence or finds the minimal polynomial for Wiedemann's algorithm.
- Program
Verify \det(AB)=\det(A)\det(B) and Hadamard's bound empirically over random integer matrices.
Solution
import random, math for _ in range(200): n = random.randint(1, 7) A = [[random.randint(-9,9) for _ in range(n)] for _ in range(n)] B = [[random.randint(-9,9) for _ in range(n)] for _ in range(n)] AB = [[sum(A[i][k]*B[k][j] for k in range(n)) for j in range(n)] for i in range(n)] assert det_bareiss(AB) == det_bareiss(A)*det_bareiss(B) had = math.prod(math.sqrt(sum(x*x for x in row)) for row in A) assert abs(det_bareiss(A)) <= had + 1e-9Exact integer arithmetic makes the multiplicativity check an exact equality — no tolerance needed — which is a far stronger test than a floating-point comparison would be.
Measuring how tight Hadamard's bound is on random matrices is instructive: the ratio |\det A| / ∏‖a_i‖ is typically around e^{-n/2}, i.e. exponentially small, because random vectors in high dimensions are nearly orthogonal but not exactly so. The bound is tight only for genuinely orthogonal rows (Hadamard matrices), which is where the name comes from.
- Program
Use determinants to test whether four points are concyclic and whether three are collinear, exactly.
Solution
from fractions import Fraction def collinear(a, b, c): return orient_exact(a, b, c) == 0 def in_circle(a, b, c, d): """>0 if d is inside the circle through a,b,c (a,b,c counterclockwise)""" def row(p, q): return [p[0]-q[0], p[1]-q[1], (p[0]-q[0])**2 + (p[1]-q[1])**2] M = [row(a,d), row(b,d), row(c,d)] return det_bareiss(M)The in-circle predicate is a 3×3 determinant after translating d to the origin — the "lifting to the paraboloid" trick, where a circle in the plane becomes a plane in 3D and the sign test becomes an orientation test one dimension up.
These two predicates,
orientandin_circle, are the entire numerical foundation of Delaunay triangulation and Voronoi diagrams. Both must be exact: an inconsistent predicate (saying d is inside for one ordering and outside for another) makes the algorithms produce invalid output or loop forever, and the failures are notoriously hard to debug because they occur only on degenerate or near-degenerate input.With integer coordinates bounded by B, the in-circle determinant is bounded by about B⁴, so 64-bit arithmetic is exact for B ≤ 10^4 and 128-bit for B\le10⁸ — worth computing before choosing a type.
Linear algebra over GF(2): the XOR basis
Over the two-element field, addition is XOR, a vector is a machine word, and Gaussian elimination becomes a handful of XOR operations. Problems about XOR subsets, lights-out puzzles and linear codes all become rank computations that run 64 times faster than their real-valued counterparts.
𝔽_2 = {0,1} with addition modulo 2. Every element is its own additive inverse (x+x=0), so subtraction is addition, and a vector over 𝔽_2^n is exactly an n-bit integer.
The XOR basis (linear basis)
Maintain a set of independent vectors in reduced form, indexed by their highest set bit. Inserting a new vector reduces it against the existing basis; if anything survives, it becomes a new basis element.
struct XorBasis {
static const int B = 60;
long long basis[B] = {0}; // basis[i] has its highest set bit at i
int size = 0;
bool insert(long long x){
for (int i = B-1; i >= 0; i--){
if (!((x >> i) & 1)) continue;
if (!basis[i]) { basis[i] = x; size++; return true; }
x ^= basis[i];
}
return false; // x was already in the span
}
bool contains(long long x){
for (int i = B-1; i >= 0; i--)
if ((x >> i) & 1) x ^= basis[i];
return x == 0;
}
long long maxXor(){ // greedy from the top bit down
long long r = 0;
for (int i = B-1; i >= 0; i--) r = max(r, r ^ basis[i]);
return r;
}
};Insertion is O(log max) — 60 iterations for 64-bit values, regardless of how many vectors have been inserted. That is what makes it practical for streams of millions of values.
Once you have a basis of size d, the set of achievable XOR values is exactly the 2^d elements of its span, and every achievable value is achieved by the same number 2^{n-d} of subsets. So "how many subsets XOR to k" is 2^{n-d} if k is in the span and 0 otherwise — one rank computation answers the whole family of questions.
What it solves
| Problem | Reduction |
|---|---|
| Maximum XOR subset | greedy over the basis from the top bit |
| k-th smallest XOR value | write k in binary over the reduced basis |
| Count subsets with XOR = t | 2^{n-d} if t is in the span, else 0 |
| Lights Out / switch puzzles | solve Ax=b over 𝔽_2 |
| Linear code decoding | syndrome = Hx, a matrix–vector product |
| Cycle space of a graph | dimension |E|-|V|+c; fundamental cycles are a basis |
| Gaussian elimination speed-up | bitset rows, 64× fewer operations |
The cycle space
Take a spanning forest. Each non-tree edge closes exactly one fundamental cycle, and those cycles form a basis of the cycle space over 𝔽_2. Its dimension is the circuit rank m - n + c.
Consequence: the set of achievable XOR values over all cycles is a subspace, so the maximum-XOR path between two vertices is (any path's XOR) combined with the cycle-space basis — a direct application of the structure above.
"Find the maximum XOR of any subset" is the canonical question, and the answer is the 15-line basis above, O(nlog C). The insight to state: the reachable values form a linear subspace, so greedy from the highest bit is optimal — at each bit, either the basis can set it or nothing can, and setting it never costs a lower bit you could otherwise have had.
- Theory
Prove that the set of XOR values achievable from a set S is a linear subspace, and that each achievable value arises from exactly 2^{n-d} subsets.
Solution
The achievable set is {\bigoplus_{i∈ T}s_i : T⊆ S}, which is by definition the span of S over 𝔽_2 — closed under addition (XOR of two subsets is the XOR of their symmetric difference) and under the only scalars, 0 and 1. So it is a subspace of dimension d = rank(S). ∎
For the count: the map T ↦ \bigoplus_{i∈ T}s_i is a linear map 𝔽_2^n → 𝔽_2^{60} with image of dimension d, so by rank–nullity its kernel has dimension n-d. Every non-empty fibre is a coset of the kernel, hence has exactly 2^{n-d} elements. ∎
So: 2^d distinct achievable values, each from 2^{n-d} subsets, and 2^d·2^{n-d} = 2^n ✓ — every subset accounted for.
- Theory
Prove the greedy maximum-XOR algorithm is optimal.
Solution
Put the basis in reduced form: b_i has its highest set bit at position i, and no other basis vector has a 1 at position i.
Process bits from high to low, maintaining a running value r. At position i, if b_i exists and r's bit i is 0, XOR it in — this sets bit i and, because the basis is reduced, changes no higher bit.
Optimality: any achievable value is a XOR of basis vectors, and bit i of the result is determined solely by whether b_i is included (reducedness). So the highest achievable bit can be set independently of everything below it, and greedy takes it. Setting a higher bit always beats any combination of lower bits, since 2^i > ∑_{j<i}2^j. ∎
The same reducedness gives the k-th smallest value: write k-1 in binary and include b_i iff bit i of k-1 is set, reading the basis vectors from lowest to highest.
- Theory
Solve the Lights Out puzzle: formulate it over 𝔽_2 and determine when a configuration is solvable.
Solution
Each button toggles itself and its neighbours. Pressing a button twice is the same as not pressing it, so a strategy is a vector x∈𝔽_2^{n²}, and the effect is linear: Ax = b where A is the toggle matrix and b the initial lit pattern.
Solvable iff b lies in the column space of A. The number of solutions is 2^{\dim N(A)}, so a puzzle with a non-trivial null space has multiple solutions (and the null space vectors are "quiet patterns" that change nothing).
For the standard 5×5 board, rank(A) = 23, so the null space has dimension 2 — there are 4 solutions when solvable, and only 2^{23} of the 2^{25} configurations (one quarter) are solvable at all. ∎
The two quiet patterns are the famous ones every Lights Out solver knows, and they fall out of the elimination for free.
- Theory
Determine the dimension of the cycle space of a graph and describe a basis.
Solution
The cycle space is the set of edge subsets in which every vertex has even degree, viewed as vectors in 𝔽_2^{|E|}. It is the null space of the incidence matrix over 𝔽_2.
The incidence matrix of a graph with c components has rank n-c, so by rank–nullity the cycle space has dimension
m - n + c — the circuit rank.
Basis: take a spanning forest; each of the m-n+c non-tree edges e closes a unique cycle with the forest path between its endpoints, and these fundamental cycles are independent (each contains an edge no other contains) and span. ∎
Application: the maximum-XOR path from s to t equals any single path's XOR combined with the XOR basis of the fundamental cycles — because two s–t paths differ by an element of the cycle space.
- Theory
Explain how a linear error-correcting code works over 𝔽_2, using Hamming(7,4) as the example.
Solution
A linear code is a subspace C⊆𝔽_2^n of dimension k, specified by a generator matrix G (rows span C) or a parity-check matrix H (with C = N(H)).
Hamming(7,4): n=7, k=4, and H's columns are the seven non-zero vectors of 𝔽_2^3. The syndrome of a received word y is Hy; if y = c + e with a single-bit error at position i, then Hy = He = column i of H — which, read as a binary number, is the error position.
Minimum distance 3, so it corrects any single error and detects any double error. ∎
The general statement: a code with minimum distance d corrects ⌊(d-1)/2⌋ errors, and for a linear code the minimum distance equals the minimum weight of a non-zero codeword — a fact that makes it computable by searching the span rather than all pairs.
- Theory
How many n× n invertible matrices are there over 𝔽_2?
Solution
Build the matrix row by row, requiring each new row to be outside the span of the previous ones. Row 1 can be any non-zero vector: 2^n-1 choices. Row 2 must avoid the span of row 1 (size 2): 2^n-2. Row i must avoid a span of size 2^{i-1}:
|GL_n(𝔽_2)| = ∏_{i=0}^{n-1}(2^n - 2^i).
For n=1,2,3: 1, 6, 168. ∎
The fraction of all 2^{n²} matrices that are invertible is ∏_{i=1}^{n}(1-2^{-i}) → 0.2888 — so a random binary matrix is invertible about 29% of the time, independent of n. That constant is why randomised algorithms that need an invertible matrix simply retry: three attempts suffice on average.
- Theory
Given n numbers, count the subsets whose XOR is zero, and explain why the answer is a power of two.
Solution
Build the XOR basis; let its size be d. By problem 1, the subsets XOR-ing to any particular achievable value number exactly 2^{n-d}, and 0 is always achievable (the empty subset).
So the answer is 2^{n-d}, including the empty subset. ∎
It is a power of two because the set of such subsets is precisely the kernel of the linear map T↦\bigoplus_{i∈ T}s_i — a subspace of 𝔽_2^n, hence of size 2^{\dim}.
Sanity check: if all n numbers are independent (d=n) the answer is 1, the empty subset only. If all are equal to some v\ne0 then d=1 and the answer is 2^{n-1} — every even-sized subset, and there are indeed 2^{n-1} of those ✓.
- Program
Implement the XOR basis with insert, contains, maximum, minimum and k-th smallest.
Solution
class XorBasis: def __init__(self, bits=60): self.bits = bits; self.b = [0]*bits; self.n = 0; self.cnt = 0 def insert(self, x): self.n += 1 for i in range(self.bits-1, -1, -1): if not (x >> i) & 1: continue if not self.b[i]: self.b[i] = x; self.cnt += 1; return True x ^= self.b[i] return False # x is in the span; zero is representable def reduce_full(self): # make it REDUCED row echelon for i in range(self.bits): if not self.b[i]: continue for j in range(i+1, self.bits): if self.b[j] >> i & 1: self.b[j] ^= self.b[i] def kth(self, k): # k is 1-indexed among distinct values self.reduce_full() vals = [v for v in self.b if v] if self.cnt < self.n: k -= 1 # zero is achievable if k >= 1 << len(vals): return None r = 0 for i, v in enumerate(vals): if k >> i & 1: r ^= v return rThe
reduce_fullstep is what makeskthcorrect: only in reduced form does bit i of the result depend solely on whether b_i is included, which is what lets the binary digits of k select basis elements independently.The
cnt < nadjustment handles whether 0 is achievable by a non-empty subset — an off-by-one that most implementations get wrong on the first try. - Program
Find the maximum XOR path between two vertices in an undirected weighted graph.
Solution
def max_xor_path(n, edges, s, t): adj = [[] for _ in range(n)] for u, v, w in edges: adj[u].append((v,w)); adj[v].append((u,w)) dist = [None]*n; dist[s] = 0 basis = XorBasis() stack = [s] while stack: u = stack.pop() for v, w in adj[u]: if dist[v] is None: dist[v] = dist[u] ^ w; stack.append(v) else: basis.insert(dist[u] ^ w ^ dist[v]) # a fundamental cycle if dist[t] is None: return None r = dist[t] for i in range(basis.bits-1, -1, -1): r = max(r, r ^ basis.b[i]) return rO((n+m)log C). The DFS assigns each vertex the XOR along the tree path from s; every non-tree edge (u,v,w) then contributes the fundamental cycle value d_u ⊕ w ⊕ d_v.
Why it is correct: any s–t walk's XOR equals d_t XOR some element of the cycle space (problem 4), and conversely every such value is achievable by a walk. So the answer is the maximum of d_t ⊕ (span of the cycle basis), which the greedy computes.
Note it is walks, not simple paths — but repeating an edge contributes its weight twice, which XORs to nothing, so the maximum is always attained by a walk that is effectively a path plus some cycles.
- Program
Solve a Lights Out puzzle of size n× n for n\le20, reporting all solutions.
Solution
def lights_out(n, initial): N = n*n rows = [] for i in range(n): for j in range(n): m = 0 for di, dj in ((0,0),(1,0),(-1,0),(0,1),(0,-1)): x, y = i+di, j+dj if 0 <= x < n and 0 <= y < n: m |= 1 << (x*n+y) rows.append(m) # solve A x = b over F2, with A's COLUMN j = rows[j] (A is symmetric here) aug = [(rows[j], (initial >> j) & 1) for j in range(N)] ... # Gaussian elimination with bitmask rowsWith N = n² ≤ 400 variables, bitmask elimination over 400-bit integers costs O(N³/64) = 10⁶ word operations — instant.
The output should report the rank, whether the system is consistent, one particular solution, and a basis for the null space (the quiet patterns). For 5×5 expect rank 23 and two quiet patterns; for n=4 the matrix is invertible so every configuration has a unique solution.
The pattern of which n give singular matrices is itself interesting — it relates to whether certain Chebyshev-like polynomials over 𝔽_2 share a factor, and it is why some board sizes are "always solvable" and others are not.
- Program
Speed up Gaussian elimination over 𝔽_2 using bitsets and measure the 64× improvement.
Solution
# each row is a single Python integer used as a bitset def rank_gf2(rows, ncols): rank = 0 pivot_row = {} for r in rows: cur = r for c in sorted(pivot_row, reverse=True): if cur >> c & 1: cur ^= pivot_row[c] if cur: pivot_row[cur.bit_length()-1] = cur rank += 1 return rankEach XOR of two n-bit rows is O(n/64) word operations rather than O(n) scalar ones. At n=4096: scalar elimination is 6.9×10^{10} operations (minutes), bitset elimination is 10⁹ word operations (about a second).
In C++ use
std::bitset<N>for compile-time-known N, orvector<uint64_t>otherwise; both compile to the same word-parallel XOR loop. Python's arbitrary-precision integers are already word-parallel internally, which is why the code above is competitive despite being interpreted.This 64× is the main reason 𝔽_2 problems admit much larger inputs than their real-valued analogues — n=5000 is routine over 𝔽_2 and demanding over ℝ.
- Program
Implement Hamming(7,4) encoding and single-error correction, and verify over all error patterns.
Solution
H = [0b1010101, 0b0110011, 0b0001111] # parity-check rows def syndrome(y): s = 0 for i, h in enumerate(H): s |= (bin(y & h).count('1') & 1) << (len(H)-1-i) return s # 0 = no error, else the position def correct(y): s = syndrome(y) return y if s == 0 else y ^ (1 << (7 - s)) # exhaustive verification over all codewords and all single-bit errors for c in codewords(): # the 16 valid codewords assert syndrome(c) == 0 for bit in range(7): assert correct(c ^ (1 << bit)) == cThe exhaustive check over 16×7 = 112 cases is a complete proof for this code — a finite domain, so enumeration is verification (Chapter 2).
Also verify that double errors are detected but mis-corrected: correct(c ⊕ e_i ⊕ e_j) ≠ c, which is the expected behaviour for a distance-3 code and the reason SECDED codes add an overall parity bit to reach distance 4.
The bit-position-as-syndrome property comes from ordering H's columns as the binary numerals 1..7 — an elegant design that makes decoding a table-free computation.
- Program
Count the subsets of a given set with XOR equal to each possible value, using the basis, in O(nlog C).
Solution
MOD = 10**9+7 def count_by_xor(nums): b = XorBasis() for x in nums: b.insert(x) d = b.cnt; n = len(nums) multiplicity = pow(2, n - d, MOD) # every achievable value, equally b.reduce_full() vals = [v for v in b.b if v] span = [] for mask in range(1 << d): # only feasible for small d v = 0 for i in range(d): if mask >> i & 1: v ^= vals[i] span.append(v) return {v: multiplicity for v in span}The key structural fact is that the multiplicity is the same for every achievable value — so you never need a per-value count, just the rank. Enumerating the span is only needed if the caller wants the actual values, and that is 2^d, feasible for d\le25.
For a query of the form "how many subsets XOR to t", no enumeration is needed at all:
multiplicity if basis.contains(t) else 0, in O(log C). That is the version to implement, and it handles n = 10⁶ with 60-bit values without difficulty. - Program
Implement an offline structure supporting "insert a value" and "query the maximum XOR with a given x over a suffix of insertions".
Solution
# Time-aware basis: keep, with each basis vector, the LATEST insertion time # that can produce it. Then a query over [l, n] uses only vectors with time ≥ l. class TimedBasis: def __init__(self, bits=30): self.bits = bits; self.b = [0]*bits; self.t = [-1]*bits def insert(self, x, time): for i in range(self.bits-1, -1, -1): if not (x >> i) & 1: continue if not self.b[i]: self.b[i] = x; self.t[i] = time; return if self.t[i] < time: # prefer the newer vector self.b[i], x = x, self.b[i] self.t[i], time = time, self.t[i] x ^= self.b[i] def max_xor_from(self, x, l): r = x for i in range(self.bits-1, -1, -1): if self.t[i] >= l and (r ^ self.b[i]) > r: r ^= self.b[i] return rThe trick is the swap: when inserting a newer vector that collides with an existing pivot, keep the newer one in the basis and push the older one down. This maintains the invariant that each pivot holds the latest-possible vector, so filtering by t_i ≥ l yields a valid basis for the suffix.
O(log C) per operation, no persistence structure needed. It answers the standard "maximum XOR of x with any element of a suffix" problem, and it generalises to arbitrary ranges only with a persistent trie or an offline divide-and-conquer — the suffix case is the one with this elegant solution.
Eigenvalues, diagonalisation and matrix powers
An eigenvector is a direction the matrix merely stretches. Finding a basis of them turns a matrix into a diagonal one, which makes powers, exponentials and long-run behaviour trivial - and explains why some iterations converge and others explode.
Not every matrix is diagonalisable — \begin{smallmatrix}1&1\\0&1\end{smallmatrix} has the single eigenvalue 1 with only a one-dimensional eigenspace. Such defective matrices need the Jordan form, and their powers pick up polynomial factors: \begin{smallmatrix}1&n\\0&1\end{smallmatrix}.
Why eigenvalues predict behaviour
Write x_0 in the eigenbasis: x_0 = ∑ c_iv_i. Then A^kx_0 = ∑ c_iλ_i^kv_i, and the term with the largest |λ_i| dominates.
| Spectral condition | Behaviour of A^k | Where it appears |
|---|---|---|
| all |λ| < 1 | decays to 0 | convergent iteration |
| λ_1 = 1, rest smaller | converges to a fixed vector | Markov chains, PageRank |
| some |λ| > 1 | blows up | unstable recurrence |
| |λ| = 1 complex | oscillates forever | periodic chains |
| gap 1 - |λ_2| | convergence rate | mixing time (Chapter 41) |
The spectral radius ρ(A) = max|λ_i| decides stability, and the spectral gap decides speed. An iteration x_{k+1} = Ax_k converges iff ρ(A)<1, at rate ρ(A)^k — which is why the damping factor in PageRank (Chapter 41) directly sets the iteration count.
The Perron–Frobenius theorem
For a matrix with strictly positive entries: there is a unique largest eigenvalue λ_1 > 0, it is simple, and its eigenvector can be taken strictly positive. Every other eigenvalue is strictly smaller in modulus.
This is what guarantees a unique stationary distribution for an irreducible aperiodic Markov chain, why PageRank has a unique answer, and why the dominant growth rate of any non-negative linear recurrence is a single real number.
Singular values and the SVD
The SVD is the numerically stable answer to almost every question that eigenvalues answer for non-symmetric or non-square matrices: rank, condition number, least squares, low-rank approximation, principal components, and pseudoinverse.
"Compute the n-th Fibonacci number in O(log n)" is matrix exponentiation; the eigenvalue version explains why it works and produces Binet's formula. Saying "the eigenvalues of \begin{smallmatrix}1&1\\1&0\end{smallmatrix} are φ and ψ, so F_n = (φ^n-ψ^n)/√5, and since |ψ|<1 the answer is round(φ^n/√5)" connects three chapters in one breath.
- Theory
Find the eigenvalues and eigenvectors of the Fibonacci matrix and derive Binet's formula.
Solution
A = \begin{smallmatrix}1&1\\1&0\end{smallmatrix}. Characteristic polynomial: λ²-λ-1=0, giving λ = (1±√5)/(2), i.e. φ≈1.618 and ψ≈-0.618.
Eigenvectors: (φ,1) and (ψ,1). Diagonalising and reading off the (1,2) entry of A^n:
F_n = (φ^n - ψ^n)/(√5). ∎
Since |ψ| = 0.618 < 1, the second term tends to 0, so F_n = round(φ^n/√5) for all n\ge0. Note this is exact mathematically and fails numerically past n≈71 in double precision (Chapter 3) — the formula is correct, the arithmetic is not.
The characteristic polynomial being the recurrence is no coincidence: it is the general fact that a linear recurrence's companion matrix has the recurrence as its characteristic polynomial.
- Theory
Prove that eigenvectors for distinct eigenvalues are linearly independent.
Solution
Suppose not, and take a minimal dependent set ∑_{i=1}^{k}c_iv_i = 0 with all c_i\ne0 and distinct eigenvalues.
Apply A: ∑ c_iλ_iv_i = 0. Also multiply the original by λ_k: ∑ c_iλ_kv_i = 0. Subtracting,
∑_{i=1}^{k-1}c_i(λ_i-λ_k)v_i = 0,
a shorter dependency with non-zero coefficients (the λ are distinct) — contradicting minimality. ∎
Corollary: a matrix with n distinct eigenvalues has n independent eigenvectors and is therefore diagonalisable. Repeated eigenvalues may still be fine (the identity is diagonal) but are the only way defectiveness can arise.
- Theory
Show that a real symmetric matrix has real eigenvalues and orthogonal eigenvectors.
Solution
Real eigenvalues: let Av = λv with v possibly complex. Then \bar{v}^TAv = λ\bar{v}^Tv = λ‖v‖². Taking the conjugate transpose of the left side and using A = A^T real, \bar{v^TAv} = v^TA\bar{v} = \bar{λ}‖v‖². But \bar{v}^TAv is a scalar equal to its own conjugate, so λ = \bar{λ} is real. ∎
Orthogonality: for Av_1 = λ_1v_1 and Av_2 = λ_2v_2 with λ_1≠λ_2,
λ_1v_1^Tv_2 = (Av_1)^Tv_2 = v_1^TAv_2 = λ_2v_1^Tv_2,
so (λ_1-λ_2)v_1^Tv_2 = 0 and the eigenvectors are orthogonal. ∎
This is why covariance matrices, Laplacians and Gram matrices — all symmetric — have such well-behaved spectra, and why PCA produces orthogonal components automatically rather than by construction.
- Theory
Give a non-diagonalisable matrix and describe how its powers behave.
Solution
J = \begin{smallmatrix}λ&1\\0&λ\end{smallmatrix} has the single eigenvalue λ with algebraic multiplicity 2 but geometric multiplicity 1 — the eigenspace is only span{(1,0)}.
Its powers are
J^n = \begin{smallmatrix}λ^n & nλ^{n-1}\\0&λ^n\end{smallmatrix},
so the growth is nλ^n rather than λ^n — an extra polynomial factor. ∎
This is exactly why a linear recurrence with a repeated root r has solutions r^n and nr^n: the companion matrix is defective, and the Jordan block produces the polynomial factor. A root of multiplicity m contributes r^n, nr^n, …, n^{m-1}r^n.
Numerically, defective matrices are the hard case: eigenvalues of a Jordan block are ill-conditioned, perturbing the matrix by ε moves them by ε^{1/m}.
- Theory
State Perron–Frobenius and explain its role in PageRank.
Solution
Perron–Frobenius (positive case): if A > 0 entrywise, then the spectral radius ρ(A) is itself an eigenvalue, it is simple, its eigenvector can be chosen strictly positive, and every other eigenvalue satisfies |λ| < ρ(A).
For a stochastic matrix, ρ = 1 (rows sum to 1, so \mathbf{1} is a right eigenvector with eigenvalue 1), and the theorem gives a unique positive left eigenvector — the stationary distribution.
In PageRank: the damped matrix dP + (1-d)/(n)J is strictly positive, so Perron–Frobenius applies directly: a unique positive stationary vector exists, and |λ_2| ≤ d so power iteration converges geometrically at rate d. Without damping the web matrix is neither positive nor irreducible and none of this holds. ∎
The theorem also gives the growth rate of any non-negative linear recurrence — the dominant root is real and positive, which is why such sequences grow smoothly rather than oscillating.
- Theory
Explain the SVD and the Eckart–Young theorem on best low-rank approximation.
Solution
SVD: A = UΣ V^T with U, V orthogonal and Σ diagonal with σ_1≥σ_2≥···\ge0. Geometrically: rotate, scale the axes, rotate — every linear map is that.
Eckart–Young: truncating to the top k singular values, A_k = ∑_{i=1}^{k}σ_iu_iv_i^T, minimises ‖A - B‖ over all rank-k matrices B, in both the spectral and Frobenius norms, with error σ_{k+1} and √{∑_{i>k}σ_i²} respectively. ∎
Applications: PCA (the top singular vectors of the centred data matrix), latent semantic analysis, image compression, and noise reduction — in each case the claim is that the signal lives in a low-dimensional subspace and the small singular values are noise.
The practical caveat: the SVD costs O(mnmin(m,n)), so for large sparse matrices use a truncated method (Lanczos, randomised SVD) that computes only the top k in O(nnz· k).
- Theory
Given the eigenvalues of a graph's adjacency matrix, what can you deduce?
Solution
For a graph on n vertices with adjacency eigenvalues λ_1≥···≥λ_n:
- ∑λ_i = tr(A) = 0 (no self-loops).
- ∑λ_i² = tr(A²) = 2|E|.
- ∑λ_i³ = tr(A³) = 6×(triangles).
- λ_1 lies between the average and maximum degree; it equals the degree iff the graph is regular.
- The graph is bipartite iff the spectrum is symmetric about 0, i.e. λ_n = -λ_1.
- The number of connected components equals the multiplicity of eigenvalue 0 in the Laplacian.
- The second-smallest Laplacian eigenvalue (algebraic connectivity, or Fiedler value) bounds the graph's expansion — the basis of spectral clustering.
Two non-isomorphic graphs can share a spectrum (cospectral graphs), so the spectrum does not determine the graph — but it determines a great deal, cheaply, and spectral methods exploit exactly that.
- Program
Implement the power method to find the dominant eigenvalue and eigenvector, and measure the convergence rate.
Solution
def power_method(matvec, n, iters=1000, tol=1e-12): import random, math v = [random.gauss(0,1) for _ in range(n)] norm = math.sqrt(sum(x*x for x in v)); v = [x/norm for x in v] lam = 0.0 for it in range(iters): w = matvec(v) nw = math.sqrt(sum(x*x for x in w)) if nw == 0: return 0.0, v, it w = [x/nw for x in w] new_lam = sum(a*b for a, b in zip(v, matvec(v))) # Rayleigh quotient if abs(new_lam - lam) < tol: return new_lam, w, it lam = new_lam; v = w return lam, v, itersConvergence is at rate |λ_2/λ_1|^k, so the iteration count is log(tol)/log|λ_2/λ_1|. Measure it on matrices with controlled spectra: a ratio of 0.5 needs ~40 iterations, 0.99 needs ~2700.
The Rayleigh quotient v^TAv/v^Tv converges quadratically for symmetric matrices even though v converges only linearly — so use it for the eigenvalue estimate rather than the ratio of successive norms.
Failure modes: a random start has zero probability of being orthogonal to the dominant eigenvector, but a structured start (all ones, say) may be — which is why random initialisation matters. And if |λ_1| = |λ_2| (a complex pair, or ±λ), the iteration oscillates instead of converging.
- Program
Compute eigenvalues of a symmetric matrix by the Jacobi rotation method and verify against the trace and determinant.
Solution
import math def jacobi_eigen(A, iters=100, tol=1e-12): n = len(A); A = [r[:] for r in A] V = [[float(i==j) for j in range(n)] for i in range(n)] for _ in range(iters): # find the largest off-diagonal entry p, q, best = 0, 1, 0.0 for i in range(n): for j in range(i+1, n): if abs(A[i][j]) > best: best, p, q = abs(A[i][j]), i, j if best < tol: break theta = 0.5*math.atan2(2*A[p][q], A[q][q]-A[p][p]) c, s = math.cos(theta), math.sin(theta) for k in range(n): akp, akq = A[k][p], A[k][q] A[k][p] = c*akp - s*akq; A[k][q] = s*akp + c*akq for k in range(n): apk, aqk = A[p][k], A[q][k] A[p][k] = c*apk - s*aqk; A[q][k] = s*apk + c*aqk for k in range(n): vkp, vkq = V[k][p], V[k][q] V[k][p] = c*vkp - s*vkq; V[k][q] = s*vkp + c*vkq return [A[i][i] for i in range(n)], VEach rotation zeroes one off-diagonal pair; subsequent rotations partially undo it, but the sum of squares of the off-diagonal entries strictly decreases, guaranteeing convergence. It is O(n³) per sweep and typically needs 6–10 sweeps.
Verification: ∑λ_i must equal the original trace and ∏λ_i the determinant. Both are invariants of the orthogonal similarity transformations being applied, so any deviation beyond rounding indicates a bug in the rotation formulas — which is exactly where these implementations go wrong.
Jacobi is slower than the QR algorithm but is simple, highly accurate for small eigenvalues, and trivially parallel — it is still used where accuracy matters more than speed.
- Program
Use eigenvalues to determine whether an iterative process converges, and find the convergence rate.
Solution
def will_converge(A, tol=1e-9): """x_{k+1} = A x_k converges to 0 iff the spectral radius < 1""" lam, _, _ = power_method(lambda v: matvec(A, v), len(A)) rho = abs(lam) if rho >= 1 - tol: return False, rho, None iters = math.log(tol) / math.log(rho) return True, rho, math.ceil(iters)For a fixed-point iteration x_{k+1} = Ax_k + b, the error satisfies e_{k+1} = Ae_k, so convergence is governed entirely by ρ(A).
Applications: Jacobi and Gauss–Seidel iteration for linear systems converge iff the spectral radius of their iteration matrices is below 1 — guaranteed for diagonally dominant matrices, which is why that condition appears everywhere in numerical analysis. Value iteration in a discounted MDP converges at rate γ, the discount factor, because the Bellman operator is a γ-contraction.
Caveat: ρ(A)<1 guarantees eventual convergence but the transient can be large for a non-normal matrix — ‖A^k‖ can grow enormously before decaying, which is why the pseudospectrum matters in fluid-dynamics stability analysis.
- Program
Implement PCA via the SVD on a dataset and interpret the explained variance.
Solution
import numpy as np def pca(X, k): X = np.asarray(X, dtype=float) mu = X.mean(axis=0) Xc = X - mu # centring is ESSENTIAL U, S, Vt = np.linalg.svd(Xc, full_matrices=False) components = Vt[:k] # principal directions scores = Xc @ components.T # projected data explained = (S**2) / (S**2).sum() return components, scores, explained[:k], muCentring is not optional: without it the first component points at the data's mean rather than its direction of greatest variance, and the whole decomposition is meaningless. Whether to also scale each feature to unit variance depends on whether the features share units — for mixed units, always scale.
The explained-variance ratios σ_i²/∑σ_j² say how much structure each component captures; a "knee" in that curve is the conventional (if unprincipled) way to choose k.
Computing PCA via
eig(X.T @ X)rather than the SVD squares the condition number (Chapter 44) and loses accuracy for the small components — use the SVD directly, always. - Program
Compute the number of walks of length n in a graph using eigen-decomposition rather than matrix powers, and compare accuracy.
Solution
import numpy as np def walks_eigen(A, n): w, V = np.linalg.eigh(A) # A symmetric return V @ np.diag(w**n) @ V.T def walks_power(A, n): return np.linalg.matrix_power(A, n)The eigen version is O(n³) once plus O(n) per exponent — better if you need many different exponents. The power version is O(n³log k) per exponent but stays in exact integer arithmetic if you use Python integers.
Accuracy: λ^n overflows a
doubleonce nlogλ > 709, so for λ=3 that is n≈645. Below that, the eigen result agrees with the exact integer count to about 12 digits; above it, it returnsinf.The lesson: eigen-decomposition is the right tool for asymptotics and for real-valued quantities, and exact integer or modular exponentiation is the right tool for counting. Using floats to count is how you get walk counts that are off by one in the last digit and therefore useless.
- Program
Implement spectral clustering using the Fiedler vector of the graph Laplacian.
Solution
import numpy as np def spectral_bisect(adj_matrix): A = np.asarray(adj_matrix, dtype=float) D = np.diag(A.sum(axis=1)) L = D - A w, V = np.linalg.eigh(L) # symmetric ⟹ real, orthogonal fiedler = V[:, 1] # eigenvector of the 2nd smallest return (fiedler > 0).astype(int) # sign gives the bipartitionThe smallest Laplacian eigenvalue is always 0 with eigenvector \mathbf{1}; the second-smallest, the Fiedler value, measures algebraic connectivity, and its eigenvector's sign pattern gives a good bipartition.
Why it works: minimising the cut subject to a balance constraint is NP-hard, but relaxing the ±1 labels to real numbers turns it into minimising x^TLx subject to x\perp\mathbf{1} and ‖x‖=1 — whose solution is exactly the Fiedler vector, by the Rayleigh quotient characterisation of eigenvalues.
Cheeger's inequality bounds how far the relaxation can be from optimal: (λ_2)/(2) ≤ h(G) ≤ √{2λ_2} where h is the conductance. So the spectral method is provably within a square-root factor — one of the cleanest approximation guarantees in the subject.
- Program
Verify the spectral theorem numerically and demonstrate its failure for a non-symmetric matrix.
Solution
import numpy as np def check_spectral(A, tol=1e-10): A = np.asarray(A, dtype=float) w, V = np.linalg.eig(A) reconstruct_err = np.abs(V @ np.diag(w) @ np.linalg.inv(V) - A).max() orthogonality = np.abs(V.T @ V - np.eye(len(A))).max() real_eigs = np.abs(w.imag).max() return reconstruct_err, orthogonality, real_eigs S = np.array([[2.,1.],[1.,3.]]) # symmetric N = np.array([[0.,1.],[-1.,0.]]) # rotation: NOT symmetric D = np.array([[1.,1.],[0.,1.]]) # defectiveFor the symmetric S: reconstruction error ~10^{-15}, orthogonality ~10^{-16}, imaginary parts 0 — all three properties hold.
For the rotation N: eigenvalues are ± i, so the real-eigenvalue check fails immediately. It is still diagonalisable, but only over C.
For the defective D:
np.linalg.eigreturns two identical eigenvalues and two nearly identical eigenvectors, so V is nearly singular and the reconstruction error explodes. That is the practical signature of defectiveness — a huge condition number for the eigenvector matrix — and it is why numerical code should use the Schur decomposition (always exists, orthogonal) rather than diagonalisation when the matrix is not known to be symmetric.
Matrix exponentiation for recurrences and DP
If a DP's state is a fixed-size vector and its transition is linear, the transition is a matrix and n steps cost O(k^3 log n) instead of O(nk). This one technique converts an enormous family of "n up to 10^18" problems into logarithmic time.
The recipe is mechanical. Identify a state vector that is sufficient to compute the next one, write the transition as a matrix, and exponentiate (Chapter 21).
a_n = c₁a_{n−1} + c₂a_{n−2} + ... + c_k a_{n−k}
⎡a_n ⎤ ⎡c₁ c₂ c₃ ... c_k⎤ ⎡a_{n−1}⎤
⎢a_{n−1}⎥ ⎢1 0 0 ... 0 ⎥ ⎢a_{n−2}⎥
⎢a_{n−2}⎥ = ⎢0 1 0 ... 0 ⎥ ⎢a_{n−3}⎥
⎢ ... ⎥ ⎢ ... ⎥ ⎢ ... ⎥
⎣a_{n−k+1}⎦ ⎣0 0 ... 1 0 ⎦ ⎣a_{n−k}⎦
v_n = M · v_{n−1} ⟹ v_n = Mⁿ⁻ᵏ⁺¹ · v_{k−1}
Cost: O(k³ log n). The characteristic polynomial of M is exactly
λᵏ − c₁λᵏ⁻¹ − ... − c_k, so eigenvalues = roots of the recurrence.Handling the awkward cases
| Recurrence | Trick |
|---|---|
| a_n = a_{n-1}+a_{n-2}+1 | append a constant 1 to the state vector |
| a_n = a_{n-1}+n | append n and 1; n updates as n+1 |
| a_n = a_{n-1}+n² | append n², n, 1 — a degree-d polynomial needs d+1 slots |
| prefix sums S_n = ∑ a_i | append S_{n-1} to the state |
| alternating signs | append (-1)^n |
| period-p coefficients | exponentiate the product of p matrices |
Anything that is a polynomial in n or a fixed linear function of earlier terms can be absorbed into the state vector. If the inhomogeneous part is p(n)r^n for a polynomial p of degree d, you need d+1 extra state slots. If it is genuinely non-linear (a_{n-1}², or a_{⌊ n/2⌋}), matrix exponentiation does not apply.
DP transitions as matrices
A DP over a bounded state space with a transition independent of the step index is a matrix power. The classic instances:
When k is large
O(k³log n) becomes prohibitive past k ≈ 300. Two better algorithms compute the same thing:
- Kitamasa: work in ℤ_p[x]/f(x) where f is the characteristic polynomial. Computing x^n mod f costs O(k²log n), or O(klog klog n) with NTT.
- Bostan–Mori: the same bound with a smaller constant, working directly on the generating function P(x)/Q(x) (Chapter 35).
"Compute F_n for n = 10^{18}" expects matrix exponentiation. The follow-up worth pre-empting: for Fibonacci specifically, fast doubling uses 3 multiplications per bit against the matrix method's 8, and for a general order-k recurrence Kitamasa beats the matrix by a factor of k. Knowing that the matrix method is the general answer but rarely the best one is the distinguishing detail.
- Theory
Write the transition matrix for a_n = 2a_{n-1} + 3a_{n-2} - a_{n-3} and state its characteristic polynomial.
Solution
⎡a_n ⎤ ⎡2 3 −1⎤ ⎡a_{n−1}⎤ ⎢a_{n−1}⎥ = ⎢1 0 0⎥ ⎢a_{n−2}⎥ ⎣a_{n−2}⎦ ⎣0 1 0⎦ ⎣a_{n−3}⎦The characteristic polynomial is λ³ - 2λ² - 3λ + 1 — the recurrence with a_{n-i} replaced by λ^{k-i}.
That correspondence is exactly why the companion matrix is called that: it is constructed so its characteristic polynomial is the given one. Consequently the eigenvalues are the roots of the recurrence, and the general solution is ∑ c_ir_i^n — the connection between the matrix method and the closed-form method of Chapter 35.
- Theory
Handle the inhomogeneous recurrence a_n = a_{n-1} + a_{n-2} + n + 1 by augmenting the state.
Solution
The state must carry enough to reproduce the inhomogeneous term. Take v_n = (a_n, a_{n-1}, n, 1)^T:
⎡a_n ⎤ ⎡1 1 1 1⎤ ⎡a_{n−1}⎤ ⎢a_{n−1}⎥ = ⎢1 0 0 0⎥ ⎢a_{n−2}⎥ ⎢ n ⎥ ⎢0 0 1 1⎥ ⎢ n−1 ⎥ ⎣ 1 ⎦ ⎣0 0 0 1⎦ ⎣ 1 ⎦Row 3 implements n = (n-1) + 1, and row 4 keeps the constant. Row 1 then reads a_n = a_{n-1}+a_{n-2}+(n-1)+1 = a_{n-1}+a_{n-2}+n — adjust the constant column to 2 if the intended term is n+1.
Getting these off-by-ones right is the fiddly part, and the reliable method is to verify against a brute-force loop for n ≤ 20 before trusting the matrix. For a degree-d polynomial term, extend the state with n^d, …, n, 1 and use the binomial expansion of (n-1+1)^j to fill in the rows.
- Theory
Explain how to compute the prefix sum S_n = ∑_{i=0}^{n}a_i of a linear recurrence in O(k³log n).
Solution
Append S to the state vector: v_n = (a_n, a_{n-1}, …, a_{n-k+1}, S_n)^T, and add the row
S_n = S_{n-1} + a_n = S_{n-1} + c_1a_{n-1} + ··· + c_ka_{n-k},
which is a linear combination of the previous state — so the augmented matrix is (k+1)×(k+1) and everything else is unchanged. ∎
The same idea handles second-order sums (∑ S_i) by appending another slot, and weighted sums ∑ i· a_i by appending both ∑ a_i and the running index.
This is a genuinely useful pattern: many contest problems ask for the sum of the first n terms of a recurrence, and the answer is one extra row rather than a separate derivation.
- Theory
Derive the matrix for counting binary strings of length n with no three consecutive 1s, and give the answer for n = 10^{18} modulo 10⁹+7.
Solution
State: the number of trailing 1s, in {0,1,2}.
⎡x₀'⎤ ⎡1 1 1⎤ ⎡x₀⎤ append 0 from any state → state 0 ⎢x₁'⎥ = ⎢1 0 0⎥ ⎢x₁⎥ append 1 from state 0 → state 1 ⎣x₂'⎦ ⎣0 1 0⎦ ⎣x₂⎦ append 1 from state 1 → state 2Start from (1,0,0) (the empty string) and the answer after n steps is the sum of the entries.
The characteristic polynomial is λ³-λ²-λ-1 — the tribonacci recurrence, matching Chapter 26 problem 7 ✓.
At n=10^{18}: 60 squarings of a 3×3 matrix, about 3,000 modular multiplications — microseconds. The general pattern for "avoid a run of k" is a k× k matrix, and for "avoid an arbitrary pattern" it is the KMP automaton of that pattern.
- Theory
Explain Kitamasa's algorithm and why it beats matrix exponentiation for large k.
Solution
Let f(x) = x^k - c_1x^{k-1} - ··· - c_k be the characteristic polynomial. Then in ℤ_p[x]/f(x), reduce x^n to a polynomial ∑_{i<k}r_ix^i. The claim is
a_n = ∑_{i=0}^{k-1}r_ia_i.
Why: the map sending x^j ↦ a_j is linear and respects the relation f(x)=0 precisely because the recurrence says a_{j+k} = ∑ c_ia_{j+k-i}. So reducing the exponent modulo f and evaluating gives the right answer. ∎
Cost: log n polynomial multiplications and reductions, each O(k²) naively or O(klog k) with NTT — against O(k³) per matrix multiplication.
At k=1000: matrix is 10⁹×60 = 6×10^{10} operations (hours), Kitamasa is 10⁶×60 = 6×10⁷ (a second), and NTT-Kitamasa is 10⁴×60 (instant).
- Theory
Show how to use matrix exponentiation over the (min,+) semiring, and what it computes.
Solution
Replace scalar + by min and scalar × by +. The product becomes
(A⊗ B)_{ij} = min_k(A_{ik}+B_{kj}),
with identity the matrix having 0 on the diagonal and +∞ elsewhere. Associativity holds (both groupings equal min_{k,l}(A_{ik}+B_{kl}+C_{lj})), so binary exponentiation applies.
(W^{⊗ n})_{ij} is the minimum weight of a walk from i to j using exactly n edges. ∎
To get "at most n edges", add zero-weight self-loops so a walk may idle. This solves problems that Dijkstra and Bellman–Ford cannot — a hard constraint on the edge count — in O(V³log n).
The same substitution with (max,+) gives longest walks and with (OR,AND) gives reachability in exactly n steps. One exponentiation routine, parameterised by the semiring.
- Theory
Determine which of these can be handled by matrix exponentiation, and why: (a) a_n = a_{n-1}+a_{n-2}, (b) a_n = a_{n-1}², (c) a_n = a_{n-1}+a_{⌊ n/2⌋}, (d) a_n = na_{n-1}.
Solution
- (a) Yes — linear with constant coefficients, the canonical case.
- (b) No — non-linear. (Though log a_n = 2log a_{n-1} is linear, giving a_n = a_0^{2^n} in closed form, so this particular recurrence is solvable another way.)
- (c) No — the index ⌊ n/2⌋ is not a fixed offset, so no finite state vector suffices. Solve by memoised recursion over the O(log n) distinct values of ⌊ n/2^j⌋ instead.
- (d) No directly — the coefficient depends on n. But a_n = a_0· n! has a closed form, and more generally a recurrence with polynomial coefficients is holonomic, solvable by a different technique (creative telescoping) rather than by a constant matrix.
The diagnostic is simple: fixed-size state, index-independent linear transition. Failing either condition, look for a different structure.
- Program
Implement a generic linear-recurrence solver via matrix exponentiation and test it on Fibonacci, tribonacci and a custom recurrence.
Solution
MOD = 10**9+7 def linear_recurrence(coeffs, init, n): """a_n = Σ coeffs[i]·a_{n-1-i}; init = [a_0, ..., a_{k-1}]""" k = len(coeffs) if n < k: return init[n] % MOD M = [[0]*k for _ in range(k)] M[0] = [c % MOD for c in coeffs] for i in range(1, k): M[i][i-1] = 1 R = mat_pow(M, n - k + 1) v = [init[k-1-i] % MOD for i in range(k)] # (a_{k-1}, ..., a_0) return sum(R[0][i]*v[i] for i in range(k)) % MOD # tests assert linear_recurrence([1,1], [0,1], 10) == 55 assert linear_recurrence([1,1,1], [0,1,1], 10) == 149 # tribonacci for n in range(30): assert linear_recurrence([2,3], [1,1], n) == brute_force([2,3],[1,1],n)The brute-force cross-check for small n is essential — the index conventions (whether
initis oldest-first, what power the matrix is raised to) are where every implementation of this goes wrong, and the errors are invisible for the specific case you happened to test.O(k³log n). Practical to k≈200 in C++ for a single query, or k≈50 for many.
- Program
Count paths of length exactly 10^{18} in a graph, and separately count paths avoiding a forbidden vertex.
Solution
def paths_exact(adj, n, k): A = Mat([[adj[i][j] for j in range(n)] for i in range(n)]) return (A ** k).a def paths_avoiding(adj, n, k, forbidden): """delete the forbidden rows and columns before exponentiating""" keep = [i for i in range(n) if i not in forbidden] B = Mat([[adj[i][j] for j in keep] for i in keep]) return (B ** k).aDeleting rows and columns is the correct way to forbid vertices: any walk through a deleted vertex simply has no representation in the reduced matrix. Forbidding edges is even simpler — zero those entries.
Forbidding a pattern of visits (say, never two consecutive visits to the same vertex, or never a specific sequence) requires enlarging the state to remember enough history — the product of the graph with the pattern's automaton. That construction, graph × automaton, is the general technique and it keeps the problem in the matrix-exponentiation framework at the cost of a larger matrix.
- Program
Solve: count the ways to tile a 3× n grid with dominoes, for n ≤ 10^{18}.
Solution
# Broken-profile DP: state = the 3-bit profile of protrusions into the next column. # Build the 8×8 transition matrix by enumerating fills of one column. def build_transition(w=3): size = 1 << w M = [[0]*size for _ in range(size)] def fill(row, cur, nxt, mask): if row == w: M[nxt][mask] += 1 # from profile `mask` to `nxt` return if cur >> row & 1: # already occupied fill(row+1, cur, nxt, mask); return fill(row+1, cur | (1<<row), nxt | (1<<row), mask) # horizontal if row+1 < w and not (cur >> (row+1) & 1): # vertical fill(row+2, cur | (3<<row), nxt, mask) for mask in range(size): fill(0, mask, 0, mask) return MThe state is which cells of the boundary stick into the next column, so the matrix is 2^w × 2^w and exponentiation costs O(8^wlog n). For w=3 that is trivial; up to w=10 is feasible.
For 3× n the answer satisfies a_n = 4a_{n-2} - a_{n-4} with a_0=1, a_2=3 (odd n gives 0), producing 1, 3, 11, 41, 153. Deriving that recurrence from the matrix — by computing its characteristic polynomial, or by fitting with Berlekamp–Massey — is a nice way to get a 2×2 problem out of an 8×8 one.
- Program
Implement Kitamasa and compare its runtime to matrix exponentiation across a range of k.
Solution
MOD = 10**9+7 def kitamasa(coeffs, init, n): k = len(coeffs) if n < k: return init[n] % MOD f = [(-coeffs[k-1-i]) % MOD for i in range(k)] + [1] # char poly def polymul(a, b): r = [0]*(len(a)+len(b)-1) for i, x in enumerate(a): if x: for j, y in enumerate(b): r[i+j] = (r[i+j] + x*y) % MOD for i in range(len(r)-1, k-1, -1): if r[i]: for j in range(k): r[i-k+j] = (r[i-k+j] - r[i]*f[j]) % MOD return r[:k] res, base, e = [1]+[0]*(k-1), ([0,1]+[0]*(k-2) if k > 1 else [0]), n while e: if e & 1: res = polymul(res, base) base = polymul(base, base); e >>= 1 return sum(res[i]*init[i] for i in range(k)) % MODMeasured crossover: matrix exponentiation wins below k≈30 (smaller constants, better cache behaviour), Kitamasa wins above. At k=500 Kitamasa is roughly 200× faster; at k=5000 only NTT-based polynomial multiplication is viable at all.
Both must agree with a brute-force loop for small n — and testing both against the loop, rather than against each other, is what catches a shared misunderstanding of the index convention.
- Program
Use Berlekamp–Massey to find the shortest linear recurrence fitting a sequence, then extrapolate with Kitamasa.
Solution
MOD = 10**9+7 def berlekamp_massey(s): ls, cur = [], [] lf = 0; ld = 0 for i in range(len(s)): t = 0 for j, c in enumerate(cur): t = (t + c*s[i-1-j]) % MOD if (t - s[i]) % MOD == 0: continue if not cur: cur = [0]*(i+1); lf = i; ld = (t - s[i]) % MOD; continue k = (t - s[i]) * pow(ld, MOD-2, MOD) % MOD c = [0]*(i-lf-1) + [k] + [(-k*x) % MOD for x in ls] if len(c) < len(cur): c += [0]*(len(cur)-len(c)) for j in range(len(cur)): c[j] = (c[j] + cur[j]) % MOD if i - len(cur) > lf - len(ls): ls, lf, ld = cur[:], i, (t - s[i]) % MOD cur = c return curBerlekamp–Massey finds the shortest recurrence of order ≤ m/2 from m terms, in O(m²). Combined with Kitamasa it is a devastating contest technique: brute-force the first 100 terms of anything, run BM, and if it finds a short recurrence you have an O(k²log n) algorithm for n=10^{18} without ever proving the recurrence exists.
The caveat: it is a conjecture, not a proof. Verify the found recurrence against 50 more brute-forced terms before trusting it — a recurrence fitting 100 terms and failing at 101 is entirely possible, and BM will happily report one.
The same algorithm decodes BCH and Reed–Solomon codes, where it finds the error-locator polynomial — the identical mathematical problem.
- Program
Compute the number of walks with a forbidden substring pattern, combining the KMP automaton with matrix exponentiation.
Solution
def count_avoiding(n, pattern, alphabet, MOD=10**9+7): m = len(pattern) fail = kmp_failure(pattern) def go(s, ch): while s and pattern[s] != ch: s = fail[s-1] return s + 1 if pattern[s] == ch else 0 # states 0..m-1 are "alive"; state m is absorbing (pattern found) and dropped M = [[0]*m for _ in range(m)] for s in range(m): for ch in alphabet: t = go(s, ch) if t < m: M[t][s] += 1 # column s → row t R = mat_pow(Mat(M), n).a return sum(R[i][0] for i in range(m)) % MODDropping the absorbing state is what makes this count strings that avoid the pattern: any transition into state m simply has no destination, so those strings are not counted.
O(m³log n). For a pattern of length 20 and n=10^{18}, that is 8000×60 = 5×10⁵ operations — instant.
Validate against a direct O(nm) DP for n\le10⁴. The count of avoiding strings, divided by |Σ|^n, is the probability from Chapter 42 problem 13 — the same automaton serving both a counting and a probability question.
- Program
Build a general framework: given a DP transition as a function, automatically construct the matrix and exponentiate.
Solution
def build_matrix(states, transition, MOD=10**9+7): """transition(s) → list of (next_state, weight)""" idx = {s: i for i, s in enumerate(states)} n = len(states) M = [[0]*n for _ in range(n)] for s in states: for t, w in transition(s): M[idx[t]][idx[s]] = (M[idx[t]][idx[s]] + w) % MOD # column s return Mat(M), idx def run(states, transition, initial, steps): M, idx = build_matrix(states, transition) v = [0]*len(states) for s, w in initial.items(): v[idx[s]] = w R = (M ** steps).a return {s: sum(R[idx[s]][j]*v[j] for j in range(len(states))) % MOD for s in states}Separating state enumeration from transition definition from exponentiation makes the technique reusable and, more importantly, testable: the same
transitionfunction can drive both a slow step-by-step simulation and the fast matrix version, and asserting they agree for smallstepsvalidates the matrix construction completely.The column-versus-row convention is the perennial bug. Fixing it once in
build_matrix— here,M[to][from], matching v' = Mv — and testing it means never thinking about it again.
Polynomials: representation, evaluation and arithmetic
A polynomial has two representations - coefficients and point-values - and they make different operations cheap. Multiplication is slow in one and trivial in the other, which is the entire motivation for the FFT.
The FFT is the bridge: it converts between the two forms in O(nlog n) by choosing the evaluation points to be roots of unity. Multiplication then costs O(nlog n) — transform, multiply pointwise, transform back.
Horner's method
A(x) = a₀ + x(a₁ + x(a₂ + ... + x·a_{n−1}))
double horner(const vector<double>& a, double x){
double r = 0;
for (int i = a.size() - 1; i >= 0; i--) r = r*x + a[i];
return r;
}
// n multiplications, n additions — provably optimal for a general polynomial
// (Motzkin), and numerically the most stable simple scheme.Horner is optimal in the number of multiplications for a polynomial with arbitrary coefficients, and it is also the derivative-free way to do synthetic division: the intermediate values r are exactly the coefficients of A(x)/(x-c), with the final value the remainder A(c).
Polynomial division and the remainder theorem
That last line is the reason so many number-theoretic algorithms require a prime modulus: over ℤ_p polynomials behave, over ℤ_n they do not.
The Schwartz–Zippel lemma
A non-zero polynomial of total degree d over a field, evaluated at a uniformly random point of a set S^k, is zero with probability at most d/|S|.
This one lemma powers a family of randomised algorithms: polynomial identity testing, Freivalds' matrix verification (Chapter 40), perfect-matching detection via the Tutte matrix, and fingerprinting schemes generally. The point is that checking whether two polynomials are equal is far cheaper than expanding them.
Choose the representation to match the operation. Coefficients for evaluation and differentiation; point-values for multiplication and pointwise operations; roots for factoring questions. Most polynomial algorithms are "convert, do the easy thing, convert back" — and the conversion cost is what the FFT reduces from O(n²) to O(nlog n).
"Evaluate a polynomial" expects Horner and the observation that the naive ∑ a_ix^i with pow is both slower and less accurate. The follow-up — "evaluate at n points" — is O(n²) naively, O(nlog²n) by multipoint evaluation with a product tree, and exactly O(nlog n) if the points happen to be roots of unity, which is the FFT.
- Theory
Prove that Horner's method uses the minimum number of multiplications for a general polynomial.
Solution
Horner uses n multiplications and n additions for a degree-n polynomial.
Motzkin's theorem: any algorithm computing a general degree-n polynomial (with coefficients treated as indeterminates) requires at least n multiplications and n additions. The proof is an algebraic-independence argument: with fewer multiplications, the set of computable polynomials forms a variety of dimension less than n+1, so it cannot contain all degree-n polynomials. ∎
The escape: with preconditioning — allowing arbitrary preprocessing of the coefficients, not counted — a degree-n polynomial can be evaluated with about n/2 multiplications (Knuth–Eve). This is worthwhile when one fixed polynomial is evaluated many times, as in special-function libraries where
sinandexpare minimax polynomials evaluated millions of times. - Theory
State and prove the Schwartz–Zippel lemma.
Solution
Lemma: for a non-zero polynomial P in k variables of total degree d over a field, and S a finite subset, P(r_1,…,r_k) = 0 with probability at most d/|S| when the r_i are drawn uniformly and independently from S.
Proof by induction on k. For k=1, a non-zero degree-d univariate polynomial has at most d roots, giving d/|S|.
For k>1, write P = ∑_{i=0}^{t}x_1^iQ_i(x_2,…,x_k) with Q_t ≠ 0 and \deg Q_t ≤ d-t. By induction P(Q_t = 0) ≤ (d-t)/|S|. Conditioned on Q_t ≠ 0, the polynomial in x_1 is non-zero of degree t, so it vanishes with probability ≤ t/|S|. Union bound: ≤ d/|S|. ∎
Use: to test whether two circuits compute the same polynomial, evaluate both at a random point over a large field. Equal outputs mean equal polynomials with probability 1-d/|S| — and choosing |S| = 2^{64} makes the error negligible without ever expanding either expression.
- Theory
Show that a degree-n polynomial over a field has at most n roots, and give a counterexample over a ring.
Solution
Induct on n. If P(c)=0 then by the factor theorem P(x) = (x-c)Q(x) with \deg Q = n-1. Any other root r satisfies (r-c)Q(r) = 0, and in a field there are no zero divisors, so r = c or Q(r)=0. By induction Q has at most n-1 roots, so P has at most n. ∎
Over a ring: x²-1 over ℤ_{15} has roots 1, 4, 11, 14 — four roots for a degree-2 polynomial, because ℤ_{15} has zero divisors (3·5 = 0).
This is precisely why Miller–Rabin works (Chapter 23): the existence of extra square roots of 1 detects compositeness, and it is why polynomial-based schemes (Shamir secret sharing, Reed–Solomon) insist on a prime or prime-power modulus.
- Theory
Explain synthetic division and how Horner's intermediate values give the quotient.
Solution
Dividing A(x) by (x-c) gives A(x) = (x-c)Q(x) + A(c). Writing Q(x) = ∑_{i=0}^{n-1}q_ix^i and comparing coefficients:
q_{n-1} = a_n, and q_{i-1} = a_i + cq_i going down.
That recurrence is exactly Horner's inner loop. So a single Horner pass produces both the value A(c) and all the quotient coefficients — for free. ∎
Applications: deflating a polynomial after finding a root (root-finding algorithms do this repeatedly), and computing A(c) and A'(c) together for Newton's method by running Horner twice (the second pass on the quotient gives the derivative's value).
- Theory
Given the point-value representation at n points, how do you recover the coefficients, and at what cost?
Solution
Solve the Vandermonde system Va = y, which is O(n³) by general elimination but O(n²) using the Vandermonde structure — or directly by Lagrange interpolation:
A(x) = ∑_{i}y_i∏_{j≠ i}(x-x_j)/(x_i-x_j),
which is O(n²) to expand.
Better: Newton's divided differences is also O(n²) but incremental — adding a new point costs O(n) rather than restarting. And if the points are roots of unity, the inverse FFT does it in O(nlog n).
Numerical warning: Vandermonde matrices are catastrophically ill-conditioned for equally-spaced real points (condition number grows exponentially), so high-degree interpolation on a uniform grid is numerically worthless — the Runge phenomenon. Use Chebyshev nodes, which cluster at the endpoints and keep the condition number polynomial.
- Theory
Derive the cost of multiplying two polynomials by three methods: schoolbook, Karatsuba, and FFT.
Solution
- Schoolbook: O(n²) — every pair of coefficients multiplied.
- Karatsuba: split each into halves, A = A_1x^{n/2}+A_0. Three products instead of four via A_1B_1, A_0B_0, and (A_1+A_0)(B_1+B_0). T(n) = 3T(n/2)+O(n) = O(n^{log_2 3}) = O(n^{1.585}).
- FFT: evaluate both at 2n roots of unity (O(nlog n)), multiply pointwise (O(n)), interpolate back (O(nlog n)). Total O(nlog n).
Crossovers: schoolbook wins below n≈32, Karatsuba between about 32 and 1000, FFT above. Toom-3 (O(n^{1.465})) occupies a band in between in high-performance bignum libraries.
The same hierarchy governs integer multiplication, since multiplying integers is multiplying polynomials in the base and then carrying.
- Theory
Explain how polynomial identity testing detects a perfect matching in a bipartite graph.
Solution
Build the Edmonds matrix: M_{ij} = x_{ij} if edge (i,j) exists, else 0, with the x_{ij} indeterminates.
Then \det M = ∑_{σ}sgn(σ)∏_i M_{i,σ(i)}, and a term is non-zero exactly when σ is a perfect matching. Different matchings give different monomials, so no cancellation occurs: \det M ≠ 0 as a polynomial iff a perfect matching exists. ∎
By Schwartz–Zippel, substituting random values from a field of size ≥ 2n and computing the numeric determinant detects this with probability ≥ 1/2, in O(n^{ω}) time.
That is asymptotically faster than Hopcroft–Karp's O(E√V) for dense graphs, and — more importantly — it parallelises and works for general (non-bipartite) graphs via the Tutte matrix, where combinatorial algorithms are far more intricate.
- Program
Implement Horner evaluation, synthetic division and derivative evaluation in a single pass.
Solution
def horner_all(a, c): """returns (A(c), A'(c), quotient coefficients of A(x)/(x-c))""" n = len(a) q = [0]*(n-1) val = a[n-1] for i in range(n-2, -1, -1): q[i] = val val = val*c + a[i] # second Horner pass on q gives A'(c) d = 0 for i in range(len(q)-1, -1, -1): d = d*c + q[i] return val, d, qTwo passes, 2n multiplications, giving the value, the derivative and the deflated polynomial. This is exactly what a Newton root-finder needs per iteration, and computing the derivative by differentiating the coefficients and running a third Horner is wasteful and less accurate.
Why A'(c) = Q(c): differentiating A(x) = (x-c)Q(x)+A(c) gives A'(x) = Q(x)+(x-c)Q'(x), and setting x=c kills the second term. ∎
- Program
Implement polynomial multiplication three ways and find the empirical crossovers.
Solution
def mul_school(a, b): r = [0]*(len(a)+len(b)-1) for i, x in enumerate(a): if x: for j, y in enumerate(b): r[i+j] += x*y return r def mul_karatsuba(a, b, cutoff=32): n = max(len(a), len(b)) if n <= cutoff: return mul_school(a, b) m = n // 2 a0, a1 = a[:m], a[m:] b0, b1 = b[:m], b[m:] z0 = mul_karatsuba(a0, b0, cutoff) z2 = mul_karatsuba(a1, b1, cutoff) z1 = mul_karatsuba(add(a0,a1), add(b0,b1), cutoff) z1 = sub(sub(z1, z0), z2) r = [0]*(len(a)+len(b)-1) for i, v in enumerate(z0): r[i] += v for i, v in enumerate(z1): r[i+m] += v for i, v in enumerate(z2): r[i+2*m] += v return rTypical crossovers with these implementations: schoolbook beats Karatsuba below n≈40; Karatsuba beats FFT below n≈600. The cutoff parameter matters a lot — tuning it is worth 20–30%.
The
if x:guard in schoolbook is a large win for sparse inputs and costs nothing for dense ones. For genuinely sparse polynomials (few non-zero terms among a high degree), a dictionary-based representation beats all three. - Program
Implement polynomial division with remainder over a field, and use it to compute a polynomial gcd.
Solution
MOD = 10**9+7 def poly_divmod(a, b): a = a[:]; db = len(b)-1 inv_lead = pow(b[-1], MOD-2, MOD) q = [0]*(len(a)-db) if len(a) > db else [0] for i in range(len(a)-1, db-1, -1): c = a[i]*inv_lead % MOD q[i-db] = c if c: for j in range(db+1): a[i-db+j] = (a[i-db+j] - c*b[j]) % MOD return q, a[:db] def poly_gcd(a, b): while any(b): a, b = b, poly_divmod(a, b)[1] while b and b[-1] == 0: b.pop() return aThe Euclidean algorithm works verbatim for polynomials over a field, because the degree plays the role of the absolute value: each remainder has strictly smaller degree, so it terminates in at most \deg steps.
Uses: gcd(P, P') detects repeated roots (a root of multiplicity m appears with multiplicity m-1 in the derivative), which is the first step of square-free factorisation. And the extended version gives modular inverses in 𝔽_p[x]/f(x), needed for Kitamasa and for finite-field arithmetic.
Over ℚ use the subresultant algorithm instead: naive rational polynomial gcd suffers the same coefficient explosion as rational Gaussian elimination.
- Program
Implement multipoint evaluation in O(nlog²n) using a product tree.
Solution
def build_tree(points, lo, hi, tree): """tree[node] = ∏ (x − points[i]) for i in [lo, hi)""" if hi - lo == 1: tree[(lo,hi)] = [(-points[lo]) % MOD, 1] return tree[(lo,hi)] mid = (lo+hi)//2 L = build_tree(points, lo, mid, tree) R = build_tree(points, mid, hi, tree) tree[(lo,hi)] = poly_mul(L, R) return tree[(lo,hi)] def eval_down(A, points, lo, hi, tree, out): if hi - lo == 1: out[lo] = A[0] if A else 0 return mid = (lo+hi)//2 eval_down(poly_divmod(A, tree[(lo,mid)])[1], points, lo, mid, tree, out) eval_down(poly_divmod(A, tree[(mid,hi)])[1], points, mid, hi, tree, out)The idea: A(x_i) = (A mod (x-x_i)), and remainders compose — A mod ∏ then mod each factor gives the same answers. So build a binary product tree, then push A down it taking remainders.
O(nlog²n) with FFT-based multiplication and division: log n levels, each doing O(nlog n) work. Against O(n²) for n separate Horner evaluations, this wins above n≈1000.
The transpose of this algorithm is fast interpolation, also O(nlog²n) — the two are dual, which is the general phenomenon of transposition in algebraic complexity.
- Program
Use Schwartz–Zippel to test whether two arithmetic circuits compute the same polynomial.
Solution
import random MOD = (1 << 61) - 1 # a large prime keeps the error tiny def identity_test(circuit1, circuit2, nvars, degree, rounds=5): for _ in range(rounds): pt = [random.randrange(MOD) for _ in range(nvars)] if circuit1(pt) % MOD != circuit2(pt) % MOD: return False, pt # a definite counterexample return True, None # equal w.p. ≥ 1 − rounds·d/MODError probability per round is d/|F|; with d = 10⁶ and |F| = 2^{61} that is 4×10^{-13}, so a single round is already conclusive for practical purposes.
The power of the method: it never expands either circuit. Two expressions that would each have 2^{100} monomials when expanded are compared in the time it takes to evaluate them once.
No deterministic polynomial-time algorithm for this problem is known — polynomial identity testing is the flagship problem in BPP not known to be in P, and derandomising it is known to imply circuit lower bounds. So this is one of the few places where randomness appears to be genuinely necessary rather than merely convenient.
- Program
Implement a fingerprinting scheme to compare large data structures by treating them as polynomials.
Solution
import random MOD = (1 << 61) - 1 R = random.randrange(2, MOD) # a fresh random point per process def fingerprint(seq): """treat the sequence as polynomial coefficients, evaluate at R""" h = 0 for v in seq: h = (h*R + v) % MOD return h def multiset_fingerprint(items): """order-independent: ∏ (R − item), so permutations agree""" h = 1 for v in items: h = h * ((R - v) % MOD) % MOD return hThe sequence version is Rabin–Karp hashing (Chapter 16); the multiset version uses the product form so that reordering does not change the value, which is what you want when comparing sets or multisets.
Two distinct inputs collide only if a specific non-zero polynomial of degree n vanishes at R — probability n/MOD by Schwartz–Zippel. With n=10⁶ and a 61-bit modulus, that is 4×10^{-13}.
Choose R at run time. A hard-coded R lets an adversary construct collisions offline, which is the anti-hash attack of Chapter 16. Randomising per process makes the guarantee hold against any fixed input.
Applications: comparing Merkle-style tree contents, verifying that a distributed shuffle preserved the multiset, and incremental set-equality checks where the fingerprint updates in O(1) per insertion.
- Program
Find all roots of a polynomial over 𝔽_p using the x^p - x trick and Cantor–Zassenhaus.
Solution
def roots_mod_p(f, p): # every element of F_p satisfies x^p = x, so gcd(f, x^p − x) collects # exactly the linear factors, i.e. the roots. xp = poly_powmod([0,1], p, f, p) # x^p mod f g = poly_gcd(f, poly_sub(xp, [0,1]), p) # squarefree, splits into roots roots = [] def split(h): if len(h) == 1: return if len(h) == 2: # linear: ax + b roots.append((-h[0]) * pow(h[1], p-2, p) % p); return while True: # Cantor–Zassenhaus a = [random.randrange(p), 1] t = poly_powmod(a, (p-1)//2, h, p) d = poly_gcd(h, poly_sub(t, [1]), p) if 1 < len(d) < len(h): split(d); split(poly_divmod(h, d)[0]); return split(g) return sorted(roots)Two ideas. First, x^p - x = ∏_{c∈𝔽_p}(x-c), so gcd(f, x^p-x) is exactly the product of f's linear factors — computing x^p mod f by binary exponentiation makes this O(log p) polynomial multiplications rather than anything involving p itself.
Second, splitting the result needs randomness: (x+a)^{(p-1)/2} is ±1 at each root according to whether c+a is a quadratic residue, so a random a separates the roots into two groups with probability about 1/2 per pair. Expected O(log) rounds.
This is how finite-field root-finding is actually done, and it is the core of factoring polynomials over 𝔽_p — used in coding theory, in the number field sieve, and in discrete-log index calculus.
The DFT and the Fast Fourier Transform
Evaluating a polynomial at the n-th roots of unity is a divide-and-conquer that reuses half its work, giving O(n log n) instead of O(n^2). That single algorithm turns convolution - which appears everywhere - from quadratic into near-linear.
The discrete Fourier transform evaluates a polynomial at the n-th roots of unity ω^k where ω = e^{2πi/n}:
Why it is fast
Split A into even- and odd-indexed coefficients: A(x) = A_e(x²) + xA_o(x²). Evaluating at the n roots of unity requires evaluating A_e and A_o at the squares — but the squares of the n-th roots are only the n/2 roots of order n/2. Half the work vanishes.
That sign flip is the entire saving: one multiplication ω^kO_k serves two outputs.
void fft(vector<complex<double>>& a, bool invert){
int n = a.size();
for (int i = 1, j = 0; i < n; i++){ // bit-reversal permutation
int bit = n >> 1;
for (; j & bit; bit >>= 1) j ^= bit;
j ^= bit;
if (i < j) swap(a[i], a[j]);
}
for (int len = 2; len <= n; len <<= 1){
double ang = 2 * M_PI / len * (invert ? -1 : 1);
complex<double> wl(cos(ang), sin(ang));
for (int i = 0; i < n; i += len){
complex<double> w(1);
for (int j = 0; j < len/2; j++){
auto u = a[i+j], v = a[i+j+len/2] * w;
a[i+j] = u + v;
a[i+j+len/2] = u - v;
w *= wl;
}
}
}
if (invert) for (auto& x : a) x /= n;
}Convolution in the time domain is pointwise multiplication in the frequency domain. That is the convolution theorem, and it is why the FFT matters far beyond polynomials: any problem expressible as c_k = ∑_{i+j=k}a_ib_j — string matching with wildcards, big-integer multiplication, signal filtering, correlation, subset-sum counting — becomes O(nlog n).
Precision
Floating-point FFT accumulates error O(εlog n) per coefficient, scaled by the magnitudes involved. For integer convolution with results bounded by M, you need
Always round the result and verify that the rounding error was small — silently wrong integer convolutions are a classic and hard-to-find bug.
"Multiply two large numbers" is polynomial multiplication in base 10^k plus carrying, so FFT gives O(nlog n). The detail worth adding: this is what GMP does above a few thousand digits, and the current theoretical best is Harvey–van der Hoeven's O(nlog n) — which finally matched the conjectured optimum in 2019, though with constants that make it purely theoretical.
- Theory
Prove the orthogonality relation ∑_{k=0}^{n-1}ω^{k(j-l)} = n[j=l] and deduce the inverse DFT.
Solution
If j = l every term is 1, giving n.
If j ≠ l, let z = ω^{j-l} ≠ 1 (since 0 < |j-l| < n and ω is a primitive n-th root). Then the sum is geometric:
∑_{k=0}^{n-1}z^k = (z^n-1)/(z-1) = (1-1)/(z-1) = 0. ∎
Inverse DFT: substitute A_k = ∑_j a_jω^{jk} into \frac1n∑_kA_kω^{-lk}:
\frac1n∑_k∑_j a_jω^{jk}ω^{-lk} = \frac1n∑_j a_j∑_kω^{k(j-l)} = \frac1n· na_l = a_l ✓. ∎
So F^{-1} = \frac1n\bar{F} — the inverse transform is the same algorithm with ω→ω^{-1} and a division by n, which is why one function with an
invertflag suffices. - Theory
Prove the convolution theorem.
Solution
Let c = a * b with c_k = ∑_{i+j=k}a_ib_j (cyclic, indices mod n). Then
C_m = ∑_k c_kω^{km} = ∑_k∑_{i+j≡ k}a_ib_jω^{km} = ∑_i∑_j a_ib_jω^{(i+j)m}
= (∑_i a_iω^{im})(∑_j b_jω^{jm}) = A_mB_m. ∎
The reindexing step uses that ω^n = 1, so ω^{(i+j)m} depends only on i+j mod n — which is why the theorem is about cyclic convolution.
Practical consequence: to compute an ordinary (non-cyclic) product of polynomials of degrees d_1 and d_2, zero-pad both to length ≥ d_1+d_2+1, rounded up to a power of two. Forgetting to pad causes the high coefficients to wrap around and corrupt the low ones — the single most common FFT bug.
- Theory
Derive the recurrence for the FFT and explain the bit-reversal permutation.
Solution
Splitting into even and odd indices gives T(n) = 2T(n/2)+O(n), hence O(nlog n) by the master theorem's balanced case.
Bit reversal: the recursion repeatedly splits by the lowest bit of the index — evens (bit 0 = 0) go left, odds go right. After log n levels, the element originally at index j has been routed according to j's bits read from least to most significant, so it lands at position reverse(j).
The iterative algorithm therefore permutes first (putting each element where the recursion would have left it) and then merges bottom-up, avoiding recursion and extra allocation entirely. ∎
The permutation is an involution (reverse(reverse(j)) = j), which is why the
if (i < j) swapguard performs each swap exactly once. - Theory
Determine the largest safe input size for a double-precision FFT convolution of integer sequences bounded by B.
Solution
The largest intermediate value is about nB² (each output coefficient sums up to n products). Relative error in the FFT is O(εlog n), so the absolute error is roughly nB²εlog n. To round correctly this must stay below 0.5:
nB² \lesssim (0.5)/(εlog n) ≈ \frac{10^{15}}{log n}.
B = 10 (digits) n up to ~10¹² always fine B = 10⁴ n up to ~10⁶ fine B = 10⁶ n up to ~100 marginal B = 10⁹ unsafe at any n — use NTT or splittingThe practical rule: compute max|a|·max|b|· n and require it below 10^{14}. Beyond that, either use NTT (Chapter 51, exact) or split each coefficient into 15-bit halves and perform four transforms, which keeps the products small enough.
Always assert that the maximum distance from the computed values to the nearest integer is below, say, 0.25 — that check costs nothing and turns a silent corruption into a loud failure.
- Theory
Show how to compute a cross-correlation with the FFT and why one input must be reversed.
Solution
Correlation is r_k = ∑_i a_i b_{i+k}, whereas convolution is c_k = ∑_i a_i b_{k-i}. Substituting b'_j = b_{-j} (i.e. reversing b) turns one into the other:
(a * b')_k = ∑_i a_ib'_{k-i} = ∑_i a_ib_{i-k} = r_{-k}.
So: reverse b, convolve, and read the result backwards. ∎
Application: string matching. Encode the text and the reversed pattern as ±1 sequences and convolve; positions where the result equals the pattern length are exact matches. With wildcards, use the sum ∑ (t_i - p_j)²t_ip_j expanded into three convolutions, which is zero exactly at a match — giving O(nlog n) wildcard matching, better than any combinatorial algorithm.
- Theory
Explain how the FFT is used for big-integer multiplication and where carrying fits in.
Solution
Write each integer in base 2^{16} (or 10^4) as a polynomial evaluated at that base. Multiply the polynomials by FFT, then carry: sweep the coefficient array propagating values ≥ the base into the next position.
The carrying step is O(n) and must come after the convolution — the polynomial product has un-normalised coefficients that can far exceed the base, which is exactly what makes the convolution linear.
Base choice is a trade-off: a larger base means fewer coefficients (smaller n) but larger products (worse precision). Base 2^{16} with double-precision FFT is safe up to about n = 10⁶ limbs, i.e. 16-million-bit integers.
GMP's schedule: schoolbook below ~30 limbs, Karatsuba to ~120, Toom-3 to ~500, Toom-6.5 to a few thousand, then FFT. Each crossover is measured per architecture at build time — a striking amount of engineering for one operation.
- Theory
What is a Bluestein transform and when do you need it?
Solution
The radix-2 FFT requires n to be a power of two. Bluestein's chirp-z algorithm computes a DFT of arbitrary length n by rewriting it as a convolution.
Use the identity jk = (j²+k²-(j-k)²)/(2):
A_k = ω^{k²/2}∑_j(a_jω^{j²/2})ω^{-(j-k)²/2},
which is a convolution of the sequence a_jω^{j²/2} with the "chirp" ω^{-j²/2}. That convolution can be computed by zero-padding to any convenient power of two. ∎
When you need it: computing a DFT of a prime length exactly (as in some NTT settings where the modulus only supports certain lengths), or evaluating a polynomial at points in geometric progression — the chirp-z transform in signal processing.
In competitive programming the usual answer is simply to zero-pad to a power of two, which is correct for polynomial multiplication. Bluestein matters when the cyclic structure of a specific length n is required.
- Program
Implement a recursive and an iterative FFT and verify both against a direct O(n²) DFT.
Solution
import cmath def dft_naive(a, invert=False): n = len(a); s = 1 if invert else -1 return [sum(a[j]*cmath.exp(s*2j*cmath.pi*j*k/n) for j in range(n))/(n if invert else 1) for k in range(n)] def fft_rec(a, invert=False): n = len(a) if n == 1: return a[:] ev = fft_rec(a[0::2], invert) od = fft_rec(a[1::2], invert) ang = (2j if invert else -2j) * cmath.pi / n out = [0]*n for k in range(n//2): t = cmath.exp(ang*k) * od[k] out[k] = ev[k] + t out[k + n//2] = ev[k] - t return [x/2 for x in out] if invert else out import random for logn in range(1, 11): n = 1 << logn a = [complex(random.gauss(0,1), random.gauss(0,1)) for _ in range(n)] assert max(abs(x-y) for x, y in zip(fft_rec(a), dft_naive(a))) < 1e-9The
/2per level in the inverse accumulates to the required 1/n overall — an alternative to dividing once at the end, and slightly better numerically.Cross-checking against the naive DFT for n ≤ 1024 is the right validation: it catches sign errors in the exponent, a missing conjugate in the inverse, and off-by-one in the butterfly indices, all of which produce plausible-looking garbage otherwise.
- Program
Multiply two polynomials with 10⁶ integer coefficients using FFT, and validate the rounding.
Solution
def poly_mul_fft(a, b): need = len(a) + len(b) - 1 n = 1 while n < need: n *= 2 fa = [complex(x) for x in a] + [0]*(n - len(a)) fb = [complex(x) for x in b] + [0]*(n - len(b)) fft(fa, False); fft(fb, False) for i in range(n): fa[i] *= fb[i] fft(fa, True) out = [] max_err = 0.0 for i in range(need): v = fa[i].real r = round(v) max_err = max(max_err, abs(v - r)) out.append(int(r)) assert max_err < 0.25, f"FFT precision loss: max error {max_err}" return outThe assertion is the important line. Without it, an input just past the precision limit produces coefficients off by one with no indication that anything went wrong — and those errors then propagate silently through whatever uses the result.
Zero-padding to n ≥ \deg A + \deg B + 1 is what makes this a linear rather than cyclic convolution. Padding to exactly the next power of two wastes at most 2× and is what every implementation does.
- Program
Implement string matching with wildcards in O(nlog n) using FFT.
Solution
# Match p (length m, wildcards = 0) against t (length n, wildcards = 0). # Position i is a match iff Σ_j p_j·t_{i+j}·(p_j − t_{i+j})² = 0 # Expanding gives three convolutions: # Σ p³t − 2 Σ p²t² + Σ p t³ def wildcard_match(t, p): n, m = len(t), len(p) pr = p[::-1] c1 = conv([x**3 for x in pr], t) c2 = conv([x**2 for x in pr], [x**2 for x in t]) c3 = conv(pr, [x**3 for x in t]) return [i for i in range(n - m + 1) if c1[i+m-1] - 2*c2[i+m-1] + c3[i+m-1] == 0]Why it works: the term p_jt_{i+j}(p_j-t_{i+j})² is zero when the characters match, zero when either is a wildcard (encoded as 0), and strictly positive otherwise. Since every term is non-negative, the sum is zero iff every position matches. ∎
Three convolutions, so O(nlog n) total — and this is essentially the only known way to get wildcards without a factor of the alphabet size or the number of wildcards.
Precision: the cubes make the values large, so with character codes up to 26 and n=10⁶ the intermediates reach 26³·26·10⁶ ≈ 10^{12} — safe, but for a large alphabet map the characters to small integers first, or use NTT.
- Program
Compute the number of pairs summing to each value using FFT, for values up to 10⁶.
Solution
def pair_sum_counts(values, maxv): cnt = [0]*(maxv+1) for v in values: cnt[v] += 1 conv_result = poly_mul_fft(cnt, cnt) # counts ORDERED pairs (i,j) # remove the i == j pairs and halve for unordered out = [0]*(2*maxv+1) for s in range(len(conv_result)): out[s] = conv_result[s] for v in values: if 2*v <= 2*maxv: out[2*v] -= 1 # drop the self-pairs return [x//2 for x in out]The count array is the polynomial ∑ cnt[v]x^v, and its square's coefficient of x^s is exactly the number of ordered pairs summing to s. O(Mlog M) where M is the value range — independent of how many numbers there are.
Subtracting the self-pairs before halving is the step that is easy to get wrong: (i,i) appears once in the ordered count while (i,j) and (j,i) appear twice, so they must be handled separately.
The same technique counts pairs with a given difference (convolve
cntwith its reverse), triples summing to s (cube the polynomial), and subset sums with bounded multiplicity. - Program
Implement a real-input FFT that computes two real transforms with one complex transform.
Solution
def two_real_ffts(a, b): """a, b real of the same length n; returns their DFTs, using ONE FFT""" n = len(a) z = [complex(a[i], b[i]) for i in range(n)] fft(z, False) A = [0]*n; B = [0]*n for k in range(n): j = (n - k) % n A[k] = (z[k] + z[j].conjugate()) / 2 # even part → a B[k] = (z[k] - z[j].conjugate()) / 2j # odd part → b return A, BWhy: for real input the DFT is conjugate-symmetric, A_{n-k} = A_k. Packing a and b into the real and imaginary parts and using that symmetry separates them afterwards, halving the transform count.
Applied to polynomial multiplication of real inputs, this turns three transforms into two — a 33% saving that every competitive FFT implementation uses. The further refinement packs a single real sequence of length 2n into a complex one of length n, halving again.
Test by comparing against two separate FFTs; the agreement should be to 10^{-12}, and any systematic discrepancy points at the conjugate index (n-k) mod n, where the k=0 case is the one people mishandle.
- Program
Implement Bluestein's algorithm for a DFT of arbitrary length and verify against the naive DFT.
Solution
import cmath def bluestein(a): n = len(a) w = [cmath.exp(-1j*cmath.pi*(k*k % (2*n))/n) for k in range(n)] # chirp-multiply the input x = [a[k]*w[k] for k in range(n)] # the chirp filter, length 2n−1, centred m = 1 while m < 2*n: m *= 2 y = [0]*m y[0] = w[0].conjugate() for k in range(1, n): y[k] = y[m-k] = w[k].conjugate() x += [0]*(m-n) fft(x, False); fft(y, False) for i in range(m): x[i] *= y[i] fft(x, True) return [x[k]*w[k] for k in range(n)]The
k*k % (2*n)is not an optimisation but a precision fix: computing k² directly for large k gives an enormous argument toexp, and the floating-point angle loses all its significant bits. Reducing modulo 2n keeps the angle in range, exploiting ω^{k²} depending only on k² mod 2n.Verify against the naive DFT for every n ≤ 100, including primes and odd composites — the cases a radix-2 FFT cannot handle at all, which is the entire reason this algorithm exists.
- Program
Benchmark FFT-based multiplication against Karatsuba and schoolbook for big integers, and find the crossovers.
Solution
import time, random def bench(f, digits, trials=5): a = random.randrange(10**(digits-1), 10**digits) b = random.randrange(10**(digits-1), 10**digits) t = time.perf_counter() for _ in range(trials): f(a, b) return (time.perf_counter() - t) / trials for d in (10, 100, 1000, 10**4, 10**5, 10**6): print(d, bench(mul_school, d), bench(mul_karatsuba, d), bench(mul_fft, d))Expected shape: schoolbook fastest below ~50 digits, Karatsuba from ~50 to ~2,000, FFT above. Python's built-in
*uses Karatsuba above 70 digits and never switches to FFT — so for very large numbers a hand-written FFT multiplication genuinely beats the language builtin, which is a surprising and demonstrable result.Plot log(time) against log(digits) and read the slopes: 2.0, 1.58 and ~1.0 respectively — the three complexity exponents, measured rather than asserted. That plot is the most convincing single demonstration of what asymptotic analysis is claiming (Chapter 8).
NTT: exact convolution modulo a prime
The FFT works in any ring that has a primitive root of unity of the right order. Working modulo a carefully chosen prime gives exactly that, with integer arithmetic - so the convolution is exact, with no precision analysis at all.
The FFT needs only three things from its scalars: a primitive n-th root of unity ω, invertibility of n, and a ring. Complex numbers are one choice. ℤ_p is another, and it is exact.
The transform
const long long MOD = 998244353, G = 3;
void ntt(vector<long long>& a, bool invert){
int n = a.size();
for (int i = 1, j = 0; i < n; i++){ // bit reversal
int bit = n >> 1;
for (; j & bit; bit >>= 1) j ^= bit;
j ^= bit;
if (i < j) swap(a[i], a[j]);
}
for (int len = 2; len <= n; len <<= 1){
long long wl = power(G, (MOD-1)/len, MOD);
if (invert) wl = power(wl, MOD-2, MOD);
for (int i = 0; i < n; i += len){
long long w = 1;
for (int j = 0; j < len/2; j++){
long long u = a[i+j], v = a[i+j+len/2] * w % MOD;
a[i+j] = (u + v) % MOD;
a[i+j+len/2] = (u - v + MOD) % MOD;
w = w * wl % MOD;
}
}
}
if (invert){
long long ninv = power(n, MOD-2, MOD);
for (auto& x : a) x = x * ninv % MOD;
}
}Compare with the FFT of Chapter 50: the loop structure is character-for-character identical. Only complex<double> becomes long long and exp(iθ) becomes a modular power.
NTT is exact — there is no precision budget, no rounding, no assertion needed. The cost is that the answer is a residue: if you need the true integer and it can exceed the modulus, you must use several primes and CRT (Chapter 22). Choose FFT when the answer is genuinely real, NTT when it is an integer or already modular.
Convolving modulo an arbitrary number
10⁹+7 is not NTT-friendly (10⁹+6 = 2·500000003, only one factor of 2). Two standard workarounds:
What convolution buys
| Problem | As a convolution |
|---|---|
| Polynomial product | the definition |
| Counting pairs with a given sum | square the count array |
| Subset-sum counting | product of (1+x^{a_i}), divide and conquer |
| Formal power series ops | Newton iteration, each step a convolution |
| Linear recurrence at huge n | Kitamasa / Bostan–Mori (Chapter 48) |
| Multipoint evaluation | product tree (Chapter 49) |
| String matching with wildcards | three convolutions (Chapter 50) |
"Why 998244353?" is a question with a precise answer: it is prime, it equals 119·2^{23}+1 so it supports transforms up to length 2^{23}, 3 is a primitive root, and it is just under 2^{30} so products fit in 64 bits. Every one of those four properties is load-bearing, and being able to say so distinguishes understanding from memorisation.
- Theory
Prove that ℤ_p contains a primitive n-th root of unity iff n | p-1.
Solution
The multiplicative group ℤ_p^× is cyclic of order p-1 (Chapter 24). An element of order n exists iff the cyclic group of order p-1 has an element of order n, which by Lagrange happens iff n | p-1. ∎
Construction: if g is a primitive root, then ω = g^{(p-1)/n} has order exactly n — its powers ω^k = g^{k(p-1)/n} equal 1 iff n | k.
For an NTT of length 2^k this requires 2^k | p-1, hence the form p = c·2^k+1. And this is exactly why 10⁹+7 cannot host an NTT of any length beyond 2.
- Theory
Verify that 3 is a primitive root modulo 998244353 and compute the root of unity for length 2^{20}.
Solution
p-1 = 998244352 = 2^{23}·7·17. To verify 3 is a primitive root, check 3^{(p-1)/q} ≠ 1 for each prime q ∈ {2,7,17} — three modular exponentiations, and all three are non-trivial, so the order of 3 is the full p-1. ∎
For a transform of length 2^{20}:
ω = 3^{(p-1)/2^{20}} = 3^{952· 2^{3}·…} = 3^{998244352/1048576} = 3^{952}.
Verify by checking ω^{2^{20}} ≡ 1 and ω^{2^{19}} ≡ -1 — the second is the important one, since it confirms the order is exactly 2^{20} and not a proper divisor. That -1 is precisely what makes the butterfly's sign flip work.
- Theory
Determine how many NTT-friendly primes are needed to convolve sequences of length 10⁶ with coefficients below 10⁹.
Solution
The largest true coefficient is at most n·(max a)(max b) = 10⁶·10⁹·10⁹ = 10^{24}.
To recover it exactly by CRT, the product of the moduli must exceed 10^{24}. Each of the standard primes is about 10⁹, so:
- two primes: ≈ 7.9×10^{16} — insufficient.
- three primes: ≈ 7.9×10^{25} — sufficient ✓.
Hence the standard three-prime approach. ∎
The failure mode of using two is instructive: the answer is correct whenever the true coefficient happens to be small, and silently wrong when it is large — so it passes the samples and fails the tests. Computing the bound explicitly, rather than guessing, is the whole discipline here.
- Theory
Compare NTT and FFT: when is each preferable?
Solution
FFT (complex) NTT (modular) exactness rounding error EXACT precision analysis required none answer type real a residue mod p length restriction any (with Bluestein) 2^k | p−1 speed ~2× faster per op slower (modmul vs. mul) big answers fine needs CRT over 3 primesUse NTT when the answer is wanted modulo a prime anyway (almost all contest problems), or when exactness matters and the answer fits.
Use FFT when the answer is genuinely real (signal processing, numerical convolution), when the length is awkward, or when the coefficients are small enough that precision is a non-issue and the speed matters.
The decisive practical point: NTT removes an entire class of bug. There is no "it worked on small inputs" failure mode, because there is no error to accumulate.
- Theory
Explain how to compute the inverse of a formal power series in O(nlog n) using NTT and Newton's method.
Solution
Given A with a_0 ≠ 0, find B with AB ≡ 1 (mod x^n).
Newton's iteration for f(B) = 1/B - A = 0 gives
B_{2k} ≡ B_k(2 - AB_k) (mod x^{2k)},
doubling the number of correct terms each step, starting from B_1 = a_0^{-1}.
Cost: step k costs O(klog k) for the convolutions, and the total is ∑_{j} O(2^jlog 2^j) = O(nlog n) — the geometric series is dominated by the last term (Chapter 9), so the whole iteration costs the same as one final-size multiplication, up to a constant. ∎
The same doubling structure gives O(nlog n) algorithms for log, \exp, square roots, and division of power series — the toolkit that makes Chapter 35's generating functions computational rather than merely descriptive.
- Theory
Derive the cost of computing ∏_{i=1}^{n}(1 + x^{a_i}) and give the efficient method.
Solution
Naive: multiply one factor at a time. The running product has degree up to S = ∑ a_i, so each step costs O(S) and the total is O(nS).
Divide and conquer: split the factors into halves, recurse, and multiply the two results with NTT. The recurrence is T(n) = 2T(n/2) + O(Slog S), but the key point is that at depth d the products have degree only S/2^d on average, so the total is
O(Slog Slog n).
At n = 10⁵ and S = 10⁶: naive is 10^{11}, divide-and-conquer is 10⁶·20·17 ≈ 3×10⁸ — a 300× improvement.
This is the standard technique for subset-sum counting, for "in how many ways can you choose items summing to k", and for computing the generating function of any product of many small factors. Using a priority queue to always multiply the two smallest current products (Huffman-style) is a further constant-factor win.
- Theory
Explain why the bit-reversal permutation can be skipped in some NTT implementations.
Solution
A decimation-in-time FFT reads bit-reversed input and produces natural-order output; a decimation-in-frequency FFT does the reverse. So if you use DIF for the forward transform and DIT for the inverse, the bit-reversed intermediate order never needs correcting — the two permutations cancel.
For convolution this is entirely safe, because the pointwise multiplication step does not care what order the frequency values are in, as long as both operands use the same order. ∎
The saving is real: bit reversal is a cache-hostile permutation (strided, unpredictable access), often 10–20% of the total runtime for large transforms.
The catch is that the intermediate array is not a meaningful DFT in the usual ordering — so if you need the frequency-domain values themselves (for filtering, or for inspecting a spectrum), you must permute. For pure convolution, do not.
- Program
Implement NTT and verify convolution against a naive O(n²) version.
Solution
MOD = 998244353; G = 3 def ntt(a, invert): n = len(a) j = 0 for i in range(1, n): bit = n >> 1 while j & bit: j ^= bit; bit >>= 1 j ^= bit if i < j: a[i], a[j] = a[j], a[i] ln = 2 while ln <= n: wl = pow(G, (MOD-1)//ln, MOD) if invert: wl = pow(wl, MOD-2, MOD) for i in range(0, n, ln): w = 1 for k in range(i, i + ln//2): u = a[k]; v = a[k + ln//2] * w % MOD a[k] = (u + v) % MOD a[k + ln//2] = (u - v) % MOD w = w * wl % MOD ln <<= 1 if invert: ninv = pow(n, MOD-2, MOD) for i in range(n): a[i] = a[i] * ninv % MOD def convolve(a, b): need = len(a)+len(b)-1 n = 1 while n < need: n <<= 1 fa = a + [0]*(n-len(a)); fb = b + [0]*(n-len(b)) ntt(fa, False); ntt(fb, False) for i in range(n): fa[i] = fa[i]*fb[i] % MOD ntt(fa, True) return fa[:need]Verify against the O(n²) version for all lengths up to 500, with random coefficients — the agreement must be exact, which is a far stronger test than the FFT's tolerance comparison.
Also test length 1 and inputs whose product degree is exactly a power of two minus one, where the padding logic is most likely to be off by one.
- Program
Implement three-prime CRT convolution to multiply polynomials modulo an arbitrary m ≤ 10⁹.
Solution
PRIMES = [167772161, 469762049, 998244353] # all with primitive root 3 def convolve_arbitrary(a, b, m): res = [convolve_mod(a, b, p) for p in PRIMES] out = [] for i in range(len(res[0])): x, _ = crt([r[i] for r in res], PRIMES) # exact integer coefficient out.append(x % m) return outEach coefficient's true value is below 10^{24} < ∏ PRIMES ≈ 7.9×10^{25}, so CRT recovers it exactly before the final reduction modulo m.
Do the CRT with big integers or with Garner's mixed-radix method (Chapter 22 problem 11) — the naive ∑ a_iN_iM_i involves numbers near 10^{26}, which overflows 64-bit types. Garner keeps every intermediate below the largest prime.
Test against a Python-integer naive convolution reduced mod m for small inputs; this is the only way to catch the case where the coefficient bound was mis-estimated, since the failure only appears on large inputs.
- Program
Compute ∏(1+x^{a_i}) for n=10⁵ items with ∑ a_i ≤ 10⁶ using divide and conquer plus NTT.
Solution
def product_of_factors(items, lo, hi): if hi - lo == 1: p = [0]*(items[lo]+1); p[0] = 1; p[items[lo]] = 1 return p mid = (lo+hi)//2 L = product_of_factors(items, lo, mid) R = product_of_factors(items, mid, hi) return convolve(L, R) # better: a heap that always merges the two shortest polynomials import heapq def product_heap(items): h = [(a+1, i, poly_for(a)) for i, a in enumerate(items)] heapq.heapify(h) ctr = len(items) while len(h) > 1: _, _, A = heapq.heappop(h) _, _, B = heapq.heappop(h) C = convolve(A, B) heapq.heappush(h, (len(C), ctr, C)); ctr += 1 return h[0][2]The heap version is the Huffman-tree argument applied to polynomial multiplication: always combining the two shortest keeps the total work minimal, and it is measurably better than a fixed midpoint split when the a_i are very uneven.
O(Slog Slog n) either way. The resulting polynomial's coefficient of x^k is the number of subsets summing to k — a complete subset-sum count, where the naive DP would be O(nS) = 10^{11}.
- Program
Implement power-series inverse, log and exp using NTT and Newton iteration.
Solution
def series_inv(a, n): b = [pow(a[0], MOD-2, MOD)] k = 1 while k < n: k *= 2 t = convolve(a[:k], b)[:k] t = [(-v) % MOD for v in t] t[0] = (t[0] + 2) % MOD # 2 − a·b b = convolve(b, t)[:k] return b[:n] def series_log(a, n): """requires a[0] == 1; log a = ∫ a'/a""" da = [a[i]*i % MOD for i in range(1, len(a))] q = convolve(da, series_inv(a, n))[:n-1] inv = [0]*(n) # modular inverses of 1..n inv[1] = 1 for i in range(2, n): inv[i] = (MOD - (MOD//i)*inv[MOD % i]) % MOD return [0] + [q[i-1]*inv[i] % MOD for i in range(1, n)] def series_exp(a, n): """requires a[0] == 0; Newton on log(b) − a = 0""" b = [1] k = 1 while k < n: k *= 2 t = series_log(b + [0]*(k-len(b)), k) t = [(a[i] if i < len(a) else 0) - t[i] for i in range(k)] t[0] = (t[0] + 1) % MOD # 1 + a − log b b = convolve(b, [v % MOD for v in t])[:k] return b[:n]Each is O(nlog n) by the doubling argument of problem 5. The preconditions matter: inverse needs a_0 ≠ 0, log needs a_0 = 1 (otherwise the result is not a power series), exp needs a_0 = 0 (otherwise it does not converge formally).
With these three, Chapter 35's generating functions become executable: e^{e^x-1} gives Bell numbers, e^{-ln(1-x)-x} gives derangements, and any combinatorial specification translates directly into code.
- Program
Benchmark NTT against FFT and against naive convolution, and find the crossovers.
Solution
import time, random def bench(fn, n, trials=3): a = [random.randrange(MOD) for _ in range(n)] b = [random.randrange(MOD) for _ in range(n)] t = time.perf_counter() for _ in range(trials): fn(a, b) return (time.perf_counter()-t)/trials for n in (16, 64, 256, 1024, 4096, 16384, 65536): print(n, bench(naive_conv, n) if n <= 4096 else None, bench(ntt_conv, n), bench(fft_conv, n))Typical crossover: naive convolution wins below n≈64 (no transform overhead, tight loop), NTT wins above. FFT is roughly 1.5–2× faster than NTT per operation because a floating multiply is cheaper than a modular one, but NTT needs no precision analysis.
The measured slopes on a log–log plot are 2.0 for naive and about 1.05 for both transforms — nlog n looking like n^{1.05} over this range, exactly as Chapter 8 predicts for empirical exponent fitting.
In practice keep a cutoff:
if min(len(a), len(b)) < 64: return naive(a, b). It costs one line and is a large win for the many small convolutions inside a divide-and-conquer. - Program
Use NTT to solve: given n numbers, count the triples summing to each possible value.
Solution
def triple_sum_counts(values, maxv): cnt = [0]*(maxv+1) for v in values: cnt[v] += 1 sq = convolve(cnt, cnt) # ordered pairs cube = convolve(sq, cnt) # ordered triples # subtract the degenerate cases via inclusion–exclusion on equal indices # ordered triples with all distinct = cube − 3·(pairs with i=j) + 2·(i=j=k) cnt2 = [0]*(2*maxv+1) for v, c in enumerate(cnt): if c: cnt2[2*v] += c # the i=j "doubled" array corr = convolve(cnt2, cnt) out = [0]*len(cube) for s in range(len(cube)): t = cube[s] - 3*(corr[s] if s < len(corr) else 0) out[s] = t for v, c in enumerate(cnt): if 3*v < len(out): out[3*v] += 2*c return [x//6 for x in out] # unorderedTwo convolutions give the ordered count; the corrections remove triples reusing an index. The inclusion–exclusion over which indices coincide is Chapter 30 applied to a convolution — and it is where these solutions go wrong, so verify against brute force for n\le50.
O(Mlog M) in the value range M, independent of n. This is the standard technique for "count k-tuples summing to each target", and it extends to any fixed k by taking the k-th power of the count polynomial with the appropriate corrections.
- Program
Write an NTT-friendly prime finder: given a required transform length, find primes of the form c·2^k+1 near a target size.
Solution
def find_ntt_primes(k, bits=30, count=5): """primes p = c·2^k + 1 with p < 2^bits, plus a primitive root""" out = [] limit = 1 << bits c = 1 while len(out) < count: p = c * (1 << k) + 1 if p >= limit: break if is_prime(p): out.append((p, c, primitive_root(p))) c += 2 if k > 0 else 1 # c must be even? no: c odd keeps 2^k exact return out def primitive_root(p): fac = set(prime_factors(p-1)) for g in range(2, p): if all(pow(g, (p-1)//q, p) != 1 for q in fac): return gRunning it for k=23 and 30 bits rediscovers 998244353 with c=119 and primitive root 3 — and shows how few such primes exist in that range, which is why the same handful appear in every implementation.
The maximum transform length a prime supports is 2^{v_2(p-1)}, so record it: choosing a prime with k=21 for a length-2^{22} transform fails at runtime with a subtly wrong answer rather than an error, because g^{(p-1)/2^{22}} is not an integer exponent and the code silently uses the floor.
Asserting
(MOD-1) % n == 0at the top of the NTT is a one-line guard that turns that into an immediate, comprehensible failure.
Lagrange interpolation and sums of powers
A degree-d polynomial is determined by d+1 points, so if you can prove a quantity is polynomial in n, you can compute it for astronomically large n by evaluating d+1 small cases and interpolating.
The consecutive-node speed-up
When the nodes are 0,1,…,d, the products telescope:
MOD = 10**9+7
def lagrange_consecutive(y, n):
"""y[i] = P(i) for i = 0..d; returns P(n)"""
d = len(y) - 1
if n <= d: return y[n] % MOD
pre = [1]*(d+2); suf = [1]*(d+2)
for i in range(d+1): pre[i+1] = pre[i] * ((n - i) % MOD) % MOD
for i in range(d, -1, -1): suf[i] = suf[i+1] * ((n - i) % MOD) % MOD
fact = [1]*(d+1)
for i in range(1, d+1): fact[i] = fact[i-1]*i % MOD
res = 0
for i in range(d+1):
num = pre[i] * suf[i+1] % MOD # ∏_{j≠i}(n − j)
den = fact[i] * fact[d-i] % MOD # i!·(d−i)!
term = y[i] * num % MOD * pow(den, MOD-2, MOD) % MOD
if (d - i) & 1: term = MOD - term # the sign of ∏(i − j)
res = (res + term) % MOD
return resThe denominator ∏_{j≠ i}(i-j) = i!(d-i)!(-1)^{d-i} is what makes this O(d) rather than O(d²) — it is a factorial, precomputable, instead of a fresh product per node.
If f(n) is a polynomial in n of degree d, you never need a formula: evaluate it at 0,1,…,d by any means (a slow DP, a brute-force loop) and interpolate. The hard part is proving the degree, and the usual argument is that f(n)-f(n-1) has degree d-1, so d successive differences reach a constant.
Sums of powers
S_k(n) = ∑_{i=1}^{n}i^k is a polynomial in n of degree k+1 (Faulhaber). So S_k(10^{18}) mod p costs O(k): compute S_k(0),…,S_k(k+1) by a loop, then interpolate.
Where interpolation rescues a problem
| Quantity | Degree in n |
|---|---|
| ∑_{i≤ n} i^k | k+1 |
| Lattice points in a dilated polytope | dimension (Ehrhart) |
| Chromatic polynomial P(G,k) | |V| in k |
| Number of DP states reachable in n steps | often polynomial — test it |
| ∑_{i≤ n}i^k r^i | not polynomial; it is r^n·(poly) |
The practical technique in a contest: compute the first 30 values by brute force, take successive differences, and if they hit a constant row, the quantity is polynomial and interpolation gives the answer for any n. That empirical check is not a proof, but it is a very reliable guide.
"Compute ∑_{i=1}^{10^{18}}i⁵ mod p" has a two-sentence answer: the sum is a degree-6 polynomial in n, so compute seven values and interpolate in O(k). Deriving Faulhaber's formula with Bernoulli numbers is the wrong answer — more work, more room for error, and no faster.
- Theory
Prove that the interpolating polynomial through d+1 distinct points is unique.
Solution
Suppose P and Q both have degree ≤ d and agree at d+1 distinct points. Then P-Q has degree ≤ d and at least d+1 roots.
Over a field, a non-zero polynomial of degree ≤ d has at most d roots (Chapter 49). So P-Q = 0, i.e. P = Q. ∎
Existence is given constructively by the Lagrange formula. Equivalently, the Vandermonde matrix at distinct nodes has non-zero determinant ∏_{i<j}(x_j-x_i), so the linear system has a unique solution.
The field hypothesis matters: over ℤ_{15}, uniqueness fails along with the root bound, which is why Shamir secret sharing and Reed–Solomon codes require a prime or prime-power modulus.
- Theory
Prove that ∑_{i=1}^{n}i^k is a polynomial in n of degree k+1.
Solution
Use the finite-difference operator Δ f(n) = f(n)-f(n-1). If f is a polynomial of degree d with leading coefficient c, then Δ f has degree d-1 with leading coefficient cd — the top terms cancel.
Conversely, if Δ f is a polynomial of degree k then f is one of degree k+1 (sum the differences; the operator Δ maps degree-(k+1) polynomials onto degree-k ones).
Here Δ S_k(n) = n^k, degree k, so S_k has degree k+1. ∎ The leading coefficient is 1/(k+1), matching the integral \int x^k = x^{k+1}/(k+1).
This argument is the general recipe: to show a quantity is polynomial of degree d, show its d-th finite difference is constant — which is also the empirical test.
- Theory
Derive the O(d) formula for interpolation at consecutive nodes.
Solution
With nodes x_i = i for i=0..d:
\ell_i(n) = ∏_{j≠ i}(n-j)/(i-j).
The numerator ∏_{j≠ i}(n-j) is the full product ∏_{j=0}^{d}(n-j) with the i-th factor removed — computable from prefix and suffix product arrays in O(1) per i after O(d) preprocessing.
The denominator is
∏_{j<i}(i-j)∏_{j>i}(i-j) = i!·(-1)^{d-i}(d-i)!,
since the factors above i are negative and there are d-i of them. So each term is a factorial lookup plus a sign. ∎
Total O(d) with precomputed factorials and inverse factorials — against O(d²) for general nodes. This is why the consecutive case is worth special-casing, and it always applies when you generate the sample points yourself.
- Theory
Explain Shamir's secret sharing and why it is information-theoretically secure.
Solution
To split a secret s among n parties with threshold k: choose a random polynomial P of degree k-1 over 𝔽_p with P(0) = s and the other k-1 coefficients uniform. Give party i the share (i, P(i)).
Reconstruction: any k shares determine P uniquely by interpolation, so s = P(0) is recoverable.
Security: given any k-1 shares, for every candidate secret s' there is exactly one degree-(k-1) polynomial through those shares with P(0)=s'. Since the coefficients were uniform, all s' remain equally likely. ∎
That is information-theoretic security — not "computationally hard to break" but "the shares contain literally zero information about the secret". No amount of computing power helps, unlike RSA or discrete log, which rest on hardness assumptions.
The price: shares are as large as the secret, and the scheme provides no integrity — a malicious participant can submit a wrong share and corrupt the reconstruction undetectably, which is what verifiable secret sharing adds.
- Theory
State the Ehrhart theorem and give an example.
Solution
Ehrhart: for a d-dimensional polytope P with integer vertices, the number of lattice points in the dilation tP is a polynomial in t of degree d:
L_P(t) = c_dt^d + ··· + c_1t + 1,
with c_d = vol(P) and c_{d-1} half the surface area in lattice units.
Example: the unit square scaled by t contains (t+1)² lattice points — degree 2, leading coefficient 1 = the area ✓, constant term 1 ✓.
Example: the triangle with vertices (0,0),(1,0),(0,1) scaled by t contains C(t+2,2) = ((t+1)(t+2))/(2) points — degree 2 with leading coefficient 1/2, the area ✓.
The computational payoff: to count lattice points in a huge dilation, count for t=0..d by brute force and interpolate. This turns a geometric enumeration into d+1 small computations, and it is how "count solutions to a system of inequalities scaled by n" problems are solved.
- Theory
Give the finite-difference test for whether a sequence is polynomial, and its failure mode.
Solution
Take successive differences. If the d-th difference row is constant and non-zero, the sequence agrees with a degree-d polynomial on the values tested.
n: 0 1 4 9 16 25 (squares) Δ: 1 3 5 7 9 Δ²: 2 2 2 2 constant ⟹ degree 2 ✓Failure mode: this is a statement about the values tested, not about the sequence. Any finite set of d+1 values is matched by some degree-d polynomial, so a constant difference row proves nothing unless you have more values than the degree plus a margin.
The discipline: if the d-th difference is constant across m values, you have m-d confirmations. Require at least 5–10 of those before trusting it, and if possible prove the degree separately (by the Δ argument of problem 2, or by an Ehrhart-type structural result).
Sequences that fool the test: anything of the form poly(n) + ε(n) where ε vanishes on the tested range — for instance n² + C(n,10), which looks quadratic until n=10.
- Theory
Explain how Reed–Solomon codes use polynomial interpolation.
Solution
Encode k data symbols as the coefficients of a degree-(k-1) polynomial P over 𝔽_q, and transmit its values at n distinct points: (P(x_1),…,P(x_n)).
Any k of the n values determine P by interpolation, so the code tolerates n-k erasures (known-position losses).
For errors (unknown positions), the minimum distance is n-k+1 — meeting the Singleton bound, so RS codes are maximum distance separable — and up to ⌊(n-k)/2⌋ errors are correctable, by Berlekamp–Massey finding the error-locator polynomial (Chapter 48).
Uses: CDs and DVDs, QR codes, DSL, deep-space communication, and erasure coding in distributed storage — where an (n,k) RS code stores data on n disks surviving any n-k failures, using n/k times the space instead of the n times that replication would need.
The whole scheme rests on one fact from problem 1: d+1 points determine a degree-d polynomial, over a field.
- Program
Implement Lagrange interpolation for general nodes in O(d²) and for consecutive nodes in O(d); verify they agree.
Solution
MOD = 10**9+7 def lagrange_general(xs, ys, n): d = len(xs) res = 0 for i in range(d): num = den = 1 for j in range(d): if j == i: continue num = num * ((n - xs[j]) % MOD) % MOD den = den * ((xs[i] - xs[j]) % MOD) % MOD res = (res + ys[i] * num % MOD * pow(den, MOD-2, MOD)) % MOD return res # verify against the O(d) consecutive version import random for d in range(1, 30): coeffs = [random.randrange(MOD) for _ in range(d+1)] ev = lambda x: sum(c*pow(x, i, MOD) for i, c in enumerate(coeffs)) % MOD ys = [ev(i) for i in range(d+1)] for n in [0, d, d+1, 10**18]: assert lagrange_general(list(range(d+1)), ys, n) \ == lagrange_consecutive(ys, n) == ev(n)Testing at n=0, n=d (inside the node range, where the formula must return the given value exactly) and n=10^{18} covers the branches that matter — particularly the
if n <= d: return y[n]early exit, which exists because the general formula divides by zero at a node.Comparing against direct evaluation of a known polynomial is the decisive test: it verifies not just internal consistency but that the interpolation reproduces the right function.
- Program
Compute ∑_{i=1}^{n}i^k mod p for n ≤ 10^{18} and k ≤ 10⁶.
Solution
def power_sum(n, k, MOD=10**9+7): d = k + 1 # the degree of S_k ys = [] s = 0 for i in range(1, d+2): s = (s + pow(i, k, MOD)) % MOD ys.append(s) ys = [0] + ys # S_k(0) = 0 return lagrange_consecutive(ys, n % MOD)O(klog k) dominated by the k+2 modular exponentiations; the interpolation itself is O(k). At k=10⁶ that is about 3×10⁷ multiplications — a couple of seconds.
Speed it up by computing i^k with a linear sieve: i ↦ i^k is completely multiplicative, so only the π(k) ≈ k/ln k primes need an exponentiation and the rest are single multiplications. That cuts the work by a factor of ln k ≈ 14.
Careful with
n % MOD: the interpolation works in ℤ_p, so n must be reduced — but the early exitif n <= dmust test the true n, not the reduced one. Getting that order wrong gives a wrong answer only when n mod p ≤ d, which is rare enough to survive testing. - Program
Implement Shamir secret sharing with split and reconstruct, over a prime field.
Solution
import random P = (1 << 127) - 1 # a Mersenne prime, larger than any secret def split_secret(secret, n, k): coeffs = [secret] + [random.randrange(P) for _ in range(k-1)] def ev(x): r = 0 for c in reversed(coeffs): r = (r*x + c) % P return r return [(i, ev(i)) for i in range(1, n+1)] def reconstruct(shares): """Lagrange interpolation at x = 0""" s = 0 for i, (xi, yi) in enumerate(shares): num = den = 1 for j, (xj, _) in enumerate(shares): if i == j: continue num = num * (-xj) % P den = den * (xi - xj) % P s = (s + yi * num % P * pow(den, P-2, P)) % P return sVerify that any k shares reconstruct correctly and that k-1 shares produce a value uniformly distributed over the field — sample many random polynomials with a fixed k-1 share set and confirm the implied secrets are spread uniformly. That experiment is the operational meaning of information-theoretic security.
Never use x=0 as a share index: P(0) is the secret itself. Starting at i=1 is not cosmetic.
Use a prime larger than the secret, and note the scheme provides no authentication — a corrupt share silently produces a wrong reconstruction, indistinguishable from a correct one.
- Program
Write a tool that determines whether a sequence is polynomial and, if so, finds its degree and interpolates.
Solution
from fractions import Fraction def detect_polynomial(seq, min_confirmations=5): rows = [list(map(Fraction, seq))] while len(rows[-1]) > 1: prev = rows[-1] rows.append([prev[i+1]-prev[i] for i in range(len(prev)-1)]) row = rows[-1] if len(row) >= min_confirmations and len(set(row)) == 1 and row[0] != 0: return len(rows) - 1, rows # the degree if all(v == 0 for v in row): return len(rows) - 2, rows return None, rows # not polynomial in this rangeThe
min_confirmationsguard implements the discipline from problem 6: a constant row of length 1 is meaningless, since the last difference of any finite sequence is a single number.Run it on: squares (degree 2 ✓), ∑ i³ (degree 4 ✓), Fibonacci (never constant — correctly rejected), 2^n (rejected), and Catalan numbers (rejected). The rejections are as important as the acceptances.
In a contest this is the workflow: brute-force 40 terms, run this, and if it reports a degree, interpolate for n=10^{18} — having first sanity-checked the claimed degree against an independent brute-force value.
- Program
Count lattice points in a dilated polytope for huge dilation factors using Ehrhart interpolation.
Solution
def ehrhart(count_at, dim, t, MOD=10**9+7): """count_at(s) brute-forces the lattice points in s·P; returns the count for t""" ys = [count_at(s) % MOD for s in range(dim + 1)] return lagrange_consecutive(ys, t) # example: the simplex x_i ≥ 0, Σ x_i ≤ s in d dimensions def simplex_count(s, d): from math import comb return comb(s + d, d)Brute-force the first d+1 dilations (small, so enumeration is feasible), then interpolate. For a 4-dimensional polytope that is 5 small counts and an O(1) interpolation, versus an impossible enumeration at t=10^{18}.
The hypothesis is essential: Ehrhart requires the polytope's vertices to be integral. For rational vertices the count is a quasi-polynomial — polynomial on each residue class of t modulo the denominators — so you must interpolate each class separately, using d+1 values with the same residue.
Detecting that case: run the finite-difference test of problem 11 and watch it fail, then retry restricted to t ≡ r for each r. The successful split reveals the period.
- Program
Implement Newton's divided-difference interpolation, which supports adding points incrementally.
Solution
class NewtonInterp: def __init__(self, MOD=10**9+7): self.xs = []; self.coef = []; self.MOD = MOD def add(self, x, y): """add a point in O(current degree)""" M = self.MOD for i, (xi, ci) in enumerate(zip(self.xs, self.coef)): y = (y - ci) * pow((x - xi) % M, M-2, M) % M # note: this divides progressively, matching the divided-difference table self.coef.append(y % M); self.xs.append(x) def eval(self, x): M = self.MOD r = 0 for i in range(len(self.coef)-1, -1, -1): r = (r * ((x - self.xs[i]) % M) + self.coef[i]) % M return rAdding a point costs O(d) and never invalidates the previous coefficients — unlike the Lagrange form, where every basis polynomial changes when a node is added.
Evaluation is Horner-like in the Newton basis 1, (x-x_0), (x-x_0)(x-x_1), …, so it is O(d) per query.
This is the right structure when the number of points is not known in advance — for instance when you are guessing a polynomial's degree and want to add samples until the next coefficient comes out zero, which is exactly the "detect and interpolate" workflow of problem 11 done in one pass.
- Program
Demonstrate the Runge phenomenon: interpolate 1/(1+25x²) on equally spaced and on Chebyshev nodes, and compare the errors.
Solution
import math def runge(x): return 1.0/(1 + 25*x*x) def equally_spaced(n): return [-1 + 2*i/n for i in range(n+1)] def chebyshev(n): return [math.cos(math.pi*(2*i+1)/(2*n+2)) for i in range(n+1)] def max_error(nodes_fn, n, samples=2001): xs = nodes_fn(n); ys = [runge(x) for x in xs] worst = 0.0 for k in range(samples): t = -1 + 2*k/(samples-1) worst = max(worst, abs(lagrange_float(xs, ys, t) - runge(t))) return worst for n in (5, 10, 20, 40): print(n, max_error(equally_spaced, n), max_error(chebyshev, n))Equally spaced nodes: the maximum error grows with n — about 0.4 at n=10, 60 at n=20, and astronomically at n=40. The interpolant oscillates wildly near ±1 despite matching the function exactly at every node.
Chebyshev nodes: the error decreases steadily, reaching 10^{-6} by n=40.
Why: the interpolation error involves ∏(x-x_i), whose maximum over [-1,1] is 2^{-n} for Chebyshev nodes and exponentially larger for uniform ones. Clustering nodes at the endpoints is not a heuristic — it is the provably optimal node placement.
The lesson generalises: high-degree polynomial interpolation on a uniform grid is never the right numerical method. Use splines, or Chebyshev nodes, or a lower degree. (None of this affects the exact modular interpolation of the earlier problems, where there is no error to amplify.)
Formal power series and Newton iteration
Once you can multiply power series in O(n log n), you can invert, take logs, exponentiate and extract roots at the same cost, because Newton's method doubles the number of correct coefficients each step. That turns combinatorial specifications into executable algorithms.
A formal power series A(x) = ∑_{n\ge0}a_nx^n is a sequence dressed as an expression. Convergence is irrelevant — x is a placeholder — and the operations are defined coefficient-wise.
Newton iteration on power series
To solve F(B) = 0 for a series B, Newton's update
doubles the number of correct coefficients at each step. Since each step's cost is proportional to its size and the sizes double, the total is a geometric series dominated by the last term — so every one of these operations costs the same as one multiplication at the final size, up to a constant factor.
INVERSE F(B) = 1/B − A
B ← B(2 − A·B) mod x^{2k}
SQUARE ROOT F(B) = B² − A
B ← (B + A/B)/2 mod x^{2k}
LOG no iteration needed:
log A = ∫ (A'/A) one inverse + one product
EXP F(B) = log B − A
B ← B(1 + A − log B) mod x^{2k}Every combinatorial construction of Chapter 35 becomes a computation. "Sets of A-objects" is \exp(\hat{A}); "sequences of A-objects" is 1/(1-A); "cycles" is log(1)/(1-A). Writing down the specification and calling exp is the entire algorithm — this is the symbolic method made executable.
The exp/log dictionary for labelled structures
| Structure | EGF | Sequence |
|---|---|---|
| Permutations | 1/(1-x) | n! |
| Cycles | ln(1)/(1-x) | (n-1)! |
| Sets of cycles = permutations | e^{ln(1)/(1-x)} = (1)/(1-x) | ✓ consistent |
| Derangements | \frac{e^{-x}}{1-x} | D_n |
| Set partitions (Bell) | e^{e^x-1} | B_n |
| Involutions | e^{x+x²/2} | I_n |
| Labelled rooted trees | T = xe^T | n^{n-1} |
| Connected labelled graphs | ln(∑ 2^{\binom n2}(x^n)/(n!)) | — |
That last row is the exponential formula in action: if \hat{C} counts connected structures and \hat{A} counts all structures, then \hat{A} = e^{\hat{C}}, so \hat{C} = ln\hat{A}. Counting connected graphs — a genuinely awkward inclusion–exclusion by hand — is one call to series_log.
"Count the connected labelled graphs on n vertices" has a two-line answer with this machinery: all graphs have EGF ∑ 2^{\binom n2}x^n/n!, connected ones are its logarithm, done in O(nlog n). Doing it by the recurrence C_n = 2^{\binom n2} - ∑C(n-1,k-1)C_k2^{\binom{n-k}2} is O(n²) and far more error-prone.
- Theory
Derive the Newton iteration for the inverse of a power series and prove the doubling property.
Solution
Solve F(B) = B^{-1} - A = 0. Then F'(B) = -B^{-2}, and Newton gives
B_{new} = B - \frac{B^{-1}-A}{-B^{-2}} = B + B - AB² = B(2 - AB).
Doubling: suppose AB ≡ 1 (mod x^k), so AB = 1 - E with E ≡ 0 (mod x^k). Then
A· B(2-AB) = (1-E)(2-(1-E)) = (1-E)(1+E) = 1 - E²,
and E² ≡ 0 (mod x^{2k)}. So the new B is correct to 2k terms. ∎
Cost: ∑_{j}O(2^jlog2^j) = O(nlog n) — the geometric series is dominated by its last term (Chapter 9), so the whole iteration costs about twice the final multiplication.
- Theory
Prove the exponential formula: if \hat{A} = e^{\hat{C}} then A-structures are sets of C-structures.
Solution
An A-structure on a labelled set is a partition into blocks, each carrying a C-structure. Grouping by the number of blocks k:
\hat{A}(x) = ∑_{k\ge0}\frac{\hat{C}(x)^k}{k!},
where \hat{C}^k counts ordered k-tuples of C-structures on a labelled split (the EGF product does the binomial-coefficient splitting automatically), and k! removes the ordering since the blocks are unordered.
That sum is exactly e^{\hat{C}}. ∎
The crucial mechanism is that EGF multiplication is binomial convolution: [x^n/n!](\hat{F}\hat{G}) = ∑_k\binom nk f_kg_{n-k} — precisely the count of ways to split n labels between two structures. Ordinary GFs do not have this property, which is why labelled structures need EGFs and unlabelled ones use OGFs.
- Theory
Derive the EGF for labelled rooted trees and deduce Cayley's formula.
Solution
A rooted labelled tree is a root together with a set of rooted subtrees. By the exponential formula:
T(x) = xe^{T(x)}.
By the Lagrange inversion formula, if T = xφ(T) then
[x^n]T = \frac1n[t^{n-1}]φ(t)^n = \frac1n[t^{n-1}]e^{nt} = \frac1n·\frac{n^{n-1}}{(n-1)!} = \frac{n^{n-1}}{n!}.
Since T is an EGF, the number of rooted labelled trees on n vertices is n!·\frac{n^{n-1}}{n!} = n^{n-1}. Dividing by the n choices of root gives n^{n-2} unrooted trees — Cayley's formula. ∎
Lagrange inversion is the general tool for implicitly-defined series T = xφ(T), which is the shape of every "a tree is a root plus subtrees" specification. It converts a functional equation into a coefficient extraction with no iteration at all.
- Theory
Derive log and \exp of a power series and state their preconditions.
Solution
Log: differentiate L = log A to get L' = A'/A, then integrate term by term:
log A = \int_0^x (A'(t))/(A(t))dt, with the constant fixed by log A(0) = log 1 = 0.
This needs a_0 = 1 — otherwise the constant term would be log a_0, which has no meaning as a formal series element (and dividing by A needs a_0\ne0 regardless).
Exp: Newton on F(B) = log B - A gives B arrow B(1 + A - log B), needing a_0 = 0 so that e^A has constant term 1.
Why the preconditions are real: they are not technicalities but statements about what the formal ring supports. \exp of a series with non-zero constant term would require e^{a_0}, a transcendental number not present in ℤ_p — and dividing by i in the integration step requires p > n, which is another silent precondition worth asserting.
- Theory
Use power series to count connected labelled graphs on n vertices.
Solution
Let \hat{G}(x) = ∑_{n\ge0}2^{\binom n2}(x^n)/(n!) count all labelled graphs, and \hat{C} count connected ones.
A graph is a set of connected components, so by the exponential formula \hat{G} = e^{\hat{C}}, hence
\hat{C} = log\hat{G}.
Computing log in O(nlog n) gives all the counts at once. ∎
Values: 1, 1, 4, 38, 728, 26704, … connected graphs on n=1..6 vertices, out of 1, 2, 8, 64, 1024, 32768 total. The fraction connected rises quickly toward 1 — almost every labelled graph is connected, which is the first result of random graph theory.
The same trick counts connected structures of any kind whose disconnected version is easy: take the log. That is a remarkably general and under-used technique.
- Theory
State Lagrange inversion and apply it to the Catalan generating function.
Solution
Lagrange inversion: if T = xφ(T) with φ(0)\ne0, then
[x^n]T = \frac1n[t^{n-1}]φ(t)^n, and more generally [x^n]H(T) = \frac1n[t^{n-1}]H'(t)φ(t)^n.
Catalan: the generating function satisfies C = 1 + xC². Substituting T = C-1 gives T = x(1+T)², so φ(t) = (1+t)² and
[x^n]T = \frac1n[t^{n-1}](1+t)^{2n} = \frac1nC(2n,n-1) = (1)/(n+1)C(2n,n) ✓. ∎
Lagrange inversion is the systematic replacement for "solve the quadratic and expand the square root" — it works for any degree, handles implicit equations with no closed-form solution, and never requires choosing a branch.
- Theory
Explain how to compute A(x)^k for large k in O(nlog n) rather than O(nlog nlog k).
Solution
If a_0 = 1, use A^k = \exp(klog A) — one log, one scalar multiply, one exp, all O(nlog n) and independent of k. Binary exponentiation would cost an extra log k factor.
If a_0 ≠ 1 but a_0 ≠ 0, factor it out: A = a_0(1 + \tilde{A}) and A^k = a_0^k(1+\tilde{A})^k.
If a_0 = 0, let v be the lowest non-zero index; then A = x^vB with b_0\ne0, and A^k = x^{vk}B^k — which is identically zero modulo x^n as soon as vk ≥ n, a case worth short-circuiting since k may be enormous.
The a_0^k and the vk ≥ n check are where implementations go wrong: k is typically given modulo p for the exponent but must be used as a true integer in the shift computation, and those are different numbers.
- Program
Implement the full power-series toolkit: inverse, log, exp, sqrt, power.
Solution
MOD = 998244353 def series_sqrt(a, n): """requires a[0] == 1""" assert a[0] == 1 b = [1]; k = 1 inv2 = pow(2, MOD-2, MOD) while k < n: k *= 2 t = convolve(a[:k] + [0]*(k-min(k,len(a))), series_inv(b, k))[:k] b = [(b[i] if i < len(b) else 0) + t[i] for i in range(k)] b = [v * inv2 % MOD for v in b] return b[:n] def series_pow(a, k_true, k_mod, n): v = next((i for i, c in enumerate(a) if c), None) if v is None: return [1] + [0]*(n-1) if k_true == 0 else [0]*n if v and v * min(k_true, n) >= n: return [0]*n # shifted out entirely shift = v * k_true c0 = a[v] b = [x * pow(c0, MOD-2, MOD) % MOD for x in a[v:]] # normalise to b[0]=1 lg = series_log(b, n - shift) lg = [x * (k_mod % MOD) % MOD for x in lg] e = series_exp(lg, n - shift) scale = pow(c0, k_mod, MOD) return [0]*shift + [x * scale % MOD for x in e]Each routine is O(nlog n) by the doubling argument. The
series_powsignature takes both the true k (for the shift, which is an integer computation) and k mod p (for the exponent) — conflating them is the standard bug when k is given as a huge number.Test every routine against a slow O(n²) reference for n\le200, and test the preconditions by asserting they are checked rather than silently violated.
- Program
Compute Bell numbers, derangements and involutions from their EGFs, and verify against the recurrences.
Solution
def from_egf(series, n, MOD=998244353): fact = [1]*(n+1) for i in range(1, n+1): fact[i] = fact[i-1]*i % MOD return [series[i]*fact[i] % MOD for i in range(n)] def bell(n): ifact = inverse_factorials(n) A = [0] + ifact[1:n] # e^x − 1 return from_egf(series_exp(A, n), n) def derangements(n): ifact = inverse_factorials(n) emx = [ifact[i] * (1 if i%2==0 else MOD-1) % MOD for i in range(n)] # e^{−x} inv1mx = [1]*n # 1/(1−x) return from_egf(convolve(emx, inv1mx)[:n], n) def involutions(n): ifact = inverse_factorials(n) A = [0]*n if n > 1: A[1] = 1 if n > 2: A[2] = pow(2, MOD-2, MOD) # x + x²/2 return from_egf(series_exp(A, n), n)All three are O(nlog n), against O(n²) for the Bell triangle and O(n) for the derangement recurrence — so the EGF route only pays for Bell numbers and for structures with no simple recurrence.
Its real value is uniformity: the same three lines compute any labelled structure once you can write its EGF, without deriving a recurrence at all. Verify each against the known recurrences (Chapters 31 and 33) for n\le500; agreement across three independent derivations is strong evidence the toolkit is correct.
- Program
Count connected labelled graphs and connected labelled trees using log and Lagrange inversion.
Solution
def connected_graphs(n, MOD=998244353): ifact = inverse_factorials(n) G = [pow(2, i*(i-1)//2, MOD) * ifact[i] % MOD for i in range(n)] C = series_log(G, n) # G[0] == 1, so log is defined fact = factorials(n) return [C[i]*fact[i] % MOD for i in range(n)] def labelled_trees(n, MOD=998244353): """n^{n−2} by Cayley; here derived via Lagrange inversion as a check""" return [pow(i, i-2, MOD) if i >= 2 else 1 for i in range(n)]The exponent \binom i2 must be computed as a true integer before the
pow— reducing it modulo p first would be wrong, since it is an exponent, not a base (Chapter 20's trap).Results: 1, 1, 4, 38, 728, 26704, 1866256 for n=1..7 — matching OEIS A001187. Cross-check against the O(n²) recurrence for small n before trusting the O(nlog n) version.
The technique generalises immediately: connected bipartite graphs, connected functional graphs, and 2-connected graphs all follow by taking logs of the appropriate easy series, with no new derivation.
- Program
Solve T = xe^T numerically as a power series and compare against n^{n-1}.
Solution
def solve_tree_egf(n, MOD=998244353): """Newton on F(T) = T − x·exp(T) = 0""" T = [0]*n k = 1 while k < n: k *= 2 e = series_exp(T[:k] + [0]*(k-len(T[:k])), k) xe = [0] + e[:k-1] # x·exp(T) num = [(T[i] if i < len(T) else 0) - xe[i] for i in range(k)] # F'(T) = 1 − x·exp(T) den = [(1 if i == 0 else 0) - xe[i] for i in range(k)] corr = convolve(num, series_inv(den, k))[:k] T = [((T[i] if i < len(T) else 0) - corr[i]) % MOD for i in range(k)] fact = factorials(n) return [T[i]*fact[i] % MOD for i in range(n)] assert solve_tree_egf(20)[1:] == [pow(i, i-1) % MOD for i in range(1, 20)]Newton on an implicit functional equation, using the toolkit's
expandinvas primitives. Each step doubles the precision, so the total is O(nlog n).The assertion against n^{n-1} confirms both the Newton implementation and Cayley's formula simultaneously.
The same pattern solves any T = xφ(T): Lagrange inversion gives a closed form when φ is simple, and Newton handles the general case where it is not — for instance T = x(1 + T + T² + T³) counting ternary-branching trees.
- Program
Implement Lagrange inversion for coefficient extraction and verify on Catalan and tree numbers.
Solution
def lagrange_inversion(phi, n, MOD=998244353): """T = x·φ(T); returns [x^k]T for k = 0..n−1""" out = [0]*n powphi = [1] + [0]*(n-1) # φ(t)^0 for k in range(1, n): powphi = convolve(powphi, phi)[:n] # φ(t)^k out[k] = powphi[k-1] * pow(k, MOD-2, MOD) % MOD return outAs written this is O(n²log n) because it recomputes each power. Better: compute φ^n once via \exp(nlogφ) and use the general form [x^n]H(T) = \frac1n[t^{n-1}]H'(t)φ(t)^n for a single target coefficient — O(nlog n).
Verify: with φ(t) = (1+t)² the output should be the Catalan numbers shifted (problem 6); with φ(t) = e^t as a series, n![x^n]T = n^{n-1}.
Lagrange inversion is the tool for "the generating function is defined implicitly and I only need one coefficient" — which is the situation for most tree and lattice-path enumerations, where solving for T explicitly is impossible or unpleasant.
- Program
Use power series to compute the number of ways to partition n into parts from a given set, for n ≤ 10⁵.
Solution
def partitions_from_set(parts, n, MOD=998244353): """∏_{a ∈ parts} 1/(1 − x^a), truncated to x^n""" # Take logs: log ∏ 1/(1−x^a) = Σ_a Σ_{k≥1} x^{ak}/k L = [0]*(n+1) for a in parts: k = 1 while a*k <= n: L[a*k] = (L[a*k] + pow(k, MOD-2, MOD)) % MOD k += 1 return series_exp(L, n+1)Building the log directly is the trick: log(1)/(1-x^a) = ∑_{k\ge1}\frac{x^{ak}}{k}, so the whole product's log is a single harmonic-style sum costing O(nlog n) to assemble (the inner loop runs n/a times, summing to nH_n). One
expthen finishes it.Against the naive coin DP at O(n·|parts|): when the part set is large (say all integers up to n) the DP is O(n²) = 10^{10} while this is O(nlog n) = 2×10⁶.
For the unrestricted partition function specifically, Euler's pentagonal recurrence (Chapter 33) is O(n√n) and simpler — this method wins when the part set is arbitrary.
- Program
Build a small symbolic-method compiler: given a combinatorial specification, produce the counting sequence.
Solution
def SEQ(A, n): return series_inv([(1 if i==0 else 0) - A[i] for i in range(n)], n) def SET(A, n): return series_exp(A, n) # labelled: EGF def CYC(A, n): return series_log(series_inv([(1 if i==0 else 0)-A[i] for i in range(n)], n), n) def PROD(A, B, n): return convolve(A, B)[:n] def UNION(A, B, n): return [(A[i]+B[i]) % MOD for i in range(n)] ATOM = lambda n: [0, 1] + [0]*(n-2) # example: permutations = SET(CYC(ATOM)) n = 20 perm = SET(CYC(ATOM(n), n), n) assert from_egf(perm, n) == [factorial(i) for i in range(n)]Each combinatorial constructor is one series operation, so a specification translates mechanically into code. The assertion checks the whole pipeline against a known answer: sets of cycles are permutations, so the EGF must be 1/(1-x) and the sequence must be n!.
This is Flajolet and Sedgewick's symbolic method, and it is genuinely how modern analytic combinatorics is done: specify the structure, read off the generating function, then either extract coefficients numerically (as here) or analyse the singularities for asymptotics (as in Chapter 32).
Extending it to unlabelled structures requires the Pólya-style MSET and CYC operators with their ∑ (1)/(k)A(x^k) corrections (Chapter 34) — the same framework, one level more intricate.
Vectors in the plane: dot, cross and orientation
Two products carry almost all of computational geometry. The dot product measures alignment and gives projections; the cross product measures signed area and gives orientation. Nearly every predicate you need is the sign of one of them.
The orientation predicate
Everything else is built on this one function:
long long cross(P o, P a, P b){
return (a.x - o.x)*(b.y - o.y) - (a.y - o.y)*(b.x - o.x);
}
// > 0 : c is to the LEFT of the directed line a→b (counterclockwise turn)
// < 0 : to the RIGHT (clockwise)
// = 0 : collinear
// Twice the signed area of triangle abc. Sign flips if you swap two points.| Question | Answered by |
|---|---|
| Which side of a line is P on? | sign(orient(a,b,p)) |
| Do segments AB and CD cross? | four orientation tests |
| Is a polygon convex? | all consecutive orientations same sign |
| Is the polygon counterclockwise? | signed area > 0 |
| Convex hull | orientation, repeatedly (Chapter 57) |
| Point in convex polygon | binary search on orientation |
| Sort points by angle | half-plane comparison + cross |
Exactness is not optional
The orientation test is a difference of two products of similar magnitude — the classic catastrophic-cancellation shape. For nearly-collinear points the computed sign is arbitrary, and an algorithm built on an inconsistent predicate does not merely produce a slightly wrong answer: convex hull can return a non-convex polygon, and Delaunay triangulation can loop forever.
Use integer coordinates whenever you can. With coordinates bounded by C, the orientation determinant is bounded by 4C² — so 64-bit integers are exact for C ≤ 1.5×10⁹, and there is no error to reason about. When floating point is unavoidable, use an adaptive predicate (compute a fast estimate with an error bound, fall back to exact arithmetic only when the estimate is inconclusive).
Angles and rotations
That last recipe is how you sort points by angle exactly: atan2 is a floating-point function and ties between nearly-equal angles are decided by rounding, whereas the half-plane-plus-cross comparator is exact for integer input and faster.
"Are three points collinear?" — cross product equals zero, in integers, with no division and no tolerance. If the interviewer offers floating-point coordinates, the strong answer names the cancellation problem and asks what the coordinates' provenance is: measured data needs a tolerance and an acknowledgement that the predicate is then non-transitive (Chapter 5).
- Theory
Prove that a× b is twice the signed area of the triangle with vertices at the origin, a and b.
Solution
The parallelogram spanned by a and b has area |a||b|\sinθ where θ is the angle from a to b. Expanding in coordinates with a = |a|(\cosα,\sinα) and b = |b|(\cosβ,\sinβ):
a× b = |a||b|(\cosα\sinβ - \sinα\cosβ) = |a||b|\sin(β-α). ∎
The triangle is half the parallelogram, so its signed area is \frac12(a× b). The sign encodes orientation: positive when b is counterclockwise from a.
Equivalently a× b = \det\begin{smallmatrix}a_x&b_x\\a_y&b_y\end{smallmatrix}, which is Chapter 45's volume interpretation in two dimensions — the determinant is the signed area, and that is why the two subjects share a primitive.
- Theory
Derive the segment-intersection test from orientation predicates, including all degenerate cases.
Solution
Segments AB and CD properly cross iff C and D are on opposite sides of line AB, and A and B are on opposite sides of line CD:
orient(A,B,C)·orient(A,B,D) < 0 and orient(C,D,A)·orient(C,D,B) < 0.
Degenerate cases, where some orientation is 0:
- If orient(A,B,C) = 0, then C is on line AB; the segments touch iff C lies within AB's bounding box.
- Collinear overlapping segments: all four orientations are 0, and they intersect iff their projections onto the dominant axis overlap.
bool on_segment(P a, P b, P p){ return cross(a, b, p) == 0 && min(a.x,b.x) <= p.x && p.x <= max(a.x,b.x) && min(a.y,b.y) <= p.y && p.y <= max(a.y,b.y); }The degenerate cases are the whole difficulty. A test that handles only the proper crossing is correct on random input and wrong on every carefully-constructed test set.
- Theory
Bound the magnitude of the orientation determinant for integer coordinates in [-C, C] and determine the required integer width.
Solution
orient(a,b,c) = (b_x-a_x)(c_y-a_y) - (b_y-a_y)(c_x-a_x).
Each difference is at most 2C in absolute value, so each product is at most 4C² and the difference at most 8C².
C = 10⁴ bound 8×10⁸ int32 (max 2.1×10⁹) — just fits C = 10⁹ bound 8×10¹⁸ int64 (max 9.2×10¹⁸) — just fits C = 10¹⁸ bound 8×10³⁶ needs __int128 or big integersTranslating the points so that a is at the origin (as the formula does) already halves the effective range compared with expanding the full 3×3 determinant, which is why this form is the standard one.
The in-circle predicate (Chapter 45) involves squared distances, so it is bounded by about C⁴ — needing
__int128already at C = 10⁵. Computing this bound before choosing a type is a two-minute exercise that prevents a whole class of silent failures. - Theory
Give an exact comparator for sorting points by polar angle, without trigonometry.
Solution
Split the plane into two half-planes and compare within them by cross product:
int half(P p){ return (p.y < 0 || (p.y == 0 && p.x < 0)) ? 1 : 0; } bool angle_less(P a, P b){ int ha = half(a), hb = half(b); if (ha != hb) return ha < hb; long long c = cross(a, b); if (c != 0) return c > 0; return norm(a) < norm(b); // same direction: shorter first }The
halffunction puts angles in [0,π) before those in [π,2π), breaking the cross product's inability to distinguish θ from θ+π (both give the same sign against a third vector only within a half-plane).Why not
atan2: it is 20–50× slower, and two points at genuinely equal angles may compare unequal due to rounding — which violates the strict weak ordering a sort requires (Chapter 5) and is undefined behaviour in C++.The comparator above is exact for integers and a total preorder, with the tie-break by length making it a strict weak ordering.
- Theory
Show how to test whether a point is inside a convex polygon in O(log n).
Solution
Assume the polygon is given counterclockwise with vertices v_0,…,v_{n-1}. Fan-triangulate from v_0: the wedges (v_0,v_i,v_{i+1}) partition the polygon.
Binary search for the wedge containing p: find the largest i with orient(v_0, v_i, p) ≥ 0. Then p is inside iff orient(v_i, v_{i+1}, p) ≥ 0.
bool inside_convex(const vector<P>& v, P p){ int n = v.size(); if (cross(v[0], v[1], p) < 0 || cross(v[0], v[n-1], p) > 0) return false; int lo = 1, hi = n - 1; while (hi - lo > 1){ int mid = (lo + hi) / 2; if (cross(v[0], v[mid], p) >= 0) lo = mid; else hi = mid; } return cross(v[lo], v[lo+1], p) >= 0; }O(log n) with exact integer arithmetic and no division. For a non-convex polygon there is no such shortcut — ray casting or the winding number is O(n) (Chapter 56).
- Theory
Derive the distance from a point to a line and to a segment.
Solution
To a line through a and b: the area of the triangle is \frac12|{×}| and also \frac12·base·height, so
d = (|(b-a)×(p-a)|)/(|b-a|).
To a segment: project p onto the line and clamp. With t = ((p-a)·(b-a))/(|b-a|²), the nearest point is a + clamp(t,0,1)(b-a).
double dist_to_segment(P a, P b, P p){ if (a == b) return dist(a, p); double t = dot(p-a, b-a) / (double)norm(b-a); t = max(0.0, min(1.0, t)); return dist(p, a + (b-a)*t); }Comparing squared distances avoids the square root entirely when you only need to rank or to compare against a threshold — and for integer input that keeps the whole computation exact.
The a = b guard is not paranoia: degenerate segments occur in real data, and the division by |b-a|² produces NaN that then propagates silently through everything downstream.
- Theory
Explain the adaptive-precision technique for floating-point orientation tests.
Solution
Shewchuk's approach has two stages.
Stage 1: compute the determinant in floating point together with a static error bound derived from the magnitudes of the inputs:
|error| ≤ ε·(|(b_x-a_x)(c_y-a_y)| + |(b_y-a_y)(c_x-a_x)|)· k
for a small constant k. If the computed value exceeds this bound in absolute value, its sign is provably correct — return immediately. This is the fast path, taken well over 99% of the time.
Stage 2: otherwise, recompute using exact floating-point expansions (sequences of doubles summing to the exact value, using error-free transformations like Knuth's two-sum and Dekker's two-product), adding terms only until the sign is determined.
The result is exactness with almost no average cost. This is what CGAL and Triangle use, and it is the reason those libraries are robust where naive implementations are not.
The alternative, when coordinates permit, is simply to use integers — no error, no bound, no fallback.
- Program
Implement a point/vector class with dot, cross, rotation and comparison, in exact integer arithmetic.
Solution
struct P { long long x, y; P operator+(P o) const { return {x+o.x, y+o.y}; } P operator-(P o) const { return {x-o.x, y-o.y}; } P operator*(long long k) const { return {x*k, y*k}; } bool operator==(P o) const { return x==o.x && y==o.y; } bool operator<(P o) const { return x != o.x ? x < o.x : y < o.y; } }; long long dot (P a, P b){ return a.x*b.x + a.y*b.y; } long long cross(P a, P b){ return a.x*b.y - a.y*b.x; } long long cross(P o, P a, P b){ return cross(a-o, b-o); } long long norm2(P a){ return dot(a, a); } // squared length P perp(P a){ return {-a.y, a.x}; } // rotate 90° CCW, exact int sgn(long long v){ return (v > 0) - (v < 0); }Everything is exact for |x|,|y| ≤ 10⁹ in 64-bit.
norm2rather thannormkeeps it that way — take a square root only at the very end, when a real distance is genuinely required as output.The
sgnhelper is worth having: orientation code should branch on signs, never on magnitudes, and writingsgn(cross(...))makes that discipline visible. - Program
Implement full segment intersection: report whether two segments intersect, and the intersection point or overlapping subsegment.
Solution
// returns: 0 = no intersection, 1 = a single point, 2 = a segment int seg_intersect(P a, P b, P c, P d, P& p1, P& p2){ long long d1 = cross(c, d, a), d2 = cross(c, d, b); long long d3 = cross(a, b, c), d4 = cross(a, b, d); if (sgn(d1)*sgn(d2) < 0 && sgn(d3)*sgn(d4) < 0){ // proper crossing — the point is rational, so return it as a fraction // p = a + (b−a)·d3/(d3−d4) p1 = intersect_point(a, b, c, d); return 1; } // degenerate: check each endpoint for lying on the other segment vector<P> hits; if (on_segment(c, d, a)) hits.push_back(a); if (on_segment(c, d, b)) hits.push_back(b); if (on_segment(a, b, c)) hits.push_back(c); if (on_segment(a, b, d)) hits.push_back(d); sort(hits.begin(), hits.end()); hits.erase(unique(hits.begin(), hits.end()), hits.end()); if (hits.empty()) return 0; p1 = hits.front(); p2 = hits.back(); return hits.size() == 1 ? 1 : 2; }The degenerate branch handles touching endpoints, T-junctions and collinear overlaps uniformly by collecting all endpoints that lie on the other segment and taking their extremes.
The intersection point of a proper crossing has rational coordinates even for integer input, so return it as a fraction (or as a double with a documented tolerance) — rounding it to an integer is a common and silently wrong shortcut.
Test cases that matter: shared endpoint, T-junction, collinear disjoint, collinear overlapping, collinear nested, one segment degenerate to a point, both degenerate.
- Program
Sort points by polar angle around a centre, exactly, and verify against
atan2.Solution
def half(p): return 1 if (p[1] < 0 or (p[1] == 0 and p[0] < 0)) else 0 def angle_sort(pts, centre=(0,0)): import functools def rel(p): return (p[0]-centre[0], p[1]-centre[1]) def cmp(p, q): a, b = rel(p), rel(q) if half(a) != half(b): return -1 if half(a) < half(b) else 1 c = a[0]*b[1] - a[1]*b[0] if c: return -1 if c > 0 else 1 na = a[0]**2 + a[1]**2; nb = b[0]**2 + b[1]**2 return (na > nb) - (na < nb) return sorted(pts, key=functools.cmp_to_key(cmp)) # verify: the atan2 order must agree except on exact ties import math sorted_exact = angle_sort(pts) sorted_atan = sorted(pts, key=lambda p: (math.atan2(p[1], p[0]), p[0]**2 + p[1]**2))The two orders agree on random integer input. They diverge on collinear points at the same angle, where
atan2returns bit-identical values and the tie-break decides — which is precisely why the exact comparator includes the length tie-break explicitly.Benchmark: the exact version is typically 10–30× faster than
atan2at n=10⁶, becauseatan2is a genuinely expensive libm call while a cross product is two multiplications. - Program
Determine whether a polygon is convex, and whether it is simple (non-self-intersecting).
Solution
def is_convex(poly): n = len(poly); sign = 0 for i in range(n): c = cross(poly[i], poly[(i+1)%n], poly[(i+2)%n]) if c: if sign and (c > 0) != (sign > 0): return False sign = c return True # all turns the same way def is_simple(poly): n = len(poly) for i in range(n): for j in range(i+1, n): if j == i or (i == 0 and j == n-1): continue # adjacent edges if j == i+1: continue if segments_intersect(poly[i], poly[(i+1)%n], poly[j], poly[(j+1)%n]): return False return Trueis_convexis O(n) and must tolerate collinear vertices (cross product 0) without rejecting — whether collinear points make a polygon "convex" is a definitional choice that should be stated.is_simpleas written is O(n²). The Shamos–Hoey sweep-line algorithm does it in O(nlog n) by detecting the first crossing among the edges (Chapter 58), which is the right answer for large n.Both are worth having as assertions in geometry code: many algorithms silently assume simplicity or convexity, and a violated assumption produces plausible-looking nonsense rather than an error.
- Program
Implement an adaptive orientation predicate: fast float path with an error bound, exact fallback.
Solution
from fractions import Fraction import math EPS_BOUND = 3.3306690738754716e-16 # (3 + 16ε)ε for the 2×2 determinant def orient_adaptive(a, b, c): detleft = (a[0]-c[0]) * (b[1]-c[1]) detright = (a[1]-c[1]) * (b[0]-c[0]) det = detleft - detright # a conservative bound on the rounding error of `det` detsum = abs(detleft) + abs(detright) if abs(det) > EPS_BOUND * detsum: return 1 if det > 0 else -1 # the sign is provably correct # fall back to exact rational arithmetic ex = (Fraction(a[0])-c[0])*(Fraction(b[1])-c[1]) \ - (Fraction(a[1])-c[1])*(Fraction(b[0])-c[0]) return (ex > 0) - (ex < 0)The fast path is taken for essentially all non-degenerate input, so the average cost is that of the plain float version. The exact fallback fires only for genuinely near-collinear triples.
Test it: generate triples that are exactly collinear, then perturb one coordinate by a single ulp. The naive float predicate returns an essentially random sign; the adaptive one is always correct. That experiment — a few lines — is what convinces people this matters.
Shewchuk's production version uses floating-point expansions rather than
Fraction, making the exact path perhaps 10× slower than the fast path instead of 1000×. For contest use, integers remain the better answer wherever the input permits. - Program
Given n points, count the triples that are collinear, in O(n²log n).
Solution
from math import gcd from collections import Counter def count_collinear_triples(pts): n = len(pts); total = 0 for i in range(n): slopes = Counter() for j in range(n): if i == j: continue dx = pts[j][0]-pts[i][0]; dy = pts[j][1]-pts[i][1] g = gcd(abs(dx), abs(dy)) or 1 dx //= g; dy //= g if dx < 0 or (dx == 0 and dy < 0): dx, dy = -dx, -dy # canonical slopes[(dx, dy)] += 1 for c in slopes.values(): total += c*(c-1)//2 # triples with i as one vertex return total // 3 # each triple counted 3 timesNormalising the direction by dividing out the gcd and fixing a canonical sign makes the slope key exact — no floating-point slope, no tolerance, no collisions between distinct directions. That canonicalisation is Chapter 5's "equivalence by canonical form" applied to directions.
O(n²log C) from the gcd calls, with O(n) memory per outer iteration. Dividing by 3 at the end is because each collinear triple is discovered once from each of its three points.
Using
dy/dxas a float key instead would merge distinct steep slopes and split equal ones — a bug that passes small tests and fails on adversarial input. - Program
Implement a geometry test harness that checks predicate consistency on random and degenerate inputs.
Solution
import random, itertools def check_orientation_consistency(orient, trials=100000): problems = [] for _ in range(trials): pts = [(random.randint(-5,5), random.randint(-5,5)) for _ in range(3)] a, b, c = pts # antisymmetry: swapping two arguments must flip the sign if orient(a,b,c) != -orient(b,a,c): problems.append(('antisym', pts)) # cyclic symmetry: rotating the arguments must preserve it if orient(a,b,c) != orient(b,c,a): problems.append(('cyclic', pts)) # translation invariance t = (random.randint(-100,100), random.randint(-100,100)) sh = [(p[0]+t[0], p[1]+t[1]) for p in pts] if orient(a,b,c) != orient(*sh): problems.append(('translate', pts)) return problemsThese three invariants — antisymmetry, cyclic symmetry and translation invariance — are exactly the properties that downstream algorithms rely on, and a floating-point predicate violates all three on near-degenerate input.
Deliberately including degenerate cases (small integer ranges produce many collinear triples) is the point: random real-valued points are almost never degenerate, so a test suite built from them exercises none of the code paths where geometry actually breaks.
Add the same style of check for the in-circle predicate (invariant under rotation and translation, sign-flipping under reflection), and for the segment-intersection routine (symmetric in its two segments, and consistent with the endpoints).
Lines, half-planes and intersection
Three representations of a line - two points, point plus direction, and the implicit ax+by=c - each make different operations easy. Choosing the right one and staying exact is most of what line geometry requires.
Line intersection
Two lines given by points intersect at a point with a clean cross-product formula:
// Line 1 through a, b; Line 2 through c, d.
// Solve a + t(b−a) = c + s(d−c).
// t = cross(c − a, d − c) / cross(b − a, d − c)
//
// The denominator is zero exactly when the lines are PARALLEL.
// The intersection is then rational, not integral — keep it as a fraction,
// or accept a double with a documented tolerance.
pair<long long,long long> intersect_t(P a, P b, P c, P d){
long long den = cross(b - a, d - c);
if (den == 0) return {0, 0}; // parallel or collinear
long long num = cross(c - a, d - c);
return {num, den}; // t = num/den
}An intersection point of integer-coordinate lines is rational, not integer. Rounding it destroys exactness and breaks every downstream predicate. Either carry it as a fraction (numerator and denominator, reduced), or restructure the algorithm so the intersection point is never needed — many algorithms only need to compare intersections, which is a cross-multiplication and stays integral.
Half-planes
A half-plane is {p : cross(d, p - o) ≥ 0} — the points to the left of a directed line. The intersection of n half-planes is a convex region (possibly empty or unbounded), computable in O(nlog n) by sorting the half-planes by angle and maintaining a deque.
Line arrangements and duality
The point–line duality (a,b) rightarrow y = ax - b exchanges points and lines while preserving incidence and the above/below relation. It converts problems about points into problems about lines and back:
| Primal | Dual |
|---|---|
| Three collinear points | Three concurrent lines |
| Convex hull | Upper/lower envelope of lines |
| Point above a line | Line below a point |
| Minimum-area triangle | A vertex of the arrangement |
The convex-hull–to–envelope duality is the useful one in practice: it is exactly why the "convex hull trick" for DP optimisation (Chapter 64) is called that.
"Do two line segments intersect?" wants the four orientation tests plus the collinear cases (Chapter 54). "Where do they intersect?" is the follow-up that separates people: the answer is rational, and the honest response asks whether the caller needs the exact point or only a comparison — because those have very different implementations.
- Theory
Derive the line–line intersection formula and identify all degenerate cases.
Solution
Set a + t(b-a) = c + s(d-c). Cross both sides with (d-c) to eliminate s:
(a-c)×(d-c) + t(b-a)×(d-c) = 0, so t = ((c-a)×(d-c))/((b-a)×(d-c)).
Degenerate cases, all signalled by the denominator being zero:
- cross(b-a, d-c) = 0 and cross(c-a, d-c) ≠ 0: parallel, no intersection.
- Both zero: the lines are identical — infinitely many intersections.
- a = b or c = d: a degenerate "line" that is a point; the formula is meaningless and must be handled separately.
Distinguishing "parallel" from "identical" requires the second test, and code that returns only a boolean "no intersection" for both is wrong for the identical case — a distinction that matters in half-plane intersection and in polygon clipping.
- Theory
Prove that the intersection of half-planes is convex, and bound the number of vertices.
Solution
Each half-plane is convex (if p and q satisfy a· x ≤ c then so does any convex combination, by linearity). An intersection of convex sets is convex: if p and q are in every set, so is every point between them. ∎
Vertex bound: each vertex of the region is the intersection of two boundary lines, so there are at most \binom n2 candidates — but the region is convex, so its boundary visits each half-plane's line at most once, giving at most n vertices.
Consequently half-plane intersection is O(nlog n) (dominated by the angular sort), not O(n²). The full arrangement of n lines, by contrast, does have Θ(n²) vertices, faces and edges — the difference between the intersection of the half-planes and the subdivision induced by all the lines.
- Theory
State the point–line duality and prove that it preserves the above/below relation.
Solution
Map the point p = (a,b) to the line p^* : y = ax - b, and the line \ell : y = mx + k to the point \ell^* = (m, -k). The map is an involution.
Incidence: p lies on \ell iff b = ma + k. And \ell^* = (m,-k) lies on p^*: y = ax - b iff -k = am - b, i.e. b = am + k — the same condition. ∎
Above/below: p is above \ell iff b > ma+k, iff -k > am - b, iff \ell^* is above p^*. So the relation is preserved (not reversed) under this particular duality. ∎
Consequence: the upper convex hull of a point set dualises to the lower envelope of the corresponding lines. That is precisely the convex hull trick — maintaining the maximum of a set of linear functions is the same problem as maintaining a convex hull, and both admit the same O(log n) query.
- Theory
Show how to clip a segment to a convex polygon (the Cyrus–Beck algorithm).
Solution
Parametrise the segment as p(t) = a + t(b-a), t∈[0,1]. For each edge of the convex polygon with inward normal n and a point q on it, the constraint is n·(p(t)-q) ≥ 0, i.e.
n·(a-q) + tn·(b-a) ≥ 0 — a linear inequality in t.
Each edge therefore gives a half-line of valid t: if n·(b-a) > 0 it is a lower bound, if < 0 an upper bound, and if = 0 the segment is parallel to the edge and either always or never satisfies it.
Intersect all the constraints with [0,1]: the result is an interval [t_{lo}, t_{hi}], empty iff the segment misses the polygon. ∎
O(n) for an n-gon, and it generalises to any convex region defined by linear constraints. The special case of an axis-aligned rectangle is the Liang–Barsky algorithm used in every graphics pipeline.
- Theory
Explain how to solve a 2D linear program using half-plane intersection, and state the complexity.
Solution
Minimise c· x subject to n linear constraints. The feasible region is the intersection of n half-planes, a convex polygon with at most n vertices, and the optimum is attained at a vertex (or along an edge, or is unbounded).
Approach 1: compute the half-plane intersection in O(nlog n), then scan its O(n) vertices.
Approach 2 (better): Seidel's randomised incremental algorithm — add constraints in random order, maintaining the optimum. When a new constraint is violated, the new optimum lies on that constraint's line, so re-solve a 1D LP on it in O(i). The probability that constraint i is violated is O(1/i) (backwards analysis: it is violated only if it is one of the two defining the optimum), so the expected cost is
∑_i (2)/(i)· O(i) = O(n). ∎
Expected linear time, and it generalises to fixed dimension d with cost O(d!n) — which is why linear programming in low dimensions is genuinely easy while the general case needs simplex or interior-point methods.
- Theory
Given n lines, how many regions do they divide the plane into?
Solution
In general position (no two parallel, no three concurrent), adding the k-th line crosses the previous k-1 lines at k-1 points, which cut it into k pieces — each piece splitting one existing region in two. So the count satisfies R_k = R_{k-1} + k with R_0 = 1:
R_n = 1 + ∑_{k=1}^{n}k = 1 + C(n+1,2) = (n²+n+2)/(2).
Equivalently R_n = \binom n0 + \binom n1 + \binom n2 — the Euler-characteristic form, which generalises to ∑_{k≤ d}\binom nk regions for n hyperplanes in ℝ^d. ∎
Degeneracies only reduce the count: parallel lines miss an intersection, concurrent lines share one. So the general-position formula is an upper bound, and computing the exact count for a specific arrangement requires the actual incidence structure.
The arrangement has \binom n2 vertices, n² edges and Θ(n²) faces — which is why algorithms that build the full arrangement are quadratic and why the half-plane intersection (linear) is such a different problem.
- Theory
Derive the reflection of a point across a line, and the closest point on a line.
Solution
Let the line pass through a with direction d = b-a, and let v = p-a.
Projection: proj = (v· d)/(d· d)d, so the closest point is a + proj.
Reflection: p' = a + 2proj - v = 2(a + proj) - p — go to the foot of the perpendicular and continue the same distance.
def closest_on_line(a, b, p): d = sub(b, a); v = sub(p, a) t = dot(v, d) / dot(d, d) return add(a, mul(d, t)) def reflect(a, b, p): c = closest_on_line(a, b, p) return sub(mul(c, 2), p)Both are exact in rationals; in integers the projection is generally not an integer point.
Application: the shortest path touching a line (the "billiard" or mirror problem) is found by reflecting the destination across the line and taking a straight line — a trick that also solves "minimise |PA|+|PB| for P on a line" instantly.
- Program
Implement exact line intersection returning a rational point, with all degenerate cases.
Solution
from fractions import Fraction def line_intersect(a, b, c, d): """returns ('point', (x,y)) | ('parallel',) | ('same',) | ('degenerate',)""" if a == b or c == d: return ('degenerate',) r = (b[0]-a[0], b[1]-a[1]) s = (d[0]-c[0], d[1]-c[1]) den = r[0]*s[1] - r[1]*s[0] ac = (c[0]-a[0], c[1]-a[1]) if den == 0: return ('same',) if ac[0]*r[1] - ac[1]*r[0] == 0 else ('parallel',) t = Fraction(ac[0]*s[1] - ac[1]*s[0], den) return ('point', (a[0] + t*r[0], a[1] + t*r[1]))Returning a tagged result rather than a nullable point forces the caller to handle the three geometric cases, which is where these routines are usually misused.
Fractionkeeps the answer exact. If the caller only needs to compare two intersection points (as in a sweep line), skip the fraction entirely and cross-multiply — that stays in integers and is far faster.Test with: crossing lines, parallel distinct, identical, a shared point, and degenerate inputs. Each is a separate branch and each occurs in real data.
- Program
Implement half-plane intersection in O(nlog n) and use it to find the area of a feasible region.
Solution
from collections import deque def half_plane_intersection(planes): """each plane: (point, direction); the region is to the LEFT of each""" planes.sort(key=lambda h: angle_key(h[1])) dq = deque() for h in planes: while len(dq) >= 2 and not left_of(h, inter(dq[-2], dq[-1])): dq.pop() while len(dq) >= 2 and not left_of(h, inter(dq[0], dq[1])): dq.popleft() if dq and cross(dq[-1][1], h[1]) == 0: # same direction if left_of(dq[-1], h[0]): continue # h is redundant dq.pop() dq.append(h) while len(dq) >= 3 and not left_of(dq[0], inter(dq[-2], dq[-1])): dq.pop() while len(dq) >= 3 and not left_of(dq[-1], inter(dq[0], dq[1])): dq.popleft() if len(dq) < 3: return [] # empty or unbounded return [inter(dq[i], dq[(i+1) % len(dq)]) for i in range(len(dq))]Three subtleties account for most bugs here. The parallel same-direction case must keep only the tighter constraint. The final closing loop is required because the deque's two ends must be checked against each other. And an unbounded region produces fewer than three half-planes surviving — which must be reported, not treated as empty.
Add a large bounding box (four half-planes at ±10⁹) up front to guarantee boundedness; then a genuinely unbounded feasible region shows up as a polygon touching the box, which is usually what you want anyway.
- Program
Implement Seidel's randomised algorithm for 2D linear programming.
Solution
import random def lp2d(constraints, cx, cy): """maximise cx·x + cy·y subject to a·x + b·y <= c""" random.shuffle(constraints) # start at a large bounding box vertex in the objective direction best = (BIG if cx > 0 else -BIG, BIG if cy > 0 else -BIG) for i, (a, b, c) in enumerate(constraints): if a*best[0] + b*best[1] <= c: continue # still feasible # the new optimum lies ON this line: solve a 1D LP along it best = solve_1d(constraints[:i], (a, b, c), cx, cy) if best is None: return None # infeasible return bestWhy it is linear: by backwards analysis, after processing i constraints the optimum is determined by at most 2 of them, so a randomly-ordered constraint i is violated with probability ≤ 2/i. The re-solve costs O(i), so the expected total is ∑_i \frac2i· O(i) = O(n). ∎
The shuffle is essential — with adversarial ordering the algorithm is Θ(n²), and the input order in a real problem is very often adversarial by accident (sorted, or grouped by source).
The 1D subproblem is intersecting intervals on a line, which is a scan for the tightest lower and upper bounds — a few lines, and the place where infeasibility is detected.
- Program
Clip a polygon against a convex polygon (Sutherland–Hodgman) and against a half-plane.
Solution
def clip_halfplane(poly, a, b): """keep the part of `poly` to the LEFT of the directed line a→b""" out = [] n = len(poly) for i in range(n): cur, nxt = poly[i], poly[(i+1) % n] c1 = cross(a, b, cur) c2 = cross(a, b, nxt) if c1 >= 0: out.append(cur) if (c1 > 0) != (c2 > 0) and c1 != c2: out.append(line_point(a, b, cur, nxt)) # the crossing point return out def clip_convex(subject, window): for i in range(len(window)): subject = clip_halfplane(subject, window[i], window[(i+1) % len(window)]) if not subject: break return subjectSutherland–Hodgman clips against one edge at a time, so it is O(nm) for an n-gon against an m-gon. It requires the window to be convex; the subject may be any simple polygon, though a concave subject can produce degenerate connecting edges.
The output vertices are rational. For an area computation that is fine (the shoelace formula handles fractions), but feeding the result back into an integer-coordinate algorithm requires care.
For convex-against-convex, the O(n+m) rotating-calipers merge is better and is what computational-geometry libraries use.
- Program
Use duality to convert a "maximum points on a line" problem into an arrangement problem.
Solution
# Primal: given n points, find the line containing the most of them. # Dual: each point (a,b) → line y = ax − b. Collinear points become # CONCURRENT lines, so the answer is the arrangement vertex of # highest degree. def max_collinear_dual(points): lines = [(a, -b) for a, b in points] # y = ax + (−b) from collections import Counter best = 0 for i in range(len(lines)): cnt = Counter() for j in range(len(lines)): if i == j: continue m1, k1 = lines[i]; m2, k2 = lines[j] if m1 == m2: continue # parallel: no intersection # the intersection x-coordinate, as an exact fraction from fractions import Fraction cnt[Fraction(k2 - k1, m1 - m2)] += 1 best = max(best, max(cnt.values(), default=0) + 1) return bestBoth the primal (group by slope from each point, Chapter 54 problem 13) and the dual (group by intersection x-coordinate) are O(n²log n) — the duality does not improve this particular problem, and that is worth knowing.
Where duality genuinely helps: problems asking about all lines through pairs of points, or about the levels of an arrangement (the k-th smallest line at each x), where the dual formulation admits a sweep that the primal does not. And the minimum-area triangle among n points is solved in O(n²) by sweeping the dual arrangement — the primal has no comparable algorithm.
- Program
Solve: given n points and a query line, count the points on each side, in O(log n) after preprocessing.
Solution
# Preprocess: build the convex layers, or use a partition tree. # Simple practical approach for many queries: sort by angle around a pivot # and answer with binary search — O(n log n) preprocessing, O(log n) per query # for lines through the pivot. For ARBITRARY lines, use a partition tree: # # Matoušek's simplicial partition gives O(n) space and O(√n) query time # for half-plane range counting — provably near-optimal. # # For offline queries, the dual is better: # each point → a line; each query line → a point; # "points below the query line" = "dual lines above the dual point", # answerable by a sweep over the arrangement.The honest answer is that O(log n) per query is not achievable for arbitrary half-plane counting with subquadratic space — the known lower bounds put the trade-off at roughly Q(n)· S(n) = Ω(n), so O(√n) query with O(n) space (partition trees) or O(log n) query with O(n²) space (the full arrangement with a point-location structure) are the two ends.
Recognising that a problem's stated bound is impossible is itself a useful skill. In practice the answers are: precompute the arrangement if queries vastly outnumber points; use a partition tree if space matters; or process offline with a sweep if all queries are known in advance, which is usually the case in a contest.
- Program
Implement a robust "point on segment" and "segments overlap" test, and fuzz it against a brute-force rational check.
Solution
from fractions import Fraction import random def on_segment(a, b, p): return (cross(a, b, p) == 0 and min(a[0],b[0]) <= p[0] <= max(a[0],b[0]) and min(a[1],b[1]) <= p[1] <= max(a[1],b[1])) def brute_on_segment(a, b, p): """exact: is p = a + t(b−a) for some t in [0,1]?""" if a == b: return p == a dx, dy = b[0]-a[0], b[1]-a[1] if dx: t = Fraction(p[0]-a[0], dx) else: t = Fraction(p[1]-a[1], dy) return 0 <= t <= 1 and a[0] + t*dx == p[0] and a[1] + t*dy == p[1] for _ in range(200000): a, b, p = [(random.randint(-3,3), random.randint(-3,3)) for _ in range(3)] assert on_segment(a,b,p) == brute_on_segment(a,b,p), (a,b,p)The small coordinate range is deliberate: it produces a high density of degenerate cases — collinear triples, coincident points, zero-length segments, axis-aligned segments — which are exactly the inputs where geometry code fails and which random real-valued points never generate.
The brute-force version is obviously correct but slow and awkward; the fast version is subtle. Fuzzing one against the other over 200,000 degenerate-rich cases is a far stronger guarantee than any set of hand-written test cases, and it takes three lines to set up.
Polygons: area, centroid and point location
The shoelace formula computes a polygon's signed area in one pass, and its sign tells you the orientation. Point-in-polygon is a ray cast whose whole difficulty is degenerate cases, and Pick's theorem links area to lattice points exactly.
The shoelace formula is a sum of triangle areas from the origin, with signs cancelling the parts outside the polygon. That is why it works for non-convex polygons and why the sign encodes orientation — it is Chapter 45's determinant summed around the boundary.
Point in polygon
bool inside(const vector<P>& poly, P p){
int n = poly.size();
bool in = false;
for (int i = 0, j = n-1; i < n; j = i++){
P a = poly[j], b = poly[i];
if (on_segment(a, b, p)) return true; // ON the boundary
// does the upward ray from p cross edge (a,b)?
// the half-open test [min, max) avoids double-counting vertices
if ((a.y > p.y) != (b.y > p.y)){
// x-coordinate of the crossing, compared without division
long long t = (b.x - a.x) * (p.y - a.y) - (p.x - a.x) * (b.y - a.y);
if ((b.y > a.y) ? (t > 0) : (t < 0)) in = !in;
}
}
return in;
}The correctness of ray casting rests entirely on the half-open comparison (a.y > p.y) != (b.y > p.y). It counts an edge exactly when the ray passes its lower endpoint but not its upper one — so a ray through a vertex crosses exactly one of the two incident edges, and a horizontal edge is never counted. Every other formulation needs special cases; this one needs none.
Pick's theorem
So I = A - B/2 + 1 counts interior lattice points in O(nlog C) — no enumeration, however large the polygon. Combined with the shoelace formula it is entirely exact.
"Compute a polygon's area" expects the shoelace formula in one pass with no triangulation. The follow-up worth volunteering: use twice the area as an integer and divide only at the end, so integer input gives an exact answer — polygon areas are always half-integers, and returning 2A avoids floating point entirely.
- Theory
Prove the shoelace formula.
Solution
Triangulate the polygon by connecting every vertex to the origin. The signed area of triangle (O, v_i, v_{i+1}) is \frac12(x_iy_{i+1}-x_{i+1}y_i) (Chapter 54).
Summing around the boundary, the triangles outside the polygon are traversed in the opposite orientation and contribute negatively, cancelling exactly the parts counted spuriously. The net is the polygon's signed area. ∎
A cleaner argument is Green's theorem: A = \frac12\oint(xdy - ydx), and evaluating the line integral along each straight edge gives exactly the shoelace term. That version makes the "any simple polygon" generality obvious, since Green's theorem needs only a simple closed curve.
The formula is translation invariant despite involving the origin — shifting all vertices by t changes each term but not the sum, which is a good invariant to assert in tests.
- Theory
Prove Pick's theorem for a triangle, then sketch the general case.
Solution
Additivity: if a polygon is split into two by a diagonal, and Pick's formula holds for each part, it holds for the whole. Let the diagonal contain d interior lattice points plus its two endpoints. Then I = I_1+I_2+d and B = B_1+B_2-2d-2, and
A_1+A_2 = (I_1+\tfrac{B_1}2-1)+(I_2+\tfrac{B_2}2-1) = I + \tfrac B2 - 1 ✓.
Base cases: for a lattice rectangle with sides w, h the formula gives (w-1)(h-1) + (2w+2h)/(2) - 1 = wh ✓. A right triangle is half a rectangle, and the diagonal's lattice points make the halving work out. A general lattice triangle decomposes into a rectangle minus right triangles.
General polygon: every simple polygon can be triangulated with diagonals, and additivity propagates the result. ∎
Pick's theorem is specific to two dimensions — the analogous statement fails in 3D (the Reeve tetrahedra have the same lattice-point counts but different volumes), which is why Ehrhart theory (Chapter 52) is the right generalisation.
- Theory
Count the lattice points on the segment from (x_1,y_1) to (x_2,y_2).
Solution
The lattice points strictly between the endpoints are those of the form (x_1,y_1) + t(Δ x,Δ y) with t∈(0,1) rational and both coordinates integral. Writing g = gcd(|Δ x|,|Δ y|), the primitive step is (Δ x/g, Δ y/g), so there are exactly g-1 interior lattice points and g+1 including both endpoints. ∎
For a polygon, summing g over all edges counts each vertex once (as the endpoint of one edge), giving
B = ∑_{edges}gcd(|Δ x|,|Δ y|).
This is exactly the boundary count Pick's theorem needs, computable in O(nlog C). The degenerate case Δ x = Δ y = 0 (a repeated vertex) gives gcd(0,0) = 0 by convention, which is the right answer for a zero-length edge.
- Theory
Explain why the half-open comparison in ray casting handles vertices and horizontal edges correctly.
Solution
The test
(a.y > p.y) != (b.y > p.y)treats the edge's y-interval as half-open: an edge is considered iff exactly one endpoint is strictly above p_y.- A vertex at the ray's height: the two edges meeting there share that vertex. If the polygon passes through the level (one edge up, one down), exactly one edge has its upper endpoint above and is counted — one crossing, correct. If the polygon merely touches the level (a local max or min), both edges are on the same side, neither is counted — zero crossings, also correct.
- A horizontal edge has a_y = b_y, so the two comparisons agree and it is never counted — correct, since a horizontal edge at the ray's level does not constitute a crossing.
Every other formulation (using ≥, or comparing to a midpoint) requires enumerating these cases explicitly. This one gets them all from a single asymmetry, which is why it is the standard. ∎
- Theory
Derive the centroid formula for a polygon and explain why it differs from the average of the vertices.
Solution
The centroid of the area is (1)/(A)\int\int_P xdA. Decomposing into triangles from the origin, triangle i has area a_i = \frac12(x_iy_{i+1}-x_{i+1}y_i) and centroid \frac13(O + v_i + v_{i+1}). Weighting by signed area:
C = (1)/(A)∑_i a_i·\frac{v_i+v_{i+1}}{3} = (1)/(6A)∑_i (v_i+v_{i+1})(x_iy_{i+1}-x_{i+1}y_i). ∎
Why not the vertex average: the vertex average weights every vertex equally, which is the centroid of point masses at the vertices — a different object. A square with one side subdivided into 100 vertices has the same area centroid but a wildly different vertex average.
Three distinct centroids exist for a polygon: of the vertices, of the boundary (weighted by edge length), and of the area. Problems rarely say which, and they differ; the area centroid is almost always the intended one.
- Theory
Given a convex polygon and a query point, describe an O(log n) test and explain why non-convex polygons need O(n).
Solution
For a convex polygon, binary search on the fan from v_0 (Chapter 54 problem 5): O(log n).
For a non-convex polygon, no such monotone structure exists — the boundary can wind arbitrarily, so a query point's status is not determined by a logarithmic number of orientation tests against the vertices. Ray casting is O(n) per query.
But with preprocessing, point location in any planar subdivision is O(log n) per query using O(n) space: build a trapezoidal decomposition (randomised incremental, expected O(nlog n)) or Kirkpatrick's triangulation hierarchy. So the O(n) is a statement about the unpreprocessed case only.
The practical rule: one query, ray cast; many queries against a fixed polygon, build a trapezoidal map or, for a monotone polygon, a simple chain decomposition giving O(log n) with much less machinery.
- Theory
Prove that every simple polygon can be triangulated using n-3 diagonals, producing n-2 triangles.
Solution
Existence of a diagonal: take the leftmost vertex v (lowest, breaking ties). Its two neighbours a and b form a triangle avb. If no other vertex lies inside avb, then ab is a diagonal. Otherwise take the vertex inside avb farthest from line ab; the segment from v to it is a diagonal. ∎
Induction: a diagonal splits the polygon into two simple polygons with k+1 and n-k+1 vertices. By induction they use k-2 and n-k-1 diagonals, and adding the splitting one gives n-3. Triangle count follows from Chapter 3's internal-node identity, or directly by induction. ∎
Algorithms: ear clipping is O(n²) and simple; monotone-polygon decomposition plus triangulation is O(nlog n) and standard; Chazelle's algorithm is O(n) and famously unimplementable.
Corollary (the art gallery theorem): 3-colour the triangulation's vertices; the least-used colour class has at most ⌊ n/3⌋ vertices and guards the whole polygon.
- Program
Implement the shoelace formula, centroid and perimeter, returning exact values where possible.
Solution
from fractions import Fraction import math def area2(poly): """returns TWICE the signed area — an exact integer for integer input""" n = len(poly); s = 0 for i in range(n): x1, y1 = poly[i]; x2, y2 = poly[(i+1) % n] s += x1*y2 - x2*y1 return s def centroid(poly): a2 = area2(poly) if a2 == 0: return None # degenerate polygon cx = cy = Fraction(0) n = len(poly) for i in range(n): x1, y1 = poly[i]; x2, y2 = poly[(i+1) % n] cr = x1*y2 - x2*y1 cx += Fraction((x1 + x2) * cr) cy += Fraction((y1 + y2) * cr) return (cx / (3*a2), cy / (3*a2)) def perimeter(poly): n = len(poly) return sum(math.dist(poly[i], poly[(i+1) % n]) for i in range(n))Returning
area2rather than the area keeps integer input exact — polygon areas are half-integers, so twice the area is always an integer. Divide by 2 only when producing output.The centroid is rational and
Fractionkeeps it exact. The perimeter is the only quantity genuinely requiring floating point, since square roots of integers are irrational.Test invariants: area is translation-invariant; reversing the vertex order negates
area2; the centroid of a rectangle is its middle; and the centroid must lie inside a convex polygon. - Program
Implement point-in-polygon by ray casting and by the winding number; compare on a self-intersecting polygon.
Solution
def ray_cast(poly, p): n = len(poly); inside = False for i in range(n): a, b = poly[i], poly[(i+1) % n] if on_segment(a, b, p): return 'boundary' if (a[1] > p[1]) != (b[1] > p[1]): t = cross(a, b, p) if (b[1] > a[1]) == (t > 0): inside = not inside return 'inside' if inside else 'outside' def winding_number(poly, p): n = len(poly); wn = 0 for i in range(n): a, b = poly[i], poly[(i+1) % n] if a[1] <= p[1]: if b[1] > p[1] and cross(a, b, p) > 0: wn += 1 else: if b[1] <= p[1] and cross(a, b, p) < 0: wn -= 1 return wnFor a simple polygon the two agree: winding number is ±1 inside and 0 outside.
For a self-intersecting polygon they differ, and the difference is meaningful. Ray casting implements the even–odd rule (a doubly-wound region counts as outside); the winding number implements the non-zero rule (it counts as inside). Both are used in graphics — SVG's
fill-rulelets you choose — and neither is more correct; they answer different questions.Constructing a pentagram and querying its centre demonstrates the divergence immediately: even–odd says outside, non-zero says inside.
- Program
Use Pick's theorem to count lattice points inside a polygon with coordinates up to 10⁹.
Solution
from math import gcd def lattice_points(poly): n = len(poly) a2 = abs(area2(poly)) # 2A, exact integer B = sum(gcd(abs(poly[i][0] - poly[(i+1)%n][0]), abs(poly[i][1] - poly[(i+1)%n][1])) for i in range(n)) # Pick: A = I + B/2 − 1 ⟹ I = (2A − B + 2)/2 I = (a2 - B + 2) // 2 return I, B, I + BExact and O(nlog C) — completely independent of the polygon's size, so a triangle with vertices at 10⁹ containing 10^{17} lattice points is handled instantly.
Preconditions: the polygon must be simple (Pick's theorem fails for self-intersecting ones) and have integer vertices. Assert both; the second is easy to violate when the polygon came from an intersection computation (Chapter 55).
The division by 2 is exact because 2A - B is always even for a lattice polygon — a small fact worth asserting, since a violation indicates non-integer vertices or a non-simple polygon.
- Program
Implement ear-clipping triangulation of a simple polygon.
Solution
def triangulate(poly): n = len(poly) idx = list(range(n)) if area2(poly) < 0: idx.reverse() # ensure counterclockwise tris = [] guard = 0 while len(idx) > 3: found = False for k in range(len(idx)): i0, i1, i2 = idx[k-1], idx[k], idx[(k+1) % len(idx)] a, b, c = poly[i0], poly[i1], poly[i2] if cross(a, b, c) <= 0: continue # reflex vertex, not an ear if any(point_in_triangle(poly[j], a, b, c) for j in idx if j not in (i0, i1, i2)): continue tris.append((i0, i1, i2)) idx.pop(k); found = True; break assert found, "no ear found — polygon is not simple" guard += 1 tris.append(tuple(idx)) return trisO(n²) — each of n-2 ear removals scans for an ear and checks containment. The two-ears theorem guarantees an ear always exists for a simple polygon with n\ge4, so the assertion is a genuine simplicity check rather than defensive padding.
Both conditions are required: convexity of the vertex (cross > 0) and emptiness of the triangle. A convex vertex whose triangle contains another vertex is not an ear, and skipping that check produces overlapping triangles on non-convex input.
For production use, monotone decomposition gives O(nlog n); ear clipping is chosen for its simplicity and because it handles holes with a standard bridging preprocessing step.
- Program
Compute the area of the union of n axis-aligned rectangles in O(nlog n).
Solution
# Sweep line over x, maintaining the total covered y-length with a # "count and cover" segment tree over the compressed y-coordinates. def rectangle_union_area(rects): events = [] ys = set() for x1, y1, x2, y2 in rects: events.append((x1, +1, y1, y2)) events.append((x2, -1, y1, y2)) ys.add(y1); ys.add(y2) ys = sorted(ys) tree = CoverTree(ys) # supports add(y1,y2,±1), covered() events.sort() area = 0; prev_x = events[0][0] for x, delta, y1, y2 in events: area += tree.covered() * (x - prev_x) tree.add(y1, y2, delta) prev_x = x return areaThe segment tree stores, per node, a coverage count and the covered length of its interval. Because ranges are only ever added and removed in matched pairs, no lazy propagation is needed — a node with positive count reports its full length, otherwise it reports the sum of its children. That specialisation is what makes this classic structure so short.
O(nlog n). The same sweep computes the union's perimeter (track the number of boundary segments), the area covered at least k times (store per-count lengths), and the maximum overlap depth.
For non-axis-aligned polygons the analogous problem is much harder and is solved by a general polygon-clipping library (Vatti, Greiner–Hormann) rather than a sweep.
- Program
Given a polygon and a line, compute the area on each side of the line.
Solution
def area_split_by_line(poly, a, b): """returns (area left of a→b, area right)""" left = clip_halfplane(poly, a, b) # Chapter 55 right = clip_halfplane(poly, b, a) # the reversed line return abs(area2(left))/2, abs(area2(right))/2Clipping to each half-plane and taking areas is correct for a convex polygon, where each half is a single connected piece.
For a non-convex polygon the line may cut it into several components, and Sutherland–Hodgman produces a single vertex list with degenerate connecting edges along the cut line. The shoelace formula still gives the correct total area, because those degenerate edges enclose zero area — so the computation works even though the polygon returned is not a faithful representation of the region.
That is worth knowing precisely: the area is right, the polygon is not. If you need the actual components, use a proper polygon-clipping algorithm that returns a list of loops.
Application: binary search on the line's offset to find the cut that bisects the area — a common contest problem, and the ham-sandwich theorem's one-dimensional case.
- Program
Verify Pick's theorem by brute force on random small lattice polygons.
Solution
import random from math import gcd def brute_lattice_count(poly, lim=12): I = B = 0 n = len(poly) for x in range(-lim, lim+1): for y in range(-lim, lim+1): p = (x, y) on = any(on_segment(poly[i], poly[(i+1)%n], p) for i in range(n)) if on: B += 1 elif ray_cast(poly, p) == 'inside': I += 1 return I, B for _ in range(500): poly = random_simple_lattice_polygon(lim=8) I, B = brute_lattice_count(poly) a2 = abs(area2(poly)) assert a2 == 2*I + B - 2, (poly, a2, I, B)Generating random simple lattice polygons is itself the hard part: pick random points and take the convex hull for a guaranteed-simple convex case, or use the 2-opt "uncross" heuristic (repeatedly swap crossing edges) to make a random permutation simple.
The assertion in the form 2A = 2I + B - 2 keeps everything integral — no division, no rounding, an exact equality that either holds or does not.
Running it on deliberately non-simple polygons shows the theorem failing, which is a good way to internalise that the simplicity hypothesis is load-bearing rather than technical.
Convex hull
The smallest convex set containing the points, computable in O(n log n) by sorting and two linear scans. It is the gateway to diameter, width, minimum enclosing shapes, and the convex hull trick for DP.
The convex hull of a point set is the smallest convex polygon containing all of them — equivalently the intersection of all half-planes containing the set, or the set of all convex combinations.
Andrew's monotone chain
vector<P> hull(vector<P> p){
sort(p.begin(), p.end()); // by x, then y
p.erase(unique(p.begin(), p.end()), p.end());
int n = p.size(), k = 0;
if (n < 3) return p;
vector<P> h(2*n);
for (int i = 0; i < n; i++){ // lower hull
while (k >= 2 && cross(h[k-2], h[k-1], p[i]) <= 0) k--;
h[k++] = p[i];
}
for (int i = n-2, t = k+1; i >= 0; i--){ // upper hull
while (k >= t && cross(h[k-2], h[k-1], p[i]) <= 0) k--;
h[k++] = p[i];
}
h.resize(k-1); // drop the repeated start
return h; // counterclockwise
}Two decisions in that code carry all the subtlety. Sorting by (x,y) rather than by angle avoids atan2 entirely and makes the algorithm exact for integer input. And the comparison <= 0 versus < 0 decides whether collinear boundary points are kept — <= 0 drops them, < 0 keeps them, and which you want depends on the problem.
Decide explicitly whether collinear points belong on the hull. For area and diameter they are harmless either way; for "how many points lie on the hull boundary" the answer changes; and for rotating calipers, keeping them can cause infinite loops. Choose, and write a comment saying which.
What the hull unlocks
| Problem | Method | Cost after the hull |
|---|---|---|
| Diameter (farthest pair) | rotating calipers | O(h) |
| Width (minimum strip) | rotating calipers | O(h) |
| Minimum-area bounding box | calipers; a side lies on the hull | O(h) |
| Maximum-distance pair to a line | ternary search on the hull | O(log h) |
| Point in hull | binary search on the fan | O(log h) |
| Minkowski sum of convex sets | merge edge vectors by angle | O(h_1+h_2) |
| DP optimisation | convex hull trick (Chapter 64) | O(log n) per query |
Rotating calipers
Two parallel supporting lines rotated around the hull. As the angle sweeps through 180°, each contact point advances monotonically, so the total work is O(h) — an amortised two-pointer argument on a cyclic sequence.
The diameter is the maximum distance over antipodal pairs; the width is the minimum over (edge, farthest vertex) pairs; the minimum-area bounding box has a side flush with a hull edge (a theorem worth knowing, since it reduces an infinite search over angles to h candidates).
"Find the two farthest points" is O(n²) naively; the strong answer is hull plus rotating calipers at O(nlog n), with the observation that the farthest pair are both hull vertices — because moving a point toward the interior along any direction only shortens distances.
- Theory
Prove that the farthest pair of points are both vertices of the convex hull.
Solution
Suppose p is not a hull vertex, so p = ∑ λ_iv_i is a convex combination of hull vertices with ∑λ_i = 1, λ_i\ge0. For any point q,
|p - q| = |∑λ_i(v_i - q)| ≤ ∑λ_i|v_i-q| ≤ max_i|v_i-q|,
by the triangle inequality and then by the fact that a weighted average is at most the maximum. So replacing p by the maximising v_i does not decrease the distance. ∎
Applying this to both points of a farthest pair shows some farthest pair consists of hull vertices — and since the hull has h ≤ n vertices, this reduces the search space, though not by itself below O(h²). Rotating calipers gets it to O(h).
The same argument shows the diameter, the minimum enclosing circle's defining points, and the minimum bounding box all depend only on the hull.
- Theory
Prove that convex hull requires Ω(nlog n) comparisons.
Solution
Reduce sorting to convex hull. Given n reals x_1,…,x_n, map each to the point (x_i, x_i²) on the parabola y = x².
Since the parabola is strictly convex, every point is a hull vertex, and the hull's lower chain visits them in increasing order of x. So reading the hull off gives the sorted order in O(n).
A convex hull algorithm faster than O(nlog n) would therefore sort faster, contradicting the comparison lower bound of Chapter 13. ∎
Caveat: the bound is for the algebraic decision-tree model. Output-sensitive algorithms achieve O(nlog h), which is better when h is small and matches the bound when h = Θ(n) — the reduction produces exactly that case, so there is no contradiction.
- Theory
Explain rotating calipers and prove the O(h) bound.
Solution
Maintain two parallel supporting lines and rotate them through 180°. At each moment the contact points are an antipodal pair — a pair admitting parallel supporting lines.
As the angle increases, each contact point moves monotonically around the hull (it never goes backwards, because the supporting direction rotates monotonically). Over a full 180° sweep each pointer therefore traverses the hull at most once.
Total pointer movement: O(h). ∎ This is the two-pointer amortisation argument (Chapter 12) applied to a cyclic sequence.
The number of antipodal pairs is O(h), which is also the reason the diameter can be found in linear time after the hull: there are only linearly many candidates, not \binom h2.
Degeneracy warning: collinear hull points create ties in the advance condition and can cause the pointers to loop. Removing collinear points from the hull is the standard precaution.
- Theory
Show that the minimum-area bounding rectangle has a side collinear with a hull edge.
Solution
Consider the minimum-area enclosing rectangle at angle θ, and let A(θ) be its area. As θ varies, the supporting points change only at discrete angles, and between those angles the width and height are w(θ) = a\cosθ+b\sinθ style expressions — so A(θ) is a smooth function whose critical points can be analysed.
The standard argument shows A is minimised at a boundary of the smooth pieces, i.e. at an angle where a rectangle side contains a hull edge. Intuitively: if no side touches an edge, the rectangle can be rotated slightly to reduce the area, because the area is locally a strictly concave function of the angle. ∎ (Freeman and Shapira, 1975.)
Consequence: only h candidate angles need testing, one per hull edge, and rotating calipers evaluates all of them in O(h) total. Without this theorem the problem would be a continuous optimisation over angles.
The analogous statement in 3D is false for the minimum-volume box, which is why that problem is much harder (O(n³) by O'Rourke).
- Theory
Define the Minkowski sum of two convex polygons and give an O(n+m) algorithm.
Solution
A ⊕ B = {a+b : a∈ A, b∈ B}. For convex polygons the sum is convex, and its edge vectors are exactly the union of A's and B's edge vectors, sorted by angle.
Why: the supporting point of A⊕ B in direction d is (supporting point of A) + (supporting point of B), so as d rotates, the boundary traces A's edges and B's edges interleaved by angle.
def minkowski(A, B): A = reorder_from_bottom_left(A); B = reorder_from_bottom_left(B) res = []; i = j = 0 while i < len(A) or j < len(B): res.append(add(A[i % len(A)], B[j % len(B)])) c = cross(edge(A, i), edge(B, j)) if c >= 0: i += 1 if c <= 0: j += 1 return resO(n+m) — a merge by edge angle. ∎
Uses: collision detection (A and B intersect iff 0 ∈ A ⊕ (-B), the basis of the GJK algorithm), motion planning with a robot of non-zero size (grow the obstacles by the robot's shape), and the "minimum distance between two convex polygons" problem.
- Theory
Describe the dynamic convex hull problem and state the achievable bounds.
Solution
Maintain the hull of a point set under insertions and deletions.
- Insertions only: O(log n) amortised. Keep the upper and lower hulls in balanced BSTs keyed by x; inserting a point removes a contiguous run of now-interior vertices, and each vertex is removed at most once — an amortisation argument (Chapter 12).
- Insertions and deletions: O(log² n) amortised by Overmars–van Leeuwen's hull tree; improved to O(log n) amortised by Brodal–Jacob (2002), though that structure is complex and rarely implemented.
- Offline with deletions: use divide and conquer over time (each point is alive on an interval; process a segment tree over time), giving O(nlog² n) with a simple static hull as the primitive.
The offline technique is the practical answer in a contest: it converts "dynamic with deletions" into O(log n) static problems, and it applies far beyond hulls — to connectivity, to minimum spanning trees, and to any structure supporting insert-only updates.
- Theory
Explain Chan's algorithm and why it achieves O(nlog h).
Solution
Guess a value m for the hull size. Partition the points into ⌈ n/m⌉ groups of size m and compute each group's hull in O(mlog m) — total O(nlog m).
Then run gift wrapping on the groups: at each step, find the next hull vertex by binary-searching each group's hull for its tangent point, taking the best. Each step costs O(\frac nmlog m), and if the true hull has h ≤ m vertices, m steps suffice — total O(nlog m).
The guessing: try m = 2^{2^t} for t = 1,2,3,…, stopping when the wrapping completes. The costs form a geometric series dominated by the last term, and the last m is at most h², giving O(nlog h) overall. ∎
This is optimal: it matches the Ω(nlog h) output-sensitive lower bound. In practice monotone chain is faster for all realistic inputs because of its tiny constant — Chan's algorithm is a beautiful result that is essentially never the right implementation choice.
- Program
Implement Andrew's monotone chain with an option to keep or drop collinear points; test both.
Solution
def convex_hull(points, keep_collinear=False): pts = sorted(set(points)) if len(pts) <= 2: return pts thresh = -1 if keep_collinear else 0 # cross <= thresh ⟹ pop def build(seq): h = [] for p in seq: while len(h) >= 2 and cross(h[-2], h[-1], p) <= thresh: h.pop() h.append(p) return h lower = build(pts) upper = build(reversed(pts)) return lower[:-1] + upper[:-1] # counterclockwiseSetting
thresh = -1makes the pop conditioncross < 0, so collinear points (cross = 0) survive. That single parameter is the whole difference, and exposing it is better than silently choosing.Test both modes on: all points collinear (should return 2 points or all of them), duplicates, fewer than 3 points, and a square with points on its edges. Those four cases catch essentially every hull bug.
The
sorted(set(...))deduplication matters: repeated points make the cross product zero and can cause the stack to underflow in thekeep_collinearmode. - Program
Find the diameter of a point set using rotating calipers.
Solution
def diameter(points): h = convex_hull(points) # collinear points REMOVED n = len(h) if n == 1: return 0, (h[0], h[0]) if n == 2: return dist2(h[0], h[1]), (h[0], h[1]) best = 0; pair = None j = 1 for i in range(n): ni = (i+1) % n # advance j while the triangle area grows — j is antipodal to edge i while cross(h[i], h[ni], h[(j+1) % n]) > cross(h[i], h[ni], h[j]): j = (j+1) % n for cand in (j, (j+1) % n): for endpoint in (i, ni): d = dist2(h[endpoint], h[cand]) if d > best: best, pair = d, (h[endpoint], h[cand]) return best, pairO(h) after the hull, so O(nlog n) total. Using squared distances keeps integer input exact — take the square root only when reporting.
The
whileadvances j to the vertex farthest from edge i, which is the antipodal partner. Because j only moves forward across the whole loop, the total advancement is O(h) — the amortisation of problem 3.Checking both j and j+1 against both endpoints of the edge handles the ties that arise when two vertices are equally far — omitting that is the classic source of an off-by-one wrong answer on symmetric inputs.
- Program
Compute the minimum-area and minimum-perimeter enclosing rectangles.
Solution
import math def min_bounding_rect(points): h = convex_hull(points) n = len(h) if n < 3: return degenerate_case(h) best_area = float('inf'); best = None j = k = l = 1 for i in range(n): ni = (i+1) % n e = sub(h[ni], h[i]) # the edge direction # advance the three calipers: farthest, rightmost, leftmost while cross_val(e, sub(h[(j+1) % n], h[i])) > cross_val(e, sub(h[j], h[i])): j = (j+1) % n while dot_val(e, sub(h[(k+1) % n], h[i])) > dot_val(e, sub(h[k], h[i])): k = (k+1) % n if i == 0: l = k while dot_val(e, sub(h[(l+1) % n], h[i])) <= dot_val(e, sub(h[l], h[i])): l = (l+1) % n elen = math.hypot(*e) height = abs(cross_val(e, sub(h[j], h[i]))) / elen width = (dot_val(e, sub(h[k], h[i])) - dot_val(e, sub(h[l], h[i]))) / elen if width * height < best_area: best_area = width * height; best = (i, width, height) return best_area, bestThree calipers rotate together: one for the height (farthest by cross product) and two for the width (extremes by dot product). All advance monotonically, so the whole loop is O(h).
By problem 4 only the h edge-aligned angles need testing, which is what makes this a discrete search rather than a continuous optimisation.
Minimum perimeter uses the same loop with 2(w+h) as the objective — and the two minima are generally attained at different angles, which is a good sanity check that the code is genuinely evaluating both.
- Program
Implement the Minkowski sum of two convex polygons and use it for collision detection.
Solution
def minkowski_sum(A, B): A = rotate_to_lowest(A); B = rotate_to_lowest(B) # start at min (y,x) A = A + A[:2]; B = B + B[:2] # wrap for edge access res = []; i = j = 0 while i < len(A)-2 or j < len(B)-2: res.append(add(A[i], B[j])) c = cross(sub(A[i+1], A[i]), sub(B[j+1], B[j])) if c >= 0 and i < len(A)-2: i += 1 if c <= 0 and j < len(B)-2: j += 1 return res def convex_polygons_intersect(A, B): """A and B intersect iff the origin is inside A ⊕ (−B)""" negB = [(-x, -y) for x, y in B] return point_in_convex(minkowski_sum(A, negB), (0, 0))Starting both polygons at their lowest-then-leftmost vertex is essential — the merge assumes both edge sequences begin at the same angular position (pointing right, by convention).
The collision test is exact for integer coordinates and O(n+m). It is the mathematical core of the GJK algorithm used in every physics engine, which computes the distance between convex shapes by searching the Minkowski difference for its closest point to the origin — without ever constructing it.
Verify by testing pairs of polygons known to touch, overlap and be disjoint, and by checking that the sum's area equals A_1 + A_2 + mixed area (the Brunn–Minkowski relation).
- Program
Compute the convex layers (onion peeling) of a point set.
Solution
def convex_layers(points): remaining = list(points) layers = [] while remaining: h = convex_hull(remaining) if not h: break layers.append(h) hs = set(h) remaining = [p for p in remaining if p not in hs] return layersNaively O(n²log n) — each peel is a full hull computation. Chazelle's algorithm computes all layers in O(nlog n) total using a dynamic hull structure, but it is intricate.
The naive version is fine when the number of layers is small, which is the typical case: for n uniformly random points in a convex region the expected number of layers is Θ(n^{2/3}) and each hull has O(log n) or O(n^{1/3}) vertices depending on the region's shape.
Uses: convex layers give a robust notion of "depth" in a point cloud — the outermost layers are the outliers — and the layer number is a two-dimensional analogue of rank, used in robust statistics as the basis of the bagplot.
- Program
Solve: given n points, find the smallest circle enclosing all of them.
Solution
import random def min_enclosing_circle(points): pts = points[:]; random.shuffle(pts) c, r2 = pts[0], 0 for i, p in enumerate(pts): if dist2(c, p) <= r2: continue c, r2 = p, 0 # p is on the boundary for j in range(i): q = pts[j] if dist2(c, q) <= r2: continue c, r2 = midpoint(p, q), dist2(p, q) / 4 # p, q on the boundary for k in range(j): s = pts[k] if dist2(c, s) <= r2: continue c = circumcentre(p, q, s); r2 = dist2(c, p) return c, r2Welzl's algorithm, in its incremental form. The triple loop looks cubic but is O(n) expected: by backwards analysis, point i lies outside the circle of the first i-1 with probability ≤ 3/i (the circle is determined by at most 3 points), so the expected cost of the inner work is ∑_i \frac3i O(i) = O(n).
The shuffle is mandatory — with adversarial ordering the algorithm is Θ(n³). This is the same randomised-incremental structure as Seidel's LP (Chapter 55), and for the same reason: the solution is determined by a constant number of constraints.
Note it does not need the convex hull, though the answer is determined by hull points. Computing the hull first is a valid constant-factor optimisation.
- Program
Stress-test a convex hull implementation against a brute-force O(n³) definition.
Solution
import random, itertools def brute_hull_vertices(pts): """p is a hull VERTEX iff some line through p has all others strictly on one side — check all directed pairs""" out = set() for p in pts: for q in pts: if p == q: continue if all(cross(p, q, r) >= 0 for r in pts): out.add(p); out.add(q) return out for _ in range(20000): n = random.randint(1, 8) pts = [(random.randint(-4,4), random.randint(-4,4)) for _ in range(n)] h = convex_hull(pts, keep_collinear=True) assert set(h) <= set(pts) assert all(cross(h[i], h[(i+1)%len(h)], h[(i+2)%len(h)]) >= 0 for i in range(len(h))) or len(h) < 3 # convexity assert all(point_in_convex(h, p) for p in pts) # containmentThree properties characterise a correct hull, and asserting all three is stronger than comparing against any single reference implementation: the output is a subset of the input, it is convex, and it contains every input point.
The small coordinate range produces many collinear and duplicate points — precisely the degenerate cases where hull implementations fail, and which uniformly random real coordinates never generate.
Add the cases n=1, n=2, all points identical, and all points collinear as explicit tests: each takes a different code path and each has broken a published implementation at some point.
Sweep line: closest pair and segment intersection
Move an imaginary line across the plane and maintain only what it touches. A two-dimensional problem becomes a one-dimensional dynamic problem, and O(n^2) becomes O(n log n).
The sweep-line paradigm has three components: an event queue (sorted positions where the active set changes), a status structure (what the line currently intersects, in sorted order), and an invariant asserting that everything before the line has been correctly handled.
Closest pair
sort points by x
best = infinity
active = ordered set, keyed by y
left = 0
for each point p in x-order:
while p.x − points[left].x >= best: # too far left to matter
active.remove(points[left]); left += 1
for q in active with q.y in [p.y − best, p.y + best]:
best = min(best, dist(p, q))
active.insert(p)
Why it is O(n log n): the strip of width `best` and height `2·best`
can hold at most 6 points pairwise ≥ best apart, so the inner loop
examines O(1) candidates per point.The packing bound is what makes sweep-line geometry fast. In the closest-pair strip, points are pairwise at distance ≥ best, so only O(1) fit in the relevant window. Nearly every O(nlog n) geometric sweep has a corresponding packing or planarity argument bounding the work per event — find it, and the complexity follows.
Bentley–Ottmann: segment intersection
Report all k intersections among n segments in O((n+k)log n).
The status order changes only at intersections, which is why detecting them by watching adjacency is complete. The output-sensitive bound matters: with k = Θ(n²) intersections it degenerates to O(n²log n), worse than the trivial algorithm — so use it when intersections are expected to be sparse.
Other sweeps worth knowing
| Problem | Status structure | Cost |
|---|---|---|
| Rectangle union area | segment tree over y | O(nlog n) |
| Maximum overlap of intervals | a counter | O(nlog n) |
| Voronoi diagram (Fortune) | beach line (BST) | O(nlog n) |
| Delaunay triangulation | dual of Voronoi | O(nlog n) |
| Convex hull (monotone chain) | a stack | O(nlog n) |
| Visibility polygon | angular sweep, BST of edges | O(nlog n) |
| Skyline / building outline | multiset of heights | O(nlog n) |
"Find the closest pair of points" expects O(nlog n), and either divide and conquer or the sweep is fine. The point that impresses is the packing argument — at most six points fit in the strip because they are pairwise at least d apart — because that is the step that makes the algorithm's complexity work, and it is the part most people cannot reconstruct.
- Theory
Prove the packing bound in the closest-pair sweep: at most a constant number of candidates per point.
Solution
When processing p, all active points q satisfy p_x - q_x < d (where d is the current best) and we only consider |p_y - q_y| < d. So candidates lie in a d × 2d rectangle to the left of p.
All active points are pairwise at distance ≥ d (otherwise d would already be smaller). Partition the rectangle into six \frac d2 × (2d)/(3) cells; each cell has diameter √{(d²)/(4)+(4d²)/(9)} = d√{(25)/(36)} = (5d)/(6) < d, so each holds at most one point.
Hence at most 6 candidates. ∎
The constant is not important; what matters is that it is bounded independently of n, which makes the inner loop O(1) amortised and the whole sweep O(nlog n), dominated by the sort and the BST operations.
The divide-and-conquer version of closest pair (Chapter 10) uses the same argument with the constant 7, for the same reason.
- Theory
Prove the key lemma of Bentley–Ottmann: intersecting segments become adjacent before crossing.
Solution
Let segments s and t intersect at a point with x-coordinate x_0, and suppose neither endpoint lies at x_0. Just before x_0 they are both in the status structure, with (say) s above t.
Consider the moment they become adjacent. Immediately before x_0 their y-coordinates converge, so any segment between them must exit the status structure or cross one of them first — and each such event occurs at a smaller x. By induction on the number of segments between them, there is a moment before x_0 at which s and t are adjacent. ∎
Consequence: testing only newly-adjacent pairs finds every intersection. Each event creates O(1) new adjacencies (an insertion creates 2, a deletion 1, a swap 2), so the total number of tests is O(n+k).
The lemma fails for degenerate input — three segments through one point, or vertical segments — which is why robust implementations perturb symbolically or handle those cases explicitly.
- Theory
Analyse the complexity of Bentley–Ottmann and explain when it is worse than brute force.
Solution
Events: 2n endpoints plus k intersections, each processed in O(log n) (a BST operation plus O(1) intersection tests). Total O((n+k)log n).
Worse than brute force when k is large: with k = Θ(n²) the sweep is O(n²log n) while the trivial all-pairs test is O(n²). The crossover is around k ≈ n²/log n.
So the algorithm is output-sensitive, and its advantage depends on intersections being sparse — which they usually are in map data, VLSI layouts and mesh processing, the applications it was designed for.
The optimal algorithm is Chazelle–Edelsbrunner at O(nlog n + k), removing the log from the output term. It is considerably more complex and rarely implemented; Bentley–Ottmann remains the practical choice.
If you only need to know whether any intersection exists, the Shamos–Hoey variant stops at the first one and is O(nlog n) unconditionally.
- Theory
Describe Fortune's algorithm for Voronoi diagrams and the role of the beach line.
Solution
Sweep a horizontal line downward. The beach line is the boundary of the region already determined — a sequence of parabolic arcs, one per site above the sweep line, where each parabola is the locus of points equidistant from its site and the sweep line.
Two event types:
- Site event: the sweep reaches a new site, inserting a new arc into the beach line and splitting an existing one.
- Circle event: an arc shrinks to zero width, meaning three sites become co-circular — this creates a Voronoi vertex.
There are n site events and O(n) circle events (each creates a vertex, and a planar graph on n sites has O(n) vertices by Euler's formula), each costing O(log n). Total O(nlog n), which is optimal. ∎
The insight that makes it work is that the beach line is monotone in x, so it can live in a balanced BST — which is not obvious, since the Voronoi diagram itself has no such ordering.
In practice, computing the Delaunay triangulation (the dual) by randomised incremental insertion is simpler to implement and equally fast, and the Voronoi diagram is read off from it.
- Theory
Design a sweep for the skyline problem: given n buildings as (l, r, h), output the outline.
Solution
Events: (l, +h) and (r, -h) for each building. Sweep left to right maintaining a multiset of active heights.
At each event x, update the multiset, then look at its maximum. If the maximum changed, emit the key point (x, new max).
events = sorted([(l, -h) for l,r,h in bs] + [(r, h) for l,r,h in bs]) # note: -h for starts so that at the same x, taller starts come FIRST # and ends come after starts, which avoids spurious key pointsO(nlog n). The event ordering is the whole difficulty: at a shared x-coordinate, starts must precede ends (so a building starting where another ends does not create a dip), and among starts the taller must come first (so no spurious key point is emitted). Encoding start heights as negative achieves both with a plain sort.
The multiset needs
maxanderase one occurrence— astd::multiset, a heap with lazy deletion, or a counting map. Lazy deletion is simplest: push to a max-heap and pop stale entries when they surface. - Theory
Explain how to compute the area of the union of n rectangles and generalise to "covered at least k times".
Solution
Sweep over x. The status is a segment tree over the compressed y-coordinates, storing per node a coverage count and the length covered at least once. Adding a rectangle's y-range increments the count on O(log n) nodes; the covered length is recomputed bottom-up.
Between consecutive events, the covered length times the x-gap contributes to the area. O(nlog n).
No lazy propagation is needed because every range increment is eventually matched by a decrement, so a node's count is always accurate for its own interval — a specialisation that makes the code much shorter than a general segment tree.
For "at least k times": store per node an array of lengths covered exactly 0,1,…,k-1 times plus "k or more". The merge is a shift by the node's own count. Cost becomes O(nklog n) — fine for small k, and this is the standard solution to "area covered by at least k rectangles".
- Theory
Describe an angular sweep for computing the visibility polygon from a point.
Solution
Sort all segment endpoints by angle around the viewpoint. Sweep a ray through 360°, maintaining a BST of the segments the ray currently crosses, ordered by distance from the viewpoint.
At each event (a segment starting or ending relative to the ray), insert or remove it; the visible point in the current direction is on the nearest segment, which is the BST's minimum.
Emit a vertex whenever the nearest segment changes. O(nlog n). ∎
The subtlety is the BST ordering: two segments' distance order can only change where they cross, and if no segments cross (the usual case for a simple polygon's edges) the order is stable within the sweep, so a plain comparator works. With crossing segments the comparator is not a valid ordering and the structure must be rebuilt or the segments split at their intersections first.
Applications: field-of-view in games, radio coverage, the art-gallery problem, and shadow computation. The same sweep with "farthest" instead of "nearest" gives the region visible from outside.
- Program
Implement closest pair by sweep line and verify against brute force.
Solution
from sortedcontainers import SortedList import math def closest_pair(points): pts = sorted(points) active = SortedList(key=lambda p: (p[1], p[0])) best = float('inf'); pair = None left = 0 for p in pts: d = math.isqrt(int(best)) + 1 if best < float('inf') else 0 while best < float('inf') and p[0] - pts[left][0] >= math.sqrt(best): active.remove(pts[left]); left += 1 lo = active.bisect_left((p[0], p[1] - math.sqrt(best) if best < float('inf') else -10**18)) for q in active[lo:]: if best < float('inf') and q[1] - p[1] > math.sqrt(best): break d2 = (p[0]-q[0])**2 + (p[1]-q[1])**2 if d2 < best: best, pair = d2, (p, q) active.add(p) return best, pairWorking with squared distances keeps integer input exact, though the strip bounds then need a square root — an alternative is to keep both
best2and its integer square root, updating the latter only whenbest2changes.Verify against the O(n²) brute force on 20,000 random small instances, including duplicate points (distance 0) and all-collinear points. The duplicate case is worth testing explicitly:
bestbecomes 0 and the strip has zero width, which must not cause an infinite loop. - Program
Implement Bentley–Ottmann to report all segment intersections.
Solution
import heapq def bentley_ottmann(segments): events = [] for i, (a, b) in enumerate(segments): if a > b: a, b = b, a heapq.heappush(events, (a, 0, i)) # 0 = left endpoint heapq.heappush(events, (b, 2, i)) # 2 = right endpoint status = OrderedSegments() # BST keyed by y at sweep x found = set() while events: pt, kind, data = heapq.heappop(events) status.set_x(pt[0]) if kind == 0: i = data; status.insert(i) for j in (status.above(i), status.below(i)): if j is not None: try_intersect(i, j, pt, events, found) elif kind == 2: i = data a, b = status.above(i), status.below(i) status.remove(i) if a is not None and b is not None: try_intersect(a, b, pt, events, found) else: # kind == 1: intersection i, j = data status.swap(i, j) # test the two NEW adjacencies created by the swap ... return foundThe implementation difficulty is entirely in
OrderedSegments: a BST whose comparator depends on the current sweep x. Every insertion and search must use a consistent x, and the swap at an intersection must be an exact reordering rather than a re-sort — otherwise the structure's invariant breaks.Degenerate cases that break naive implementations: vertical segments (undefined slope), three or more segments through one point (multiple simultaneous swaps), overlapping collinear segments (no well-defined order), and shared endpoints. Handling all four correctly is what makes production implementations long.
For a contest, the O(n²) all-pairs test is usually the right answer unless n is large and k is known to be small.
- Program
Solve the skyline problem with a sweep and a max-heap with lazy deletion.
Solution
import heapq def skyline(buildings): events = [] for l, r, h in buildings: events.append((l, -h, r)) # start: negative h sorts taller first events.append((r, 0, 0)) # end marker events.sort() result = [] heap = [(0, float('inf'))] # (−height, end); ground at 0 for x, negh, r in events: while heap[0][1] <= x: heapq.heappop(heap) # lazy deletion if negh: heapq.heappush(heap, (negh, r)) cur = -heap[0][0] if not result or result[-1][1] != cur: result.append((x, cur)) return resultLazy deletion avoids needing a heap with arbitrary removal: entries carry their end coordinate, and stale ones are popped when they reach the top and their building has ended. Each entry is pushed and popped once, so the total is O(nlog n).
The sentinel
(0, inf)represents ground level and guarantees the heap is never empty — removing it introduces an empty-heap check in three places.The event encoding (
-hfor starts,0for ends) makes a plain sort produce the correct order: at equal x, starts come before ends, and among starts the taller comes first. Getting that ordering right is the whole problem; the rest is bookkeeping. - Program
Compute the area of the union of n rectangles using a coverage segment tree.
Solution
class CoverTree: def __init__(self, ys): self.ys = ys; n = len(ys) - 1 self.cnt = [0] * (4*n); self.cov = [0] * (4*n); self.n = n def _update(self, node, lo, hi, l, r, delta): if r <= lo or hi <= l: return if l <= lo and hi <= r: self.cnt[node] += delta else: mid = (lo+hi)//2 self._update(2*node+1, lo, mid, l, r, delta) self._update(2*node+2, mid, hi, l, r, delta) if self.cnt[node] > 0: self.cov[node] = self.ys[hi] - self.ys[lo] # fully covered elif hi - lo == 1: self.cov[node] = 0 else: self.cov[node] = self.cov[2*node+1] + self.cov[2*node+2]The recomputation of
covafter the recursion is the crux: a node with positive count reports its whole interval regardless of its children, and only when the count drops to zero do the children's contributions matter. Because increments and decrements are matched, the count is never negative and no lazy propagation is required.O(nlog n) total. Coordinate-compressing the y-values first is essential — the tree is over the 2n distinct y-coordinates, not over the coordinate range.
Verify against a brute-force grid computation on small integer rectangles, including nested, identical and disjoint ones.
- Program
Implement a sweep to find the maximum number of intervals overlapping at any point, and the k intervals achieving it.
Solution
def max_overlap(intervals): events = [] for i, (l, r) in enumerate(intervals): events.append((l, 1, i)) events.append((r, -1, i)) # half-open [l, r): ends sort first events.sort(key=lambda e: (e[0], e[1])) # −1 before +1 at equal x cur = 0; best = 0; active = set(); best_set = set() for x, delta, i in events: if delta == 1: active.add(i); cur += 1 else: active.discard(i); cur -= 1 if cur > best: best, best_set = cur, set(active) return best, best_setThe tie-break
(x, delta)with -1 < +1 implements half-open intervals: an interval ending exactly where another begins does not count as overlapping. Reversing it gives closed intervals. Which one is wanted is a specification question and is the most common source of off-by-one here.O(nlog n). Maintaining the
activeset makes it O(nlog n + output); if only the count is needed, drop the set and it becomes a two-line sweep.This is the minimum-number-of-rooms scheduling problem (Chapter 5), and by Dilworth's theorem the answer is also the size of the largest antichain in the interval order — three formulations of one sweep.
- Program
Detect whether any two of n segments intersect, in O(nlog n) (Shamos–Hoey).
Solution
def any_intersection(segments): events = [] for i, (a, b) in enumerate(segments): if a > b: a, b = b, a events.append((a, 0, i)); events.append((b, 1, i)) events.sort() status = SortedList() # ordered by y at the sweep x for pt, kind, i in events: if kind == 0: pos = status.insert_and_locate(i, pt[0]) for j in status.neighbours(pos): if segments_intersect(*segments[i], *segments[j]): return True else: a, b = status.neighbours_of(i) status.remove(i) if a is not None and b is not None: if segments_intersect(*segments[a], *segments[b]): return True return FalseBecause it stops at the first intersection, no intersection events ever enter the queue — the sweep is O(nlog n) unconditionally, with none of Bentley–Ottmann's output sensitivity.
This is the practical tool for testing whether a polygon is simple (Chapter 54 problem 11): run it on the polygon's edges, excluding adjacent-edge pairs which legitimately share an endpoint.
The status comparator still depends on the sweep x, but since no swaps occur before the first intersection is found, the ordering remains valid throughout — which is precisely why this variant avoids the implementation difficulties of the full algorithm.
- Program
Build a generic sweep-line framework and instantiate it for three different problems.
Solution
import heapq class Sweep: def __init__(self, events, status, handle): self.q = list(events); heapq.heapify(self.q) self.status = status self.handle = handle self.output = [] def run(self): while self.q: e = heapq.heappop(self.q) new_events = self.handle(e, self.status, self.output) for ne in (new_events or ()): heapq.heappush(self.q, ne) return self.output # instantiate: interval overlap (status = counter), # rectangle union (status = CoverTree), # closest pair (status = SortedList by y)Factoring out the loop makes the three components explicit — event queue, status, handler — and forces you to name them, which is the design discipline the paradigm rewards. The handler returning new events is what supports Bentley–Ottmann's dynamically-discovered intersection events.
The framework is thin by design. Sweep-line problems differ almost entirely in their status structure and in the event ordering at ties, and both of those resist abstraction — which is why in practice most people write the loop inline. The value of writing it once is that it makes the tie-breaking rule a visible, testable parameter rather than an accident of a sort comparator.
Circles, tangents and 3D geometry
Circles bring square roots and therefore genuine numerical care, but most predicates still reduce to comparing squared quantities exactly. The 3D extensions - planes, spheres, the triple product - follow the same pattern one dimension up.
Circle intersections
// Circles (c1, r1), (c2, r2). Let d = |c2 − c1|.
// The intersection points lie on the RADICAL LINE, perpendicular to c1c2 at
// a = (d² + r1² − r2²) / (2d) distance from c1 along the axis
// h = √(r1² − a²) half the chord length
// Then: base = c1 + a·(c2−c1)/d, points = base ± h·perp((c2−c1)/d)
pair<P,P> circle_intersect(P c1, double r1, P c2, double r2){
P dv = c2 - c1;
double d2 = norm2(dv), d = sqrt(d2);
double a = (d2 + r1*r1 - r2*r2) / (2*d);
double h2 = r1*r1 - a*a;
if (h2 < 0) return {}; // no intersection
P base = c1 + dv * (a/d);
P off = perp(dv) * (sqrt(h2)/d);
return {base + off, base - off};
}The radical axis of two circles is the locus of points with equal power |P-C|²-r². Subtracting the two circle equations eliminates the quadratic terms, leaving a line — which is why circle intersection reduces to line–circle intersection, and why three circles' radical axes are concurrent (the radical centre).
Keep everything squared for as long as possible. "Is P inside the circle", "which of two points is closer", and "do these circles intersect" are all comparisons of squared quantities and stay exact in integers. Take a square root only when a length must be output — never to make a decision.
Tangents
Three dimensions
"Do two circles intersect?" should never involve a square root: compare d² against (r_1+r_2)² and (r_1-r_2)². Volunteering that — and noting it keeps integer input exact — turns a routine question into a demonstration of numerical judgement.
- Theory
Derive the circle–circle intersection formula and enumerate the degenerate cases.
Solution
Place c_1 at the origin and c_2 at (d,0). The two equations are x²+y² = r_1² and (x-d)²+y² = r_2². Subtracting eliminates y²:
2dx - d² = r_1²-r_2², so x = (d²+r_1²-r_2²)/(2d),
and y = ±√{r_1²-x²}. Rotating back gives the general formula. ∎
Degenerate cases, all detectable by comparing d² exactly:
- d = 0 and r_1 = r_2: identical circles, infinitely many points.
- d = 0, r_1 ≠ r_2: concentric, none. (The formula divides by d — guard first.)
- d² = (r_1+r_2)² or d² = (r_1-r_2)²: tangent, one point.
- r_1² < x²: no real intersection.
All the comparisons are on squared quantities, so with integer centres and radii the classification is exact even though the points are irrational.
- Theory
Define the power of a point and prove that the radical axis of two circles is a line.
Solution
The power of P with respect to circle (C,r) is pow(P) = |P-C|² - r² — positive outside, zero on the circle, negative inside. It equals the squared tangent length from P.
The radical axis is {P : pow_1(P) = pow_2(P)}:
|P-C_1|²-r_1² = |P-C_2|²-r_2².
Expanding, the |P|² terms cancel and what remains is linear in P — a line, perpendicular to C_1C_2. ∎
Radical centre: three circles' three pairwise radical axes are concurrent, because a point on two of them has equal power to all three. This gives an elegant construction and is the basis of the "power diagram" (a weighted Voronoi diagram) used in mesh generation.
When the circles intersect, the radical axis is the line through their two intersection points — which is why the intersection derivation of problem 1 works.
- Theory
Derive the circumcircle of a triangle and state when it fails to exist.
Solution
The circumcentre is equidistant from all three vertices, so it lies on the perpendicular bisector of each side. Solving two of them:
With a at the origin, b and c the other vertices, and D = 2(b_xc_y - b_yc_x):
O_x = (c_y|b|² - b_y|c|²)/(D), O_y = (b_x|c|² - c_x|b|²)/(D).
D = 2cross(b,c) is twice the orientation determinant, so it vanishes exactly when the three points are collinear — and then there is no circumcircle (or, projectively, a line). ∎
The radius is R = (abc)/(4K) where a, b, c are the side lengths and K the area — a formula worth knowing because it computes R without finding the centre.
The circumcentre is rational for integer input, so it can be kept exact as a fraction. The radius generally is not, which is why the in-circle predicate (Chapter 45) is formulated as a determinant rather than as a distance comparison.
- Theory
Find the tangent lines from an external point to a circle.
Solution
Let d = |P-C| with d > r. The tangent length is L = √{d²-r²} (Pythagoras, since the radius is perpendicular to the tangent at the touch point).
The tangent points are at distance r from C and at distance L from P — so they are the intersections of circle (C,r) with circle (P,L), solvable by problem 1.
Equivalently (Thales): the tangent points lie on the circle with diameter PC, since the angle \angle CTP is a right angle. So intersect (C,r) with the circle centred at the midpoint of PC with radius d/2. ∎
The angle between PC and each tangent is \arccos(r/d), giving a direct rotation-based construction that avoids a second circle intersection.
Degenerate cases: d = r gives one tangent (P is on the circle), d < r gives none. Both are detected by comparing d² with r² exactly.
- Theory
Count and construct the common tangents of two circles in every configuration.
Solution
separate (d > r₁+r₂) 4 tangents (2 external, 2 internal) externally tangent (d = r₁+r₂) 3 (the internal pair merges) intersecting 2 (external only) internally tangent 1 one inside the other 0 identical infinitely manyExternal tangents: shrink both circles by min(r_1,r_2); one becomes a point, and the problem reduces to tangents from a point to a circle of radius |r_1-r_2|. Then translate the resulting lines back outward.
Internal tangents: the same construction with radius r_1+r_2, which corresponds to reflecting one circle's radius.
Both reductions turn a two-circle problem into problem 4, which is why that one is the primitive worth implementing carefully.
All six cases are distinguished by comparing d² against (r_1± r_2)² — exactly, with no square roots.
- Theory
Derive the 3D orientation predicate and explain its geometric meaning.
Solution
For four points a, b, c, d:
orient3d(a,b,c,d) = (b-a)·\big((c-a)×(d-a)\big) = \det\begin{smallmatrix}b_x-a_x & c_x-a_x & d_x-a_x\\ b_y-a_y & c_y-a_y & d_y-a_y\\ b_z-a_z & c_z-a_z & d_z-a_z\end{smallmatrix}.
This is six times the signed volume of the tetrahedron abcd. Its sign says whether d is above or below the plane through a, b, c (with "above" meaning the side the normal (b-a)×(c-a) points to), and zero means the four points are coplanar. ∎
It is the direct analogue of the 2D orientation test, and it plays the same role: 3D convex hull, Delaunay tetrahedralisation and point location all reduce to its sign.
Magnitude bound: for coordinates in [-C,C] the determinant is at most 48C³, so 64-bit integers are exact only to C ≈ 6×10⁵ — considerably tighter than the 2D case, and a common source of overflow in 3D code.
- Theory
Explain the "lifting to a paraboloid" trick that turns Delaunay triangulation into a 3D convex hull.
Solution
Map each planar point (x,y) to (x,y,x²+y²) on the paraboloid z = x²+y².
Key fact: a circle in the plane lifts to a plane in 3D, and "inside the circle" becomes "below the plane". So the in-circle predicate becomes a 3D orientation test:
incircle(a,b,c,d) = orient3d(\hat a,\hat b,\hat c,\hat d) where \hat{} denotes the lift.
Consequence: the lower convex hull of the lifted points projects exactly to the Delaunay triangulation, because a triangle is Delaunay iff its circumcircle is empty iff the corresponding lifted plane has all other lifted points above it. ∎
This reduces a 2D problem to a 3D hull, unifies the algorithms, and explains why the in-circle determinant has the form it does (Chapter 45 problem 14) — the x²+y² column is the lift.
The same trick in one dimension higher relates 3D Delaunay to 4D hulls, and the general statement is that Delaunay in ℝ^d is a hull in ℝ^{d+1}.
- Program
Implement exact circle predicates: point in circle, circle intersection classification, using only integer arithmetic.
Solution
def point_vs_circle(p, c, r): """−1 inside, 0 on, +1 outside — exact for integer input""" d2 = (p[0]-c[0])**2 + (p[1]-c[1])**2 return (d2 > r*r) - (d2 < r*r) def circle_relation(c1, r1, c2, r2): d2 = (c1[0]-c2[0])**2 + (c1[1]-c2[1])**2 s = (r1 + r2)**2 t = (r1 - r2)**2 if d2 > s: return 'separate' if d2 == s: return 'externally tangent' if d2 > t: return 'intersecting' if d2 == t: return 'identical' if d2 == 0 and r1 == r2 else 'internally tangent' return 'contained'Every comparison is between integers, so the classification is exact — no epsilon, no ambiguity at tangency. That matters because tangency is precisely where a floating-point version gives the wrong answer, and tangency is precisely what test-setters include.
The
d2 == 0 and r1 == r2check for identical circles must come before the internal-tangent case, since concentric equal circles satisfy d² = (r_1-r_2)² = 0.The intersection points are irrational and require floating point or a symbolic representation — but no decision in this function does.
- Program
Compute the intersection points of two circles and of a line and a circle, with numerical care.
Solution
import math def line_circle(a, b, c, r): """intersections of segment/line ab with circle (c, r)""" d = (b[0]-a[0], b[1]-a[1]) f = (a[0]-c[0], a[1]-c[1]) A = d[0]*d[0] + d[1]*d[1] B = 2*(f[0]*d[0] + f[1]*d[1]) C = f[0]*f[0] + f[1]*f[1] - r*r disc = B*B - 4*A*C if disc < 0: return [] sq = math.sqrt(disc) # numerically stable quadratic: avoid subtracting nearly-equal numbers q = -0.5 * (B + math.copysign(sq, B)) t1, t2 = q/A, C/q if q != 0 else 0.0 return sorted({t1, t2})The stable quadratic formula is the point. The naive \frac{-B±√{B²-4AC}}{2A} suffers catastrophic cancellation for one root when B² \gg 4AC — exactly the near-tangent case. Computing the larger root first and obtaining the other as C/(Aq) (from t_1t_2 = C/A) keeps both accurate.
The discriminant is exact for integer input, so the existence test is reliable; only the point coordinates are approximate.
For a segment rather than a line, clamp t to [0,1] and discard roots outside — and note that a tangent gives a double root, which the
setcollapses to one. - Program
Find all common tangent lines of two circles.
Solution
import math def common_tangents(c1, r1, c2, r2): out = [] d2 = (c2[0]-c1[0])**2 + (c2[1]-c1[1])**2 for sign1 in (1, -1): # external then internal r = r2*sign1 - r1 if d2 < r*r - 1e-12: continue h = math.sqrt(max(0.0, d2 - r*r)) for sign2 in (1, -1): # the tangent line is nx·x + ny·y = c nx = ((c2[0]-c1[0])*r + (c2[1]-c1[1])*h*sign2) / d2 ny = ((c2[1]-c1[1])*r - (c2[0]-c1[0])*h*sign2) / d2 out.append((nx, ny, nx*c1[0] + ny*c1[1] + r1)) if h == 0: break # tangent circles: one line return outThe unified formulation handles all four tangents with two sign parameters:
sign1selects external versus internal,sign2selects which of the pair. Shrinking by r_1 reduces both cases to tangents from a point, as in problem 5.The
h == 0break prevents emitting a duplicate line at tangency — which is how the counts 3 and 1 in the classification table arise.Verify by checking that each returned line is at distance exactly r_1 from c_1 and r_2 from c_2 (up to tolerance), and that the number returned matches the exact classification from problem 8.
- Program
Compute the area of intersection of two circles, and of a circle with a polygon.
Solution
import math def circle_circle_area(c1, r1, c2, r2): d = math.dist(c1, c2) if d >= r1 + r2: return 0.0 if d <= abs(r1 - r2): return math.pi * min(r1, r2)**2 a1 = math.acos((d*d + r1*r1 - r2*r2) / (2*d*r1)) a2 = math.acos((d*d + r2*r2 - r1*r1) / (2*d*r2)) return r1*r1*(a1 - math.sin(2*a1)/2) + r2*r2*(a2 - math.sin(2*a2)/2) def circle_polygon_area(c, r, poly): """sum of signed circular-triangle areas, one per polygon edge""" total = 0.0 n = len(poly) for i in range(n): total += circle_triangle_area(c, r, poly[i], poly[(i+1) % n]) return abs(total)The two-circle formula sums two circular segments, each being a sector minus a triangle. The
acosarguments must be clamped to [-1,1] — rounding can push them microscopically outside and produce a NaN that silently poisons the result.The circle–polygon area uses the same shoelace decomposition as Chapter 56: sum the signed area of the region between the circle and each edge's triangle from the centre. Each term is either a triangle, a sector, or a mixture depending on how the edge cuts the circle — a case analysis of about 20 lines that is worth writing once and keeping.
Verify against Monte Carlo integration: sample points in a bounding box and count how many fall in both shapes. Agreement to three digits with 10⁶ samples confirms the case analysis.
- Program
Implement 3D vector operations and the orientation predicate, with an overflow analysis.
Solution
struct P3 { long long x, y, z; }; P3 sub(P3 a, P3 b){ return {a.x-b.x, a.y-b.y, a.z-b.z}; } long long dot(P3 a, P3 b){ return a.x*b.x + a.y*b.y + a.z*b.z; } P3 cross(P3 a, P3 b){ return {a.y*b.z - a.z*b.y, a.z*b.x - a.x*b.z, a.x*b.y - a.y*b.x}; } __int128 orient3d(P3 a, P3 b, P3 c, P3 d){ P3 u = sub(b,a), v = sub(c,a), w = sub(d,a); return (__int128)u.x*(v.y*w.z - v.z*w.y) - (__int128)u.y*(v.x*w.z - v.z*w.x) + (__int128)u.z*(v.x*w.y - v.y*w.x); }Overflow analysis: with coordinates in [-C,C], the differences are ≤ 2C, the 2×2 minors are ≤ 8C², and the full determinant is ≤ 48C³. For
int64that requires C \lesssim 6×10⁵; the__int128accumulator in the outer products raises it to C \lesssim 4×10^{11}.Note the inner minors are still computed in
long longand must not overflow on their own — at C = 10⁶ they reach 8×10^{12}, safe; at C = 10⁹ they reach 8×10^{18}, marginal. Casting them too is the safe choice.This kind of bound should be computed and written in a comment above every geometric predicate — it is the difference between code that is exact and code that is exact on the test data.
- Program
Compute the convex hull of points in 3D and verify Euler's formula.
Solution
# Incremental 3D hull: maintain a set of faces; for each new point, # delete the faces it can "see" and cone the resulting horizon to it. def hull3d(points): faces = initial_tetrahedron(points) for p in points[4:]: visible = [f for f in faces if orient3d(*f, p) > 0] if not visible: continue # inside the hull horizon = boundary_edges(visible) faces = [f for f in faces if f not in visible] faces += [(e[0], e[1], p) for e in horizon] return faces # verification V = len({v for f in faces for v in f}) E = len({tuple(sorted(e)) for f in faces for e in edges_of(f)}) F = len(faces) assert V - E + F == 2 # Euler's formula assert E == 3*F // 2 # every face is a triangle assert F <= 2*V - 4 # the triangulated boundRandomised incremental insertion gives O(nlog n) expected. The three assertions are the right correctness checks: Euler's formula must hold for any convex polyhedron, and with all faces triangular 2E = 3F, which combine to F ≤ 2V-4 and E ≤ 3V-6.
Those bounds also answer the complexity question: a 3D hull has O(n) faces, so the output size is linear — unlike 4D and above, where the hull can have Θ(n^{⌊ d/2⌋}) faces and output-sensitivity becomes essential.
Degeneracies (four coplanar points, all points coplanar) need explicit handling; the initial tetrahedron construction must search for four non-coplanar points and report failure if none exist.
- Program
Solve: given n circles, compute the area of their union.
Solution
import math def circles_union_area(circles): """Green's theorem: integrate around the boundary of the union.""" total = 0.0 for i, (c, r) in enumerate(circles): # find the arcs of circle i NOT covered by any other circle events = [] covered_fully = False for j, (c2, r2) in enumerate(circles): if i == j: continue d = math.dist(c, c2) if d + r <= r2: covered_fully = True; break # i is inside j if d >= r + r2 or d + r2 <= r: continue # no overlap ang = math.atan2(c2[1]-c[1], c2[0]-c[0]) half = math.acos(max(-1.0, min(1.0, (d*d + r*r - r2*r2)/(2*d*r)))) events.append((ang - half, ang + half)) # the covered arc if covered_fully: continue for a, b in uncovered_arcs(events): total += arc_contribution(c, r, a, b) # Green's term return totalGreen's theorem turns the area into a boundary integral: A = \frac12\oint(xdy - ydx). Each uncovered arc contributes a closed-form term (a sector plus a triangle), so the whole computation is O(n²log n) — for each circle, sort and merge the covered arc intervals.
The two containment checks (
d + r <= r2andd + r2 <= r) must come before the arc computation, sinceacos's argument is out of range in those cases — and clamping it, as the code does, silently produces a wrong arc rather than an error.Verify with Monte Carlo, and with the two-circle formula of problem 11 as an exact special case. Circle-union area is a problem where a subtly wrong case analysis produces answers that look plausible, so independent verification is not optional.
Counting trees: Cayley, Prufer and the matrix-tree theorem
There are n^(n-2) labelled trees on n vertices, and the bijective proof - the Prufer sequence - is also an encoding, a random-tree generator, and a way to count trees with prescribed degrees.
The Prüfer sequence
A bijection between labelled trees on {1,…,n} and sequences in {1,…,n}^{n-2}. Since there are n^{n-2} such sequences, Cayley's formula follows immediately.
ENCODE: repeat n−2 times:
take the LEAF with the smallest label
append its neighbour to the sequence
delete the leaf
DECODE: degree[v] = 1 + (number of times v appears in the sequence)
repeat for each element a of the sequence:
take the smallest leaf v (degree 1, not yet used)
add edge (v, a); degree[v]−−; degree[a]−−
join the last two remaining vertices
Both directions run in O(n log n) with a heap, or O(n) with a pointer trick.A vertex of degree d appears exactly d-1 times in the Prüfer sequence. That single fact turns every "count trees with degree constraints" question into a multiset-permutation count — and it is why leaves (degree 1) never appear at all.
The matrix-tree theorem
For a graph with Laplacian L = D - A, every cofactor of L counts the spanning trees (Chapter 45). For weighted graphs the same determinant computes ∑_T∏_{e∈ T}w_e.
Counting other tree-like objects
| Object | Count |
|---|---|
| Labelled trees on n vertices | n^{n-2} |
| Labelled rooted trees | n^{n-1} |
| Labelled forests, k trees, given roots | kn^{n-k-1} |
| Binary trees with n nodes (unlabelled) | C_n (Catalan, Chapter 32) |
| Unlabelled rooted trees | no closed form; ~ 0.44·2.956^n n^{-3/2} |
| Functional graphs on n nodes | n^n |
| Spanning trees of a graph | matrix-tree determinant |
"How many trees on n labelled vertices?" — n^{n-2}, and the proof worth naming is the Prüfer bijection rather than the determinant, because the bijection is constructive: it also gives you an O(n) encoding, a uniform random-tree generator, and the degree-constrained count, all from the same map.
- Theory
Prove that the Prüfer encoding is a bijection.
Solution
Well-defined: a tree with n\ge2 vertices has at least two leaves, so the encoding never gets stuck, and it produces exactly n-2 symbols.
Injective: the decoding is deterministic. From the sequence, \deg(v) = 1 + count(v), so the leaves are exactly the vertices absent from the sequence, and the smallest is uniquely determined at each step. Following the decoding reconstructs the tree uniquely.
Surjective: given any sequence, the decoding produces a graph with n-1 edges. It is acyclic because each step attaches a vertex that is then removed and never reappears, so no cycle can close. An acyclic graph with n-1 edges on n vertices is a tree. ∎
Hence trees correspond bijectively to n^{n-2} sequences — Cayley's formula, proved constructively.
- Theory
Prove that a vertex of degree d appears exactly d-1 times in its Prüfer sequence.
Solution
Induct on n. Let v be the smallest leaf removed first, with neighbour u.
- v has degree 1 and, being removed immediately, is never appended — it appears 0 times ✓.
- u is appended once, and in the remaining tree its degree is \deg(u)-1, so by induction it appears \deg(u)-2 more times, totalling \deg(u)-1 ✓.
- Any other vertex w has the same degree in the smaller tree and appears \deg(w)-1 times by induction ✓.
The base case n=2 has an empty sequence and both degrees 1. ∎
Corollary: the number of labelled trees with prescribed degrees d_1,…,d_n (summing to 2n-2) is the number of sequences with vertex i appearing d_i-1 times — the multinomial ((n-2)!)/(∏(d_i-1)!).
- Theory
Derive the number of spanning trees of K_{m,n}.
Solution
By the matrix-tree theorem, compute a cofactor of the Laplacian. For K_{m,n} the Laplacian is
L = \begin{smallmatrix}nI_m & -J_{m× n}\\ -J_{n× m} & mI_n\end{smallmatrix}.
Its eigenvalues are 0 (once), m+n (once), n with multiplicity m-1, and m with multiplicity n-1. By the eigenvalue form of the matrix-tree theorem (the number of spanning trees is \frac1{m+n}∏_{nonzero}λ_i):
\tau(K_{m,n}) = \frac{(m+n)· n^{m-1}· m^{n-1}}{m+n} = m^{n-1}n^{m-1}. ∎
Check K_{1,n} (a star): 1^{n-1}n^0 = 1 ✓. Check K_{2,2} (a 4-cycle): 2^1·2^1 = 4 ✓, matching \tau(C_4) = 4.
The eigenvalue formula \tau = \frac1n∏_{i\ge2}λ_i is often easier than a cofactor when the graph is symmetric enough for the spectrum to be known.
- Theory
Count the labelled forests on n vertices consisting of k trees with specified roots.
Solution
The answer is kn^{n-k-1}.
Proof by a generalised Prüfer argument: add a new vertex n+1 joined to all k roots. This gives a tree on n+1 vertices in which n+1 has degree exactly k. By problem 2, such trees correspond to Prüfer sequences of length n-1 in which the symbol n+1 appears exactly k-1 times.
Choosing the positions and the other symbols: C(n-1,k-1)n^{n-k}. Dividing by the C(n,k) ways the roots could have been chosen and re-normalising yields kn^{n-k-1} for specified roots. ∎
Check k=1: 1· n^{n-2} ✓ (Cayley, with a distinguished root that does not matter since the tree is unrooted). Check k=n: n· n^{-1} = 1 ✓ (n isolated vertices, one forest).
This is the standard tool for "count the ways to connect components" problems.
- Theory
Use the generalised Cayley formula to count the ways to add edges connecting given components into a tree.
Solution
Given k components with sizes a_1,…,a_k summing to n, the number of ways to add k-1 edges making the whole thing a tree is
n^{k-2}∏_{i=1}^{k}a_i.
Why: contract each component to a single vertex. The resulting structure is a labelled tree on k vertices, of which there are k^{k-2} — but each edge of that tree can attach to any vertex within its component, contributing a factor. Careful counting via the degree-constrained Prüfer formula gives the stated product. ∎
Check k=2: n^0a_1a_2 = a_1a_2 ✓ — any pair of endpoints, one from each component.
Check all a_i = 1 (so k=n): n^{n-2}·1 = n^{n-2} ✓, recovering Cayley.
This formula answers "how many ways to make the graph connected with the minimum number of edges", which is a common contest question.
- Theory
Explain the matrix-tree theorem for directed graphs (arborescences).
Solution
An arborescence rooted at r is a spanning tree with all edges directed toward r (or away, depending on convention).
Tutte's theorem: build the Laplacian L = D_{out} - A where D_{out} is the out-degree diagonal matrix. Delete row r and column r; the determinant of what remains counts the arborescences oriented toward r.
Unlike the undirected case, different roots give different counts — so "the cofactor" is not root-independent here, and which row and column to delete is part of the specification.
BEST theorem combines this with Eulerian circuits: the number of Eulerian circuits in a connected digraph with all in-degrees equal to out-degrees is
ec(G) = \tau_r(G)∏_v(\deg^+(v)-1)!,
where \tau_r is the arborescence count from any root — one of the few problems where counting is polynomial while the corresponding Hamiltonian question is #P-hard.
- Theory
Derive the number of unlabelled rooted trees asymptotically, and explain why there is no closed form.
Solution
Let T(x) = ∑ t_nx^n count unlabelled rooted trees. A tree is a root with a multiset of subtrees, so by Pólya's multiset construction (Chapter 34):
T(x) = x\exp(∑_{k\ge1}(T(x^k))/(k)).
The T(x^k) terms account for repeated identical subtrees and make the equation non-algebraic — unlike the labelled case T = xe^T, which Lagrange inversion solves exactly (Chapter 53).
Asymptotics (Otter, 1948): t_n ~ Cα^n n^{-3/2} with α ≈ 2.9557653 and C ≈ 0.4399240, obtained by singularity analysis of the functional equation. Both constants are believed to be transcendental and have no closed form. ∎
Values: 1, 1, 2, 4, 9, 20, 48, 115, 286 — computable to thousands of terms by iterating the functional equation, but never in closed form.
The contrast with the labelled case is instructive: removing labels destroys the exponential structure and replaces a clean formula with a numerically-defined constant.
- Program
Implement Prüfer encoding and decoding in O(n), and verify they are inverse.
Solution
def prufer_encode(adj, n): deg = [len(adj[v]) for v in range(n)] ptr = next(v for v in range(n) if deg[v] == 1) leaf = ptr code = [] for _ in range(n - 2): nxt = next(u for u in adj[leaf] if deg[u] > 0 or u > leaf) # (in practice, track the single remaining neighbour explicitly) code.append(nxt) deg[nxt] -= 1 if deg[nxt] == 1 and nxt < ptr: leaf = nxt # the new smallest leaf else: ptr += 1 while deg[ptr] != 1: ptr += 1 leaf = ptr deg[leaf] = 0 return code def prufer_decode(code, n): deg = [1]*n for v in code: deg[v] += 1 ptr = next(v for v in range(n) if deg[v] == 1) leaf = ptr edges = [] for v in code: edges.append((leaf, v)) deg[v] -= 1 if deg[v] == 1 and v < ptr: leaf = v else: ptr += 1 while deg[ptr] != 1: ptr += 1 leaf = ptr deg[leaf] = 0 edges.append((leaf, n-1)) return edgesThe O(n) trick is the
ptrpointer: it only ever moves forward, so the total work across all iterations is O(n) rather than the O(nlog n) a heap would cost. The new leaf is either the vertex just decremented (if it is smaller than the pointer) or the next one the pointer finds.Verify by round-tripping all n^{n-2} trees for n\le6 (1296 trees at n=6) and confirming the encode–decode composition is the identity and that every code is produced exactly once.
- Program
Generate a uniformly random labelled tree on n vertices, and a random tree with prescribed degrees.
Solution
import random def random_tree(n): if n == 1: return [] if n == 2: return [(0, 1)] code = [random.randrange(n) for _ in range(n-2)] return prufer_decode(code, n) def random_tree_with_degrees(degs): """degs must sum to 2n − 2, each ≥ 1""" n = len(degs) assert sum(degs) == 2*n - 2 and all(d >= 1 for d in degs) code = [] for v, d in enumerate(degs): code += [v]*(d-1) random.shuffle(code) return prufer_decode(code, n)Uniformity is immediate from the bijection: a uniform random sequence gives a uniform random tree. This is far better than the alternatives — random edge addition with cycle rejection is slow and non-uniform if done carelessly, and a random permutation-based construction gives only path-like trees.
The degree-constrained version is equally direct: the multiset of code symbols is determined by the degrees, so shuffling it uniformly samples uniformly among trees with those degrees.
Verify uniformity for n=4: generate 10⁶ trees and check that all 16 appear about 62{,}500 times each, with a chi-square test.
- Program
Count spanning trees of a graph modulo a prime using the matrix-tree theorem.
Solution
MOD = 10**9+7 def spanning_tree_count(n, edges): L = [[0]*n for _ in range(n)] for u, v in edges: L[u][u] += 1; L[v][v] += 1 L[u][v] -= 1; L[v][u] -= 1 M = [[L[i][j] % MOD for j in range(1, n)] for i in range(1, n)] return det_mod(M, MOD)O(n³), so n\le500 is comfortable — for a count that can be astronomically large (500^{498} for K_{500}).
Multi-edges are handled correctly by the construction (each parallel edge increments the degree and decrements the off-diagonal), and they genuinely multiply the tree count. Self-loops must be discarded — they contribute to no spanning tree and would corrupt the degree.
Validate on the three cases whose answers are known independently: K_n → n^{n-2}, C_n → n, and any tree → 1. Those exercise the sign conventions and the row/column deletion.
For an exact integer answer rather than a residue, use Bareiss (Chapter 45) or several primes plus CRT with the Hadamard bound.
- Program
Count the ways to connect k given components into a spanning tree.
Solution
MOD = 10**9+7 def ways_to_connect(component_sizes): k = len(component_sizes) n = sum(component_sizes) if k == 1: return 1 prod = 1 for a in component_sizes: prod = prod * a % MOD return prod * pow(n, k-2, MOD) % MODThe formula n^{k-2}∏ a_i from problem 5. O(k + log n).
Edge cases: k=1 needs a special case, since n^{-1} is meaningless as written (though the modular inverse happens to give the right answer, relying on that is fragile). k=2 gives a_1a_2, which is a good hand-checkable case.
The typical problem statement is "a graph has some edges; count the ways to add the minimum number of edges to connect it". Run a union-find to get the component sizes, then apply this formula — two lines of real work, and the formula is the entire difficulty.
Verify by brute force for n\le7: enumerate all C(\binom n2,k-1) edge sets of the right size and count how many connect the components.
- Program
Enumerate all spanning trees of a small graph, and count them for a large one.
Solution
from itertools import combinations def all_spanning_trees(n, edges): out = [] for subset in combinations(range(len(edges)), n-1): dsu = DSU(n); ok = True for i in subset: if not dsu.unite(*edges[i]): ok = False; break if ok: out.append(subset) return outBrute force is C(m,n-1) — feasible only for tiny graphs, but the right way to validate the matrix-tree implementation.
For genuine enumeration there are output-sensitive algorithms (Kapoor–Ramesh) running in O(N + V + E) for N spanning trees, based on exchanging one edge at a time. But since N can be n^{n-2}, enumeration is rarely what you want — the count, a random sample, or the minimum is.
Random sampling uniformly from spanning trees is elegant: Wilson's algorithm (loop-erased random walks) or Aldous–Broder produce a uniform spanning tree in expected O(cover time), connecting this chapter to Chapter 41's random walks.
- Program
Compute the number of labelled forests with exactly k trees, for all k, in O(n²).
Solution
MOD = 10**9+7 def forests_by_component_count(n): """F[k] = number of labelled forests on n vertices with exactly k trees""" C = binomial_table(n) rooted = [0]*(n+1) for i in range(1, n+1): rooted[i] = pow(i, i-1, MOD) # rooted trees on i F = [[0]*(n+1) for _ in range(n+1)] F[0][0] = 1 for i in range(1, n+1): for k in range(1, i+1): # the component containing vertex 1 has size j for j in range(1, i-k+2): F[i][k] = (F[i][k] + C[i-1][j-1] * pow(j, j-2, MOD) * F[i-j][k-1]) % MOD return F[n]O(n³) as written; the standard trick of fixing vertex 1's component makes each term unambiguous and avoids overcounting by a factor of k!.
The O(n²) version uses the EGF: forests are sets of trees, so \hat{F}(x,u) = e^{u\hat{T}(x)} and the coefficient of u^k is \hat{T}^k/k! — computable by the power-series toolkit of Chapter 53 in O(nlog n) per k.
Sanity check: F[1] = n^{n-2} (Cayley), F[n] = 1 (all isolated), and ∑_k F[k] is the total number of labelled forests — OEIS A001858: 1, 1, 2, 7, 38, 291.
- Program
Verify Cayley's formula and the degree-constrained count by exhaustive enumeration for small n.
Solution
from itertools import combinations, product from math import factorial def enumerate_trees(n): verts = list(range(n)) all_edges = list(combinations(verts, 2)) trees = [] for subset in combinations(all_edges, n-1): dsu = DSU(n) if all(dsu.unite(u, v) for u, v in subset): trees.append(subset) return trees for n in range(1, 8): trees = enumerate_trees(n) assert len(trees) == (n**(n-2) if n >= 2 else 1), n # degree-constrained check from collections import Counter by_deg = Counter() for t in trees: d = [0]*n for u, v in t: d[u] += 1; d[v] += 1 by_deg[tuple(d)] += 1 for degs, cnt in by_deg.items(): expected = factorial(n-2) for d in degs: expected //= factorial(d-1) assert cnt == expected, (degs, cnt, expected)At n=7 there are C(21,6) = 54{,}264 edge subsets to test, yielding 7^5 = 16{,}807 trees — fast enough, and it validates both Cayley's formula and the multinomial degree count simultaneously.
The degree check is the stronger test: it verifies the refined statement of problem 2, which a bug in the Prüfer implementation would violate even if the total count happened to be right.
Matchings, Hall's theorem and combinatorial duality
Every max-min theorem in combinatorics has the same shape: the largest "packing" equals the smallest "covering". Konig, Hall, Dilworth and Menger are four faces of that one statement, and each turns a hard-looking question into a matching computation.
Hall's condition
The condition is necessary trivially: if |N(S)| < |S| the vertices of S have too few partners. Sufficiency is the theorem, and the standard proof is by induction on |L| with two cases — either every proper non-empty subset has strict slack (|N(S)| > |S|), in which case match any edge and recurse; or some S has |N(S)| = |S|, in which case split the problem at S.
Hall's condition has 2^{|L|} instances, so it is a certificate, not an algorithm: you verify a specific violating set, or you find a matching. To decide, run a matching algorithm — and if it fails, the algorithm hands you the violating set (the reachable vertices from an unmatched one along alternating paths).
Augmenting paths
Berge's lemma: a matching M is maximum iff there is no augmenting path — a path alternating between non-matching and matching edges, with both endpoints unmatched. Flipping such a path increases the matching by one.
bool try_kuhn(int v, vector<bool>& used){
for (int to : adj[v]){
if (used[to]) continue;
used[to] = true;
if (match_r[to] == -1 || try_kuhn(match_r[to], used)){
match_r[to] = v;
return true;
}
}
return false;
}
// for each left vertex: reset `used`, try to augment. O(V·E) total.
// Hopcroft–Karp finds many shortest augmenting paths per phase: O(E√V).| Problem | Algorithm | Cost |
|---|---|---|
| Bipartite maximum matching | Kuhn / Hopcroft–Karp | O(VE) / O(E√V) |
| Bipartite min-cost perfect matching | Hungarian algorithm | O(n³) |
| General graph maximum matching | Blossom (Edmonds) | O(V³) or O(E√V) |
| Minimum path cover on a DAG | n - max matching | O(E√V) |
| Maximum antichain in a poset | Dilworth via matching | O(n³) + matching |
| Maximum independent set (bipartite) | n - max matching (König) | O(E√V) |
The last row is worth emphasising: maximum independent set is NP-hard in general and polynomial on bipartite graphs, purely because of König's theorem. Recognising bipartiteness is therefore often the whole solution.
"Assign n tasks to n workers minimising total cost" is the Hungarian algorithm at O(n³). "Can every task be assigned?" is Hall's condition, decided by a matching. And "what is the minimum number of workers who together can cover every task?" is König's minimum vertex cover — same computation, different reading.
- Theory
Prove Hall's theorem.
Solution
Necessity: if S is matched injectively into N(S), then |N(S)| ≥ |S|.
Sufficiency, by induction on |L|. Two cases:
- Every non-empty proper S ⊂neq L has |N(S)| ≥ |S|+1. Pick any edge (u,v) and match it. In the remaining graph, any S has lost at most one neighbour, so |N'(S)| ≥ |N(S)|-1 ≥ |S| — Hall's condition still holds, and induction finishes.
- Some S has |N(S)| = |S| exactly. By induction match S into N(S) perfectly. For the rest, take any T ⊆ L\setminus S: then |N(S∪ T)| ≥ |S|+|T|, and since N(S∪ T) = N(S)∪ N(T) with |N(S)| = |S|, the neighbours of T outside N(S) number at least |T|. So Hall holds for the remainder and induction applies. ∎
The proof is non-constructive in the sense that it does not suggest an efficient algorithm — augmenting paths do that.
- Theory
Prove König's theorem from Hall's, or directly from max-flow min-cut.
Solution
König: in a bipartite graph, max matching = min vertex cover.
Easy direction: every vertex cover must contain at least one endpoint of each matching edge, and matching edges are disjoint, so |cover| ≥ |matching|.
Hard direction, constructively: take a maximum matching M and let U be the unmatched left vertices. Let Z be the set of vertices reachable from U by alternating paths. Then
C = (L\setminus Z) ∪ (R∩ Z)
is a vertex cover of size exactly |M|. Every edge is covered (an uncovered edge would extend an alternating path), and |C| = |M| because every vertex of C is matched and no matching edge has both endpoints in C. ∎
This construction is what makes König algorithmic: after running any matching algorithm, one BFS gives the minimum vertex cover — and its complement gives the maximum independent set.
- Theory
State Dilworth's theorem and show how to compute the maximum antichain via matching.
Solution
Dilworth: in a finite poset, the minimum number of chains covering all elements equals the maximum antichain size.
Algorithm: build a bipartite graph with a left and a right copy of each element, and an edge u_L → v_R whenever u < v in the poset (using the transitive closure, not just the covering relation). Then
min chain cover = n - max matching,
because each matching edge merges two chains into one. By Dilworth that equals the maximum antichain size.
To recover the antichain, take the minimum vertex cover from König and complement it. ∎
Cost: O(n³/64) for the transitive closure with bitsets, plus O(E√V) for Hopcroft–Karp. Using the covering relation instead of the closure is the classic bug — it computes a path cover rather than a chain cover, and the two differ.
- Theory
Prove Berge's lemma: a matching is maximum iff it admits no augmenting path.
Solution
(⟸ contrapositive) If an augmenting path exists, flipping it yields a larger matching, so M was not maximum.
(⟹ contrapositive) Suppose M is not maximum and let M^* be larger. Consider M \triangle M^* (the symmetric difference). Every vertex has degree at most 2 in it (at most one edge from each matching), so the components are paths and even cycles, alternating between M and M^*.
Cycles have equal numbers of edges from each. Since |M^*| > |M|, some component must have more M^* edges than M edges — necessarily a path starting and ending with M^* edges, i.e. an augmenting path for M. ∎
This lemma is what licenses the whole augmenting-path family of algorithms: it says the greedy "improve while you can" strategy cannot get stuck in a local optimum.
It holds for general graphs too, but finding augmenting paths is harder there because of odd cycles — which is exactly what Edmonds' blossom algorithm handles by contracting them.
- Theory
Explain why maximum independent set is polynomial on bipartite graphs but NP-hard in general.
Solution
The complement of an independent set is a vertex cover, so α(G) = n - \tau(G) for every graph. By König, \tau(G) = \nu(G) (the matching number) on bipartite graphs, and matching is polynomial — hence so is independent set.
In general graphs König fails: the triangle K_3 has \nu = 1 but \tau = 2. The obstruction is odd cycles, and indeed König's theorem characterises bipartite graphs.
Without König, the only relation is the LP-duality bound \nu ≤ \tau ≤ 2\nu, and computing \tau exactly is NP-hard (it is one of Karp's original 21 problems). ∎
The practical lesson: when an NP-hard problem appears, check whether the instance is bipartite, an interval graph, a tree, or otherwise structured. A large family of hard problems becomes polynomial on such classes, and recognising the class is often the whole solution.
- Theory
Derive the minimum path cover of a DAG and prove the formula.
Solution
Minimum path cover: the fewest vertex-disjoint paths covering all vertices of a DAG.
Build a bipartite graph with left copy u_L and right copy v_R for each vertex, with an edge whenever u→ v is an arc. A matching corresponds to choosing, for each vertex, at most one successor and at most one predecessor — that is, a set of disjoint paths.
Starting from n isolated vertices (n paths) and adding each matching edge merges two paths, so
min path cover = n - \nu. ∎
For a chain cover (paths in the transitive closure — i.e. allowing skips), take the transitive closure first; the result is Dilworth's theorem via König.
The distinction matters: minimum path cover uses the arcs as given, minimum chain cover uses reachability. A DAG that is a single long path has path cover 1 either way; a DAG with a "diamond" has path cover 2 and chain cover 2, but examples where they differ are easy to construct.
- Theory
State the deficiency version of Hall's theorem and use it to compute the maximum matching size.
Solution
Deficiency Hall (Ore): the maximum matching in a bipartite graph saturates |L| - d left vertices, where
d = max_{S⊆ L}\big(|S| - |N(S)|\big),
the maximum deficiency over all subsets (which is at least 0, from S=∅).
So \nu = |L| - max_S(|S|-|N(S)|), and Hall's theorem is the special case d = 0. ∎
Proof sketch: add d dummy vertices on the right connected to everything; the modified graph satisfies Hall's condition, so it has a perfect matching on L, and removing the dummies leaves |L|-d matched.
Use: this gives a certificate of optimality. After computing a maximum matching, the set S = L ∩ Z (reachable by alternating paths from unmatched left vertices) achieves the maximum deficiency — so you can prove your answer is optimal without trusting the algorithm.
- Program
Implement Kuhn's algorithm and Hopcroft–Karp; compare on dense and sparse graphs.
Solution
def kuhn(n_left, adj): match_r = [-1]*n_right def try_augment(v, used): for to in adj[v]: if used[to]: continue used[to] = True if match_r[to] == -1 or try_augment(match_r[to], used): match_r[to] = v; return True return False result = 0 for v in range(n_left): if try_augment(v, [False]*n_right): result += 1 return result, match_rKuhn is O(VE) worst case but very fast in practice, especially with the greedy pre-matching optimisation (match any free pair first, which typically finds 80–90% of the matching in O(E)).
Hopcroft–Karp is O(E√V): each phase does a BFS to find the shortest augmenting-path length, then a DFS to find a maximal set of vertex-disjoint shortest paths. There are O(√V) phases, because after √V phases the shortest augmenting path has length >√V and at most √V augmentations remain.
Measured: on sparse graphs with V=10⁵ the difference is 10–50×; on small or dense instances Kuhn with the greedy start often wins on constant factors. Implement Kuhn first, upgrade if it is too slow.
- Program
Compute the minimum vertex cover and maximum independent set of a bipartite graph via König's construction.
Solution
def konig(n_left, n_right, adj, match_r): match_l = [-1]*n_left for r, l in enumerate(match_r): if l != -1: match_l[l] = r # Z = vertices reachable from unmatched LEFT vertices by alternating paths visL = [False]*n_left; visR = [False]*n_right stack = [v for v in range(n_left) if match_l[v] == -1] for v in stack: visL[v] = True while stack: v = stack.pop() for to in adj[v]: # non-matching edges L→R if to == match_l[v] or visR[to]: continue visR[to] = True u = match_r[to] # matching edge R→L if u != -1 and not visL[u]: visL[u] = True; stack.append(u) cover = [('L', v) for v in range(n_left) if not visL[v]] \ + [('R', r) for r in range(n_right) if visR[r]] independent = [('L', v) for v in range(n_left) if visL[v]] \ + [('R', r) for r in range(n_right) if not visR[r]] return cover, independentOne BFS after the matching. Assert
len(cover) == matching_size— that equality is König's theorem, and a violation means either the matching was not maximum or the alternating-path traversal is wrong.Also assert that the cover genuinely covers every edge and that the independent set has no internal edge. Those two checks plus the size equality prove optimality without trusting either algorithm.
- Program
Implement the Hungarian algorithm for minimum-cost perfect matching in O(n³).
Solution
INF = float('inf') def hungarian(cost): """cost is n×m with n ≤ m; returns (min total cost, assignment)""" n, m = len(cost), len(cost[0]) u = [0]*(n+1); v = [0]*(m+1); p = [0]*(m+1); way = [0]*(m+1) for i in range(1, n+1): p[0] = i; j0 = 0 minv = [INF]*(m+1); used = [False]*(m+1) while True: used[j0] = True i0 = p[j0]; delta = INF; j1 = -1 for j in range(1, m+1): if used[j]: continue cur = cost[i0-1][j-1] - u[i0] - v[j] if cur < minv[j]: minv[j] = cur; way[j] = j0 if minv[j] < delta: delta = minv[j]; j1 = j for j in range(m+1): if used[j]: u[p[j]] += delta; v[j] -= delta else: minv[j] -= delta j0 = j1 if p[j0] == 0: break while j0: # augment along the path j1 = way[j0]; p[j0] = p[j1]; j0 = j1 ans = [0]*(n+1) for j in range(1, m+1): if p[j]: ans[p[j]] = j return -v[0], ansThe potentials u and v maintain the invariant c_{ij} - u_i - v_j ≥ 0 with equality on matched pairs — LP duality made explicit (Chapter 62). The algorithm is primal–dual: it grows the matching while adjusting the potentials to keep complementary slackness.
O(n²m) as written, O(n³) for square inputs. Verify against brute-force n! enumeration for n\le8, and note that
-v[0]holding the answer is a well-known quirk of this particular formulation worth commenting. - Program
Compute the maximum antichain of a poset using Dilworth's theorem.
Solution
def max_antichain(n, less_than): """less_than[u][v] = True iff u < v in the poset (TRANSITIVELY CLOSED)""" adj = [[v for v in range(n) if less_than[u][v]] for u in range(n)] size, match_r = hopcroft_karp(n, n, adj) cover, independent = konig(n, n, adj, match_r) # an element is in the antichain iff BOTH its copies avoid the cover in_cover_L = {v for side, v in cover if side == 'L'} in_cover_R = {v for side, v in cover if side == 'R'} return [v for v in range(n) if v not in in_cover_L and v not in in_cover_R]The chain of reductions is: maximum antichain → (Dilworth) minimum chain cover → n - \nu → (König) the complement of a minimum vertex cover.
The transitive closure is essential — Dilworth is about chains (totally ordered subsets), not paths in the covering relation. Compute it with bitset Floyd–Warshall at O(n³/64).
Verify: the returned set must be pairwise incomparable, and its size must equal n - \nu. Both are cheap assertions and together they certify optimality.
Application: the maximum set of mutually independent tasks, the longest antichain in a divisibility poset, and the maximum number of boxes none of which fits inside another — all the same computation.
- Program
Solve: given n intervals and k rooms, decide whether all intervals can be scheduled, and find a violating set if not.
Solution
def can_schedule(intervals, k): """the maximum overlap must not exceed k (Chapter 58)""" events = sorted([(l, 1) for l, r in intervals] + [(r, -1) for l, r in intervals]) cur = best = 0; worst_x = None for x, d in events: cur += d if cur > best: best, worst_x = cur, x if best <= k: return True, None # a violating set: all intervals containing worst_x return False, [i for i, (l, r) in enumerate(intervals) if l <= worst_x < r]For intervals the answer is a simple sweep, because interval graphs are perfect: the chromatic number equals the clique number, so the number of rooms needed is exactly the maximum overlap. No matching is required.
The violating set is the certificate — a point covered by more than k intervals — which is the interval-graph version of Hall's condition and is far easier to exhibit than the general one.
The contrast worth noting: for general graphs, deciding k-colourability is NP-hard; for interval graphs the greedy algorithm on a sweep is optimal. This is the same "check the graph class" lesson as problem 5.
- Program
Implement a Hall's-condition checker for small bipartite graphs by exhaustive subset enumeration.
Solution
def hall_violation(n_left, adj): """returns a violating subset S, or None — O(2^n · n) so n ≤ 20""" worst = None; worst_def = 0 for mask in range(1, 1 << n_left): nbrs = 0 size = 0 for v in range(n_left): if mask >> v & 1: size += 1 for to in adj[v]: nbrs |= 1 << to deficiency = size - bin(nbrs).count('1') if deficiency > worst_def: worst_def, worst = deficiency, mask return worst, worst_defO(2^n n), so usable only for n\le20 — but it directly computes the maximum deficiency of problem 7, and therefore the matching size |L| - d.
Its value is as a test oracle: run it alongside a matching algorithm on random small graphs and assert \nu = |L| - d. That verifies the matching implementation against a completely independent computation, which catches errors that self-consistency checks miss.
Speed it up with a SOS/subset-sum transform (Chapter 30) to compute all |N(S)| in O(2^n n) word operations rather than recomputing the union per mask.
- Program
Implement Edmonds' blossom algorithm for maximum matching in general graphs.
Solution
# The key difficulty over bipartite: an odd cycle (a "blossom") can be # traversed in two directions, so a naive alternating-path search misses # augmentations. Edmonds' insight: CONTRACT the blossom to a single vertex, # find an augmenting path in the smaller graph, then lift it back. def blossom(n, adj): match = [-1]*n; p = [0]*n; base = list(range(n)) def lca(a, b): ... # lowest common ancestor in the alternating forest def mark_path(v, b, child): ... def find_path(root): used = [False]*n; p = [-1]*n; base = list(range(n)) used[root] = True; q = [root] while q: v = q.pop(0) for to in adj[v]: if base[v] == base[to] or match[v] == to: continue if to == root or (match[to] != -1 and p[match[to]] != -1): curbase = lca(v, to) mark_path(v, curbase, to); mark_path(to, curbase, v) for i in range(n): if blossom_flag[base[i]]: base[i] = curbase if not used[i]: used[i] = True; q.append(i) elif p[to] == -1: p[to] = v if match[to] == -1: return to # augmenting path found used[match[to]] = True; q.append(match[to]) return -1O(V³) in this formulation. The blossom contraction is what distinguishes general from bipartite matching: bipartite graphs have no odd cycles, so no blossoms arise and Kuhn's simpler search is complete.
This is a genuinely intricate algorithm — worth having a tested implementation to hand rather than writing under time pressure. Verify against brute force for n\le10 over all graphs, and against a bipartite matcher on bipartite inputs.
The weighted version (minimum-cost general matching) is harder still, at O(V³) with substantial constants, and is where most people reach for a library.
Max-flow min-cut and linear programming duality
The maximum flow equals the minimum cut, and that is a special case of LP duality - every maximisation has a dual minimisation with the same optimum. Recognising a problem as a flow or an LP is usually the entire solution.
What flow models
| Problem | Construction |
|---|---|
| Bipartite matching | source→left (cap 1), edges (cap 1), right→sink (cap 1) |
| Vertex capacities | split v into v_{in}→ v_{out} with the capacity |
| Minimum path cover | matching on the split-vertex graph |
| Project selection / max closure | min cut: profits from s, costs to t |
| Image segmentation | min cut with pairwise smoothness terms |
| Edge/vertex connectivity | min cut over source–sink pairs |
| Assignment with costs | min-cost max-flow |
| Scheduling with deadlines | flow with time-indexed nodes |
The project-selection pattern
The most useful non-obvious modelling trick. Given projects with profits and prerequisites with costs, choose a subset maximising profit minus cost.
Build a network:
s → each profitable item, capacity = its profit
each costly item → t, capacity = its cost
prerequisite edges u → v with capacity ∞ (choosing u FORCES choosing v)
Then: max profit = (sum of all positive profits) − (min cut)
Why: a finite cut cannot cut an ∞ edge, so the s-side is closed under
prerequisites. Cutting an s-edge means forgoing that profit; cutting a
t-edge means paying that cost. Minimising the cut maximises the net.Infinite-capacity edges express implication: if u is on the source side then v must be too. That single device turns "choose a subset subject to closure constraints" into a min cut, and it is the basis of project selection, image segmentation, and the reduction of many 2-variable optimisation problems to flow.
LP duality
Max-flow min-cut is exactly this: the flow LP's dual is the cut LP, and the integrality of the optimum follows from the constraint matrix being totally unimodular — which is also why bipartite matching, shortest paths and min-cost flow all have integer optima automatically.
"Maximise profit choosing projects with dependencies" is the max-closure construction, and the answer that lands is naming the ∞-capacity trick: a finite cut cannot break a dependency, so the source side is automatically a valid selection. That one sentence is the whole reduction.
- Theory
Prove the max-flow min-cut theorem.
Solution
Weak duality: for any flow f and any s–t cut (S,T), the value of f equals the net flow across the cut, which is at most the cut's capacity. So max f ≤ min cut.
Strong duality: run any augmenting-path algorithm to termination. Let S be the set of vertices reachable from s in the residual graph; then t\notin S (else an augmenting path exists).
Every edge from S to T = V\setminus S is saturated (otherwise it would have residual capacity and its head would be in S), and every edge from T to S carries zero flow (otherwise the reverse residual edge would reach into T). Therefore
|f| = ∑_{u∈ S,v∈ T}c(u,v) - 0 = cap(S,T).
So this flow's value equals this cut's capacity, and by weak duality both are optimal. ∎
The proof is constructive: it produces the minimum cut from the final residual graph with one BFS, which is how you extract the cut in practice.
- Theory
Show that bipartite matching is a special case of max flow, and that Dinic's algorithm gives O(E√V).
Solution
Add a source s with unit-capacity edges to every left vertex, unit-capacity edges for the original bipartite edges, and unit-capacity edges from every right vertex to t. An integral flow of value k corresponds exactly to a matching of size k, since unit capacities force each vertex to carry at most one unit.
Integrality is guaranteed because the constraint matrix is totally unimodular — the LP optimum is automatically integral.
Dinic's bound on unit-capacity networks: after O(√E) phases the level graph has depth >√E, and the remaining flow is at most √E because the residual paths are vertex-disjoint. Each phase is O(E), giving O(E√E); for bipartite graphs the sharper analysis gives O(E√V) — matching Hopcroft–Karp exactly. ∎
So Dinic on this network is Hopcroft–Karp, which is why implementing one good flow routine covers both problems.
- Theory
Derive the maximum-closure construction and prove its correctness.
Solution
A closure is a vertex subset closed under outgoing edges. Maximise the total weight of a closure, where weights may be negative.
Build: s→ v with capacity w_v for w_v>0; v→ t with capacity -w_v for w_v<0; and u→ v with capacity ∞ for every dependency.
Correctness: any finite cut (S,T) cannot cut an ∞ edge, so S\setminus{s} is closed. Its capacity is
cap(S,T) = ∑_{v\notin S, w_v>0}w_v + ∑_{v∈ S, w_v<0}(-w_v) = P - ∑_{v∈ S}w_v
where P = ∑_{w_v>0}w_v. Minimising the cut therefore maximises the closure weight, with
maxclosure = P - mincut. ∎
The construction is a bijection between finite cuts and closures, which is what makes the equality exact rather than merely a bound.
- Theory
State LP duality and derive the dual of the max-flow LP.
Solution
Primal (max flow): variables f_e ≥ 0, maximise the net outflow from s subject to capacity f_e ≤ c_e and conservation at every internal vertex.
Dual: introduce y_e ≥ 0 for each capacity constraint and z_v (free) for each conservation constraint. The dual is
min ∑_e c_ey_e subject to y_{uv} + z_v - z_u ≥ 0, z_s = 1, z_t = 0, y ≥ 0.
Interpreting z_v as a "potential", the constraints say y_{uv} ≥ z_u - z_v. At an integral optimum z_v∈{0,1} and y_e∈{0,1}, and z is the indicator of the source side of a cut while y indicates the cut edges. So the dual is exactly minimum cut. ∎
The integrality of the optimum is not automatic for a general LP — it holds here because the incidence matrix of a directed graph is totally unimodular, which is the deep reason flow problems are combinatorially tractable.
- Theory
Explain complementary slackness and use it to certify optimality of a flow.
Solution
Complementary slackness: at optimality, for each pair of primal variable and dual constraint, at least one is "tight":
- x_j > 0 \Rightarrow the j-th dual constraint holds with equality;
- y_i > 0 \Rightarrow the i-th primal constraint holds with equality.
For flow: given a flow f and a cut (S,T), they are simultaneously optimal iff every edge from S to T is saturated and every edge from T to S carries zero flow.
That is checkable in O(E) and constitutes a certificate: you need not trust the algorithm, only verify the two conditions. ∎
The same principle certifies the Hungarian algorithm (Chapter 61 problem 10): the potentials u, v satisfy c_{ij}-u_i-v_j\ge0 with equality on matched pairs, so ∑ u + ∑ v is a lower bound achieved by the matching.
Producing a certificate alongside an answer is good practice generally — it converts "I ran an algorithm" into "here is a proof".
- Theory
Explain why Ford–Fulkerson may not terminate with irrational capacities, and how Edmonds–Karp fixes it.
Solution
With integer capacities each augmentation increases the flow by at least 1, so Ford–Fulkerson terminates in at most |f^*| iterations — but that bound depends on the capacity values, not the graph size, so it is exponential in the input length (the classic bad example uses capacities 10^9 and a bottleneck edge of capacity 1, forcing 2×10^9 augmentations).
With irrational capacities the flow value can converge to a limit strictly below the maximum — Ford and Fulkerson's original counterexample uses (√5-1)/2 and never terminates.
Edmonds–Karp chooses the shortest augmenting path (BFS). One shows the distance from s to any vertex never decreases, and each edge becomes a bottleneck at most O(V) times, giving O(VE) augmentations and O(VE²) total — independent of the capacities. ∎
Dinic improves this to O(V²E) by finding a blocking flow per phase. The lesson: the augmenting-path choice rule is what makes the algorithm polynomial, not the augmenting-path idea itself.
- Theory
Show how to model vertex capacities and lower bounds on edges.
Solution
Vertex capacities: split each vertex v into v_{in} and v_{out}, joined by an edge of capacity c_v. All incoming edges go to v_{in}, all outgoing leave v_{out}. Any flow through v must traverse the internal edge and is therefore bounded. ∎
Lower bounds \ell_e ≤ f_e ≤ c_e: substitute f_e = \ell_e + f'_e with 0≤ f'_e≤ c_e-\ell_e. The lower bounds create an excess or deficit at each vertex:
d_v = ∑_{e→ v}\ell_e - ∑_{e arrow v}\ell_e.
Add a super-source S and super-sink T: for d_v>0 add S→ v with capacity d_v, and for d_v<0 add v→ T with capacity -d_v. Also add t→ s with capacity ∞ to allow circulation. A feasible flow exists iff the max S→ T flow saturates all those edges. ∎
These two reductions cover most "flow with side constraints" problems, and together they let a single max-flow routine solve a surprisingly wide class.
- Program
Implement Dinic's algorithm with scaling and test it on bipartite matching.
Solution
class Dinic: def __init__(self, n): self.n = n; self.g = [[] for _ in range(n)] def add(self, u, v, cap): self.g[u].append([v, cap, len(self.g[v])]) self.g[v].append([u, 0, len(self.g[u])-1]) # residual edge def bfs(self, s, t): self.level = [-1]*self.n; self.level[s] = 0 q = [s] for u in q: for v, cap, _ in self.g[u]: if cap > 0 and self.level[v] < 0: self.level[v] = self.level[u]+1; q.append(v) return self.level[t] >= 0 def dfs(self, u, t, f): if u == t: return f while self.it[u] < len(self.g[u]): e = self.g[u][self.it[u]] v, cap, rev = e if cap > 0 and self.level[v] == self.level[u]+1: d = self.dfs(v, t, min(f, cap)) if d > 0: e[1] -= d; self.g[v][rev][1] += d return d self.it[u] += 1 return 0 def maxflow(self, s, t): flow = 0 while self.bfs(s, t): self.it = [0]*self.n while True: f = self.dfs(s, t, float('inf')) if not f: break flow += f return flowThe
itarray (current-arc optimisation) is what makes Dinic O(V²E) rather than worse: once an edge is exhausted in a phase it is never reconsidered, so the DFS work per phase is O(VE).On unit-capacity networks this is O(E√V) and matches Hopcroft–Karp. Verify by running it on the bipartite construction of problem 2 and comparing against a dedicated matcher.
- Program
Extract the minimum cut from a maximum flow and verify complementary slackness.
Solution
def min_cut(dinic, s): """after maxflow: S = vertices reachable from s in the RESIDUAL graph""" seen = [False]*dinic.n; seen[s] = True stack = [s] while stack: u = stack.pop() for v, cap, _ in dinic.g[u]: if cap > 0 and not seen[v]: seen[v] = True; stack.append(v) cut = [(u, v) for u in range(dinic.n) if seen[u] for v, cap, rev in dinic.g[u] if not seen[v] and dinic.g[v][rev][1] > 0] return seen, cut # certificate check assert sum(original_capacity[u][v] for u, v in cut) == flow_value assert all(residual_capacity(u, v) == 0 for u, v in cut) # saturatedThe reachable set in the residual graph is the minimum cut's source side — the constructive half of the max-flow min-cut proof (problem 1).
The two assertions are the complementary-slackness certificate: the cut's capacity equals the flow value, and every cut edge is saturated. Together they prove optimality independently of the algorithm, which is exactly what you want when debugging a flow implementation whose answer looks suspicious.
- Program
Solve the project-selection problem: choose projects with prerequisites to maximise profit.
Solution
def project_selection(profits, dependencies): """profits[i] may be negative; dependencies (u, v) mean u REQUIRES v""" n = len(profits) s, t = n, n+1 d = Dinic(n+2) total = 0 for i, p in enumerate(profits): if p > 0: d.add(s, i, p); total += p elif p < 0: d.add(i, t, -p) for u, v in dependencies: d.add(u, v, float('inf')) # choosing u forces choosing v return total - d.maxflow(s, t)The answer is P - mincut by problem 3. To recover which projects to select, run the min-cut extraction of problem 9 and take the source side.
Common modelling errors: the dependency edge must point from the dependent project to its prerequisite (not the reverse), and it must have infinite capacity. Reversing it silently computes a different, wrong optimum; using a finite capacity allows the cut to break a dependency.
Verify against brute-force 2^n subset enumeration for n\le18, checking closure validity — that catches both errors immediately.
- Program
Implement min-cost max-flow with SPFA and use it for the assignment problem.
Solution
def mcmf(n, edges, s, t): """SPFA-based successive shortest paths; handles negative costs""" graph = build_residual(n, edges) flow = cost = 0 while True: dist = [INF]*n; dist[s] = 0 inq = [False]*n; prevv = [-1]*n; preve = [-1]*n q = deque([s]); inq[s] = True while q: # SPFA (Bellman–Ford queue) u = q.popleft(); inq[u] = False for i, (v, cap, cst, _) in enumerate(graph[u]): if cap > 0 and dist[u] + cst < dist[v]: dist[v] = dist[u] + cst prevv[v] = u; preve[v] = i if not inq[v]: q.append(v); inq[v] = True if dist[t] == INF: break # no augmenting path f = INF; v = t while v != s: f = min(f, graph[prevv[v]][preve[v]][1]); v = prevv[v] v = t while v != s: e = graph[prevv[v]][preve[v]] e[1] -= f; graph[v][e[3]][1] += f; v = prevv[v] flow += f; cost += f * dist[t] return flow, costSuccessive shortest paths: each augmentation uses a minimum-cost path, which keeps the flow optimal for its value at every step. SPFA handles the negative residual costs that appear after augmenting.
O(VE·flow) worst case. With Johnson potentials (reweight so all costs are non-negative, then use Dijkstra) it becomes O(flow· Elog V) — worth doing whenever the flow value is large.
For the assignment problem this reproduces the Hungarian algorithm's answer; verify against it, and against brute force for n\le8.
- Program
Model and solve: minimum number of guards to cover a grid, or maximum independent set on a grid, via flow.
Solution
# A grid graph is BIPARTITE (colour by (i+j) parity), so König applies: # max independent set = n − max matching # min vertex cover = max matching def grid_max_independent(grid): n, m = len(grid), len(grid[0]) free = [(i,j) for i in range(n) for j in range(m) if grid[i][j] == '.'] idx = {p: k for k, p in enumerate(free)} left = [p for p in free if (p[0]+p[1]) % 2 == 0] adj = {p: [] for p in left} for (i,j) in left: for di,dj in ((0,1),(1,0),(0,-1),(-1,0)): q = (i+di, j+dj) if q in idx: adj[(i,j)].append(q) matching = hopcroft_karp(left, adj) return len(free) - matchingThe whole solution is recognising that a grid graph is bipartite — then König turns an NP-hard-in-general problem into a matching.
This models "place the maximum number of non-attacking pieces", "maximum set of cells no two adjacent", and the domino-tiling feasibility question (a perfect matching exists iff the two colour classes are equal and Hall's condition holds).
For the minimum number of cells to remove so no two adjacent remain, the answer is the minimum vertex cover = the matching size, extracted by the König construction of Chapter 61 problem 9.
- Program
Verify LP duality numerically: solve a small LP and its dual and compare the optima.
Solution
from scipy.optimize import linprog import numpy as np def check_duality(A, b, c): # primal: max cᵀx s.t. Ax ≤ b, x ≥ 0 (linprog minimises, so negate) p = linprog(-np.array(c), A_ub=A, b_ub=b, bounds=[(0, None)]*len(c)) # dual: min bᵀy s.t. Aᵀy ≥ c, y ≥ 0 d = linprog(np.array(b), A_ub=-np.array(A).T, b_ub=-np.array(c), bounds=[(0, None)]*len(b)) return -p.fun, d.fun, abs(-p.fun - d.fun)The two optima agree to numerical precision — strong duality. Also verify complementary slackness: for each i, either y_i = 0 or the i-th primal constraint is tight; and for each j, either x_j = 0 or the j-th dual constraint is tight.
Construct the flow LP explicitly for a small network and confirm the dual's optimum equals the min cut computed combinatorially — that makes the abstract duality concrete and is the most convincing demonstration that max-flow min-cut is not a separate theorem but an instance.
Test the degenerate cases too: an infeasible primal has an unbounded dual, and vice versa — the four-way classification that weak duality alone does not determine.
- Program
Compute global minimum cut (over all vertex pairs) using Stoer–Wagner.
Solution
def stoer_wagner(n, w): """w is a symmetric weight matrix; returns the global min cut value""" verts = list(range(n)) best = float('inf') while len(verts) > 1: # maximum adjacency ordering a = [verts[0]] weights = {v: w[verts[0]][v] for v in verts[1:]} while len(a) < len(verts): nxt = max(weights, key=weights.get) a.append(nxt); del weights[nxt] for v in weights: weights[v] += w[nxt][v] s, t = a[-2], a[-1] cut_of_the_phase = sum(w[t][v] for v in verts if v != t) best = min(best, cut_of_the_phase) # merge t into s for v in verts: if v != s and v != t: w[s][v] += w[t][v]; w[v][s] = w[s][v] verts.remove(t) return bestO(V³) (or O(VE + V²log V) with a heap), and it needs no flow computation at all. The key lemma: in a maximum-adjacency ordering, the last two vertices s and t satisfy "the cut isolating t is a minimum s–t cut" — so either the global min cut separates them (found in this phase) or it does not (and merging them is safe).
The alternative is n-1 max-flow computations with a fixed source, which is correct but slower and much more code.
Verify against brute force over all 2^{n-1}-1 bipartitions for n\le15. Stoer–Wagner is one of those algorithms that is short, non-obvious, and worth having memorised.
Combinatorial game theory: Nim and Sprague-Grundy
Every impartial game under normal play is equivalent to a single Nim heap, and its size is computable by one recursive rule. Sums of games XOR their Grundy values, which is why a position with XOR zero is exactly a loss for the player to move.
Nim
Bouton's theorem: a Nim position with heaps a_1,…,a_n is a loss for the player to move iff a_1⊕ a_2⊕···⊕ a_n = 0.
The proof has two halves. From a zero-XOR position, every move changes exactly one heap and therefore makes the XOR non-zero. From a non-zero-XOR position, let k be the highest set bit of the XOR; some heap has that bit set, and reducing it to a_i ⊕ X (which is smaller, since the bit clears) makes the XOR zero. ∎
The Sprague–Grundy theorem
The XOR in the sum rule is not a coincidence — it is Nim's structure imported. A sum of games is equivalent to a sum of Nim heaps of the corresponding sizes, and Bouton's theorem then says the whole thing is a loss iff the XOR is zero. So "decompose into independent components, compute each Grundy value, XOR" solves every impartial normal-play game.
Standard games and their Grundy values
| Game | Grundy value |
|---|---|
| Nim heap of size n | n |
| Subtraction game, remove 1..k | n mod (k+1) |
| Remove a square number | no pattern; compute by DP |
| Kayles (knock down 1 or 2 adjacent pins) | eventually periodic, period 12 |
| Green Hackenbush | trees: XOR of branch values +1 |
| Turning Turtles / Mock Turtles | odious/evil number structure |
| Wythoff's game | P-positions at (⌊ kφ⌋, ⌊ kφ²⌋) |
Many games have eventually periodic Grundy sequences, which is the practical route for large n: compute the first few thousand values, detect the period, and extrapolate. Proving the periodicity is usually harder than finding it.
"There are k piles; players remove any number from one pile; who wins?" is Nim, answered by the XOR in one line. The follow-up that matters is why: from a zero XOR every move breaks it, and from a non-zero XOR one move restores it — a two-sentence invariant argument (Chapter 7) rather than a case analysis.
- Theory
Prove Bouton's theorem for Nim.
Solution
Let X = a_1⊕···⊕ a_n.
If X = 0, every move gives X ≠ 0: a move changes exactly one heap from a_i to a_i' ≠ a_i, so the new XOR is a_i ⊕ a_i' ≠ 0.
If X ≠ 0, some move gives X = 0: let k be the highest set bit of X. Some heap a_i has bit k set (otherwise X could not). Then a_i ⊕ X < a_i, because bit k flips from 1 to 0 and only lower bits change. Reduce heap i to a_i⊕ X; the new XOR is X⊕ a_i⊕(a_i⊕ X) = 0.
The terminal position (all heaps empty) has X=0 and is a loss. By induction, X=0 positions are exactly the P-positions. ∎
This is an invariant argument in the sense of Chapter 7: X=0 is preserved against the opponent — you can always restore it, and they can never maintain it.
- Theory
Prove the Sprague–Grundy theorem.
Solution
Claim: a position with Grundy value g behaves exactly like a Nim heap of size g.
Proof by induction on the game DAG. Let g = mex{G(p)} over the moves.
- For every h < g, some move leads to a position with Grundy value h — because mex means all of 0,…,g-1 appear among the options. So the position can simulate any Nim move that decreases the heap.
- No move leads to a position with Grundy value exactly g — because mex excludes g. So the position cannot "stay the same", matching Nim.
These are precisely Nim's move rules for a heap of size g. ∎
The sum rule follows: a sum of games is a sum of the equivalent Nim heaps, and Bouton's theorem gives the XOR. Formally one shows G(A+B) = G(A)⊕ G(B) by exhibiting a strategy in A + B + Nim(G(A)⊕ G(B)) that keeps the XOR at zero. ∎
- Theory
Compute the Grundy values for the subtraction game where you may remove 1, 3 or 4 tokens.
Solution
n: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 G: 0 1 0 1 2 3 2 0 1 0 1 2 3 2 0Computed by G(n) = mex{G(n-1), G(n-3), G(n-4)} (omitting negative indices).
The sequence is periodic with period 7 from the start: 0,1,0,1,2,3,2 repeating. So G(n) = G(n mod 7), and the P-positions are n ≡ 0 or 2 ±od 7.
Periodicity is guaranteed for any finite subtraction set: the Grundy value depends only on the previous max(S) values, so the state (a window of that many values) must eventually repeat, and once it does the sequence is periodic from there. The period is at most (max G + 1)^{max S} — enormous in theory, tiny in practice.
- Theory
Analyse the game where a position is a pile of n and a move splits it into two non-empty unequal piles.
Solution
This is Grundy's game. The Grundy value is
G(n) = mex{G(a)⊕ G(n-a) : 0<a<n-a},
using the sum rule because the two resulting piles are independent games.
n: 1 2 3 4 5 6 7 8 9 10 11 12 13 G: 0 0 1 0 2 1 0 2 1 0 2 1 3The sequence is conjectured to be eventually periodic but this is unproven — it has been computed to n > 2×10^{11} with no period found. ∎(not)
That is a striking situation: a completely elementary game whose Grundy sequence resists analysis. It illustrates that the Sprague–Grundy theory tells you how to compute values but says nothing about whether they have a describable pattern.
For a contest, compute the values by DP up to the required n; the O(n²) recurrence is fine to n≈10⁴.
- Theory
Explain Wythoff's game and derive its P-positions.
Solution
Wythoff's game: two piles; a move removes any number from one pile, or the same number from both.
The P-positions are (⌊ kφ⌋, ⌊ kφ²⌋) for k = 0,1,2,…, where φ = (1+√5)/(2):
(0,0) (1,2) (3,5) (4,7) (6,10) (8,13) (9,15) (11,18) ...Why: the two sequences ⌊ kφ⌋ and ⌊ kφ²⌋ form a Beatty pair — since \frac1φ+\frac1{φ²} = 1, they partition the positive integers. That partition is exactly what the greedy P-position construction produces: each new P-position uses the smallest unused integer as its first coordinate and that plus k as its second. ∎
Consequence: given a position (a,b) with a≤ b, it is a P-position iff a = ⌊(b-a)φ⌋ — an O(1) test using the golden ratio, valid for a,b up to 10^{18} provided you compute ⌊ kφ⌋ with enough precision (use integer arithmetic on the Fibonacci-based approximation, not a
double). - Theory
Why is misère play much harder than normal play?
Solution
In misère Nim (the last player to move loses) the answer is nearly the same but not quite: the winning condition is XOR ≠ 0 if some heap has size \ge2, and XOR = 0 if all heaps are size 1. So one special case suffices.
But in general, misère play has no Sprague–Grundy theory. The Grundy value does not determine the outcome of a sum, because the equivalence "this position behaves like a Nim heap" fails — two positions with the same Grundy value can behave differently when added to a third game.
The correct framework is the misère quotient, a commutative monoid capturing exactly the distinctions that matter. For some games it is small and computable; for others it is infinite. ∎
The practical consequence: a problem stating "the player who cannot move wins" is not a routine Grundy computation, and the safe approach is a direct P/N-position DP over the actual state space, or finding the game-specific special case as in misère Nim.
- Theory
Analyse the "staircase Nim" game and show it reduces to ordinary Nim.
Solution
Staircase Nim: coins sit on steps 1,…,n; a move takes any number of coins from step i to step i-1; coins reaching step 0 are out of play. Last to move wins.
Claim: the game is equivalent to Nim on the odd-numbered steps only, so the position is a loss iff \bigoplus_{i odd} c_i = 0.
Why: moving coins from an odd step to an even one is a genuine Nim move (it decreases an odd-indexed heap). Moving from an even step to an odd one increases an odd heap — but the opponent can always undo it by moving those same coins down again, restoring the invariant. So even steps are "free moves" that cannot help. ∎
This is a strategy-stealing / pairing argument, and it is the standard technique for games where some moves are reversible: identify the moves that matter and show the rest can be neutralised.
The same idea analyses "Mock Turtles", coin-turning games, and several token-sliding games.
- Program
Compute Grundy values for an arbitrary impartial game defined by a move function.
Solution
from functools import lru_cache def make_grundy(moves): """moves(state) → iterable of successor states""" @lru_cache(None) def G(state): s = {G(nxt) for nxt in moves(state)} m = 0 while m in s: m += 1 # mex return m return G # example: subtraction game with set S G = make_grundy(lambda n: [n-k for k in S if k <= n]) print([G(n) for n in range(30)])The mex computation is O(|S|) using a set, and memoisation makes the whole table O(states×branching).
For an array-based version (faster, no recursion limit), fill
g[0..n]bottom-up with a boolean scratch array reused per state — that avoids allocating a set per position and is 5–10× faster.Always print the first 50 values and look for a period. Most subtraction games are eventually periodic, and detecting the period turns an O(n) DP into an O(1) lookup, which is what makes n = 10^{18} tractable.
- Program
Implement a Nim solver that reports the winner and, if winning, a specific winning move.
Solution
def nim_move(heaps): X = 0 for h in heaps: X ^= h if X == 0: return None # losing position for i, h in enumerate(heaps): target = h ^ X if target < h: return (i, h - target) # take (h − target) from heap i return None # unreachableThe move is: find a heap whose value decreases when XORed with X, and reduce it to h⊕ X. Such a heap always exists, namely one with the highest set bit of X (problem 1).
Verify by playing: implement a random opponent and confirm that following this strategy from any N-position wins every time over 10⁵ games. That end-to-end test is more convincing than checking the XOR condition, because it exercises the move construction too.
For misère Nim, add the special case: if every heap has size \le1, the winner is determined by the parity of the number of heaps rather than by the XOR.
- Program
Detect periodicity in a Grundy sequence and extrapolate to huge n.
Solution
def find_period(g, min_period_repeats=3): """g is a list of Grundy values; find the smallest eventual period""" n = len(g) for start in range(n // 2): for p in range(1, (n - start) // min_period_repeats + 1): if all(g[i] == g[i + p] for i in range(start, n - p)): return start, p return None start, period = find_period(compute_grundy(5000)) def G_large(n): if n < start: return g[n] return g[start + (n - start) % period]Requiring at least 3 full repetitions before declaring a period guards against coincidence — with only one repetition, any sequence "has a period" equal to half its length.
It is a conjecture, not a proof. For finite subtraction games periodicity is provable (problem 3); for Grundy's game (problem 4) it is open. So verify by computing further: if the period holds for 50,000 terms after being detected at 5,000, it is safe to use in a contest.
Octal games have a rich theory of periodicity, and some (like .6) have periods in the thousands with long pre-periods — which is why the search must allow a substantial
start. - Program
Solve a game on a graph: tokens on vertices of a DAG, a move slides one token along an edge.
Solution
def graph_game(n, adj, token_positions): """DAG; a move slides ONE token along one edge; last to move wins""" from functools import lru_cache @lru_cache(None) def G(v): s = {G(u) for u in adj[v]} m = 0 while m in s: m += 1 return m x = 0 for v in token_positions: x ^= G(v) return 'first' if x else 'second'Each token is an independent game (moving one does not affect the others), so the sum rule applies and the answer is the XOR of the Grundy values.
The DAG requirement matters: with cycles the game may not terminate, and Grundy values are undefined. For games on general graphs where positions can repeat, the standard convention is that a repeated position is a draw, and the analysis becomes a retrograde P/N/D labelling rather than a Grundy computation.
Multiple tokens on the same vertex are fine — their Grundy values XOR, so an even number of tokens on a vertex contributes nothing, which is a useful simplification.
- Program
Implement Wythoff's game with an O(1) P-position test valid to 10^{18}.
Solution
def is_wythoff_losing(a, b): if a > b: a, b = b, a k = b - a # the P-position with difference k is (⌊kφ⌋, ⌊kφ²⌋); test a == ⌊kφ⌋ # compute ⌊kφ⌋ exactly using integer arithmetic on √5 import math lo = (k * (1 + math.isqrt(5 * k * k) // k if k else 0)) // 2 # unreliable # RELIABLE: ⌊kφ⌋ = (k + isqrt(5k²)) // 2 return a == (k + math.isqrt(5 * k * k)) // 2The identity ⌊ kφ⌋ = ⌊\frac{k+√{5k²}}{2}⌋ lets the test use
isqrt— exact integer square root — rather than a floating-point φ.This matters: with k = 10^{18}, kφ needs 19 significant digits and a
doublehas 16, so the floating-point version gives wrong answers for large k. Theisqrtversion is exact for arbitrary size.Verify against a brute-force P-position DP for a,b\le500, then spot-check large values against the Beatty-sequence property (the two sequences must partition the positive integers).
- Program
Write a general game solver that handles both normal and misère play by direct P/N computation.
Solution
from functools import lru_cache def solve_game(moves, terminal_loses=True): """terminal_loses=True → normal play; False → misère""" @lru_cache(None) def win(state): nxt = list(moves(state)) if not nxt: return not terminal_loses # terminal position return any(not win(s) for s in nxt) return winThis computes the raw P/N labelling with no Grundy theory, so it is correct for misère play too — at the cost of losing the sum rule, and therefore of exploring the full product state space rather than each component separately.
The trade-off is stark: Grundy values let you handle a sum of k games in O(∑|S_i|); direct P/N computation needs O(∏|S_i|). So for normal play always use Grundy; for misère, use this and hope the state space is small, or find the game-specific structure.
Use it as a test oracle: for small instances of a normal-play game, verify that
win(state)agrees with G(state) ≠ 0. That validates a Grundy implementation against an independent computation. - Program
Analyse a two-pile game where players remove a square number of tokens from one pile.
Solution
def square_subtraction_grundy(n): g = [0]*(n+1) for i in range(1, n+1): s = set() k = 1 while k*k <= i: s.add(g[i - k*k]); k += 1 m = 0 while m in s: m += 1 g[i] = m return g g = square_subtraction_grundy(100000) # two piles: the position (a, b) is a loss iff g[a] ^ g[b] == 0O(n√n) — about 3×10⁷ at n=10⁵, a few seconds.
The Grundy sequence for square subtraction is not periodic and has no known pattern — the P-positions (G=0) begin 0, 2, 5, 7, 10, 12, 15, 17, 20, 22, 34, 39, 44, …, which look almost periodic with period 5 and then break. So the DP is genuinely necessary.
Once the table is built, any number of piles is handled by the XOR rule — the whole point of Sprague–Grundy. Verify the two-pile answers against the direct P/N solver of problem 13 for a,b\le40.
Convexity, ternary search and the convex hull trick
A convex function has one minimum and no local traps, which makes it searchable in logarithmic time and optimisable greedily. In DP, convexity of the cost turns an O(n^2) transition into O(n log n) or O(n).
Ternary search
double ternary(double lo, double hi, function<double(double)> f){
for (int it = 0; it < 200; it++){ // fixed iterations, not epsilon
double m1 = lo + (hi - lo) / 3;
double m2 = hi - (hi - lo) / 3;
if (f(m1) < f(m2)) hi = m2; else lo = m1;
}
return (lo + hi) / 2;
}
// INTEGER version — the loop above does not terminate correctly on integers
long long ternary_int(long long lo, long long hi, function<long long(long long)> f){
while (hi - lo > 2){
long long m1 = lo + (hi - lo) / 3, m2 = hi - (hi - lo) / 3;
if (f(m1) < f(m2)) hi = m2; else lo = m1;
}
long long best = lo;
for (long long x = lo; x <= hi; x++) if (f(x) < f(best)) best = x;
return best;
}Each iteration discards a third of the range, so the count is log_{1.5} of the reduction — about 200 iterations for full double precision. Use a fixed iteration count, not an epsilon test: with a flat function near the optimum, |f(m_1)-f(m_2)| can be zero long before the interval is small, and an epsilon loop then exits early or never.
Ternary search needs strict unimodality, not convexity — but a plateau breaks it. If f(m_1) = f(m_2) on a flat region, neither side can be safely discarded. For integer domains, finish with a small linear scan; for reals, perturb or use golden-section search, which reuses one evaluation per step and needs 1 instead of 2.
The convex hull trick
DP transitions of the form
are a query "evaluate the lower envelope of the lines y = b_jx + dp_j at x = a_i". The envelope is convex (a min of linear functions), so it is maintained as a hull and queried in O(log n), or O(1) amortised when both the slopes and the queries are monotone.
| Situation | Structure | Cost per op |
|---|---|---|
| Slopes added in sorted order, queries monotone | a deque / pointer | O(1) amortised |
| Slopes sorted, arbitrary queries | binary search on the hull | O(log n) |
| Arbitrary slopes and queries | Li Chao tree | O(log C) |
| Cost satisfies the quadrangle inequality | divide-and-conquer opt | O(nlog n) total |
| Monotone optimal split points | Knuth optimisation | O(n²) from O(n³) |
| Concave/convex Monge condition | SMAWK | O(n) |
Divide-and-conquer optimisation
If the optimal split point opt(i) is non-decreasing in i, then computing the DP row by divide and conquer — solve the middle index, then recurse left with the split range [lo, opt(mid)] and right with [opt(mid), hi] — costs O(nlog n) instead of O(n²).
Monotonicity of opt follows from the quadrangle inequality C(a,c)+C(b,d) ≤ C(a,d)+C(b,c) for a≤ b≤ c≤ d — a condition that is usually easier to verify than to prove monotonicity directly, and which many natural cost functions satisfy.
"Minimise a function of one variable" should prompt the question is it unimodal? — ternary search if yes, and something else if not. For DP, the recognisable shape is dp_i = min_j(dp_j + b_j a_i): a product of a term depending on j and a term depending on i is exactly a line evaluated at a point, and that is the convex hull trick.
- Theory
Prove that a local minimum of a convex function is global.
Solution
Let x^* be a local minimum and suppose f(y) < f(x^*) for some y. For λ∈(0,1), convexity gives
f(λy + (1-λ)x^*) ≤ λf(y) + (1-λ)f(x^*) < f(x^*).
As λ\to0 the point λy+(1-λ)x^* approaches x^*, so every neighbourhood of x^* contains points with strictly smaller value — contradicting local minimality. ∎
Consequence: gradient descent, Newton's method and any local search converge to the global optimum on a convex function. That is why convexity is the dividing line in optimisation — convex problems are solvable, non-convex ones are heuristic territory.
The minimum set is convex (an interval in one dimension), and it is a single point iff f is strictly convex.
- Theory
Prove that ternary search is correct for a strictly unimodal function, and show it fails on a plateau.
Solution
Let f be strictly decreasing then strictly increasing, with minimum at x^*. Take m_1 < m_2 in the current interval.
- If f(m_1) < f(m_2) then x^* < m_2 — because if x^*≥ m_2, f would be decreasing on [m_1,m_2], giving f(m_1)≥ f(m_2). So discard (m_2, hi].
- Symmetrically if f(m_1) > f(m_2), discard [lo, m_1). ∎
Failure on a plateau: if f(m_1) = f(m_2) and the function is flat between them but dips outside — e.g. f(x) = max(0, |x|-1) queried at m_1=-0.5, m_2=0.5 — neither side can be discarded without possibly losing the minimum. Strictly unimodal functions have no such plateau by definition, which is why the hypothesis is stated that way rather than as convexity.
For integer domains, the safe pattern is to narrow until the interval has \le3 points and then scan them.
- Theory
Show that the maximum of linear functions is convex, and derive the convex hull trick.
Solution
Let f(x) = max_i(a_ix+b_i). For any x, y and λ∈[0,1], let i achieve the max at λx+(1-λ)y. Then
f(λx+(1-λ)y) = a_i(λx+(1-λ)y)+b_i = λ(a_ix+b_i)+(1-λ)(a_iy+b_i) ≤ λf(x)+(1-λ)f(y). ∎
So the upper envelope of a set of lines is convex, and by symmetry the lower envelope is concave. Either way, the envelope consists of O(n) pieces in slope order, and a line is on the envelope iff it is not dominated — which is exactly the convex-hull condition in the dual (Chapter 55 problem 3).
The trick: maintain the envelope's pieces in a structure keyed by the x-ranges where each line is optimal. A query is a binary search for the range containing x; an insertion (in slope order) pops dominated lines from the end. Both are the monotone-chain hull algorithm in disguise.
- Theory
State the quadrangle inequality and prove it implies monotone optimal split points.
Solution
Quadrangle inequality (QI): for a≤ b≤ c≤ d,
C(a,c)+C(b,d) ≤ C(a,d)+C(b,c).
("Crossing costs at least as much as nesting.")
Claim: if the cost satisfies QI then opt(i) ≤ opt(i+1), where opt(i) is the smallest minimising split for state i.
Proof sketch: suppose j < k ≤ opt(i) but k is better for i+1. Apply QI to the four combinations (j,i), (k,i), (j,i{+}1), (k,i{+}1); it forces the improvement at i+1 to also be an improvement at i, contradicting the choice of opt(i). ∎
Monotone opt is exactly what divide-and-conquer optimisation needs, and QI is usually far easier to check: for C(i,j) = (prefix[j]-prefix[i])² or C = cost of a contiguous group, it follows from convexity of the underlying function.
- Theory
Explain Knuth's optimisation and the class of DPs it applies to.
Solution
For an interval DP dp[i][j] = min_{i≤ k<j}(dp[i][k]+dp[k+1][j]) + C(i,j), if C satisfies the quadrangle inequality and is monotone on intervals (C(b,c)≤ C(a,d) for a≤ b≤ c≤ d), then
opt[i][j-1] ≤ opt[i][j] ≤ opt[i+1][j].
So when filling dp[i][j], the split search runs only over that bracketed range. Summing the ranges telescopes: the total work per diagonal is O(n), and over all diagonals O(n²) instead of O(n³). ∎
Applies to: optimal binary search trees (Knuth's original application), optimal file merging, matrix-chain multiplication in some formulations, and interval-partitioning costs derived from convex functions.
Does not apply when the cost is not quadrangle-convex — and the failure is silent, producing a wrong answer rather than a slow one. Verify by running the O(n³) version alongside for small n.
- Theory
Derive the Li Chao tree and explain when it beats the standard hull.
Solution
A Li Chao tree is a segment tree over the query coordinate range. Each node stores one line — the one that is best at the node's midpoint.
Inserting a line: compare it with the node's line at the midpoint, keep the better one there, and recurse into the half where the loser might still win (determined by comparing at the endpoints). Since two lines cross at most once, the loser can only be better on one side, so the recursion goes one way — O(log C) per insertion.
Querying x: walk down to the leaf, taking the minimum over the stored lines on the path — O(log C). ∎
When it beats the hull: the standard convex hull trick requires slopes to be inserted in monotone order (or a balanced BST for arbitrary order). Li Chao accepts arbitrary slopes and arbitrary queries with no ordering requirement, at the cost of O(log C) instead of O(1) amortised — and it is far shorter to implement correctly.
It also extends to line segments (insert into O(log C) nodes covering the segment's range), which the hull does not.
- Theory
Give an example where a DP looks like the convex hull trick but is not, and say why.
Solution
The pattern dp_i = min_j(dp_j + b_ja_i) is a line evaluation. But
dp_i = min_j(dp_j + (a_i - a_j)²) = min_j(dp_j + a_j² - 2a_ja_i) + a_i²
also fits, with slope -2a_j and intercept dp_j+a_j² — so squared costs do work after expansion.
What does not work: dp_i = min_j(dp_j + f(a_i, a_j)) where f does not separate into a product plus terms depending on only one index. For instance f = max(a_i, a_j) or f = |a_i - a_j|^{1.5} cannot be written as a line in a_i.
Those cases may still admit divide-and-conquer or Knuth optimisation if the quadrangle inequality holds — which is why the two techniques are complementary: CHT needs linearity in the query variable, D&C optimisation needs only monotonicity of the optimum.
The diagnostic: expand the cost and check whether it has the form (stuff(j))·(stuff(i)) + stuff(j) + stuff(i).
- Program
Implement ternary search for real and integer domains, and test on a function with a plateau.
Solution
def ternary_real(f, lo, hi, iters=200): for _ in range(iters): m1 = lo + (hi - lo) / 3 m2 = hi - (hi - lo) / 3 if f(m1) < f(m2): hi = m2 else: lo = m1 return (lo + hi) / 2 def ternary_int(f, lo, hi): while hi - lo > 2: m1 = lo + (hi - lo) // 3 m2 = hi - (hi - lo) // 3 if f(m1) < f(m2): hi = m2 else: lo = m1 return min(range(lo, hi+1), key=f)The fixed iteration count is deliberate. With 200 iterations the interval shrinks by (2/3)^{200} ≈ 10^{-35} — far below double precision, so the loop is guaranteed to converge without an epsilon test that can misfire on a flat function.
Test on a plateau: f(x) = max(0, |x|-1) on [-5,5]. Ternary search returns some point in [-1,1], all of which are minima — correct here. But f(x) = min(|x-3|, 1) is not unimodal and ternary search may return a non-minimum, which is the failure the hypothesis excludes.
Golden-section search reuses one evaluation per iteration (φ makes the new probe coincide with an old one), halving the function calls — worth it when f is expensive.
- Program
Implement the convex hull trick with a monotone deque and with binary search.
Solution
class CHT: """minimum queries; lines added in DECREASING slope order""" def __init__(self): self.m = []; self.b = [] def _bad(self, l1, l2, l3): # l2 is unnecessary if l3 becomes better than l1 before l2 does return ((self.b[l3]-self.b[l1])*(self.m[l1]-self.m[l2]) <= (self.b[l2]-self.b[l1])*(self.m[l1]-self.m[l3])) def add(self, m, b): self.m.append(m); self.b.append(b) while len(self.m) >= 3 and self._bad(len(self.m)-3, len(self.m)-2, len(self.m)-1): self.m.pop(-2); self.b.pop(-2) def query(self, x): lo, hi = 0, len(self.m)-1 while lo < hi: mid = (lo+hi)//2 if self.m[mid]*x + self.b[mid] <= self.m[mid+1]*x + self.b[mid+1]: hi = mid else: lo = mid+1 return self.m[lo]*x + self.b[lo]The
_badtest uses cross-multiplication rather than division, so it stays exact for integer inputs — dividing to compare intersection x-coordinates introduces rounding that corrupts the hull.Watch for overflow: the products in
_badare of order (Δ b)(Δ m), which for 10^{18}-scale inputs exceeds 64 bits. Use__int128in C++ or Python integers.If queries are also monotone, replace the binary search with a moving pointer for O(1) amortised. If slopes arrive in arbitrary order, use a Li Chao tree instead — it is shorter and harder to get wrong.
- Program
Implement a Li Chao tree supporting arbitrary line insertion and point queries.
Solution
INF = float('inf') class LiChao: def __init__(self, lo, hi): self.lo, self.hi = lo, hi self.line = {} # node → (m, b) def _f(self, line, x): return line[0]*x + line[1] def insert(self, new, node=1, lo=None, hi=None): if lo is None: lo, hi = self.lo, self.hi cur = self.line.get(node) if cur is None: self.line[node] = new; return mid = (lo + hi) // 2 if self._f(new, mid) < self._f(cur, mid): self.line[node], new, cur = new, cur, new if lo == hi: return if self._f(new, lo) < self._f(self.line[node], lo): self.insert(new, 2*node, lo, mid) elif self._f(new, hi) < self._f(self.line[node], hi): self.insert(new, 2*node+1, mid+1, hi) def query(self, x, node=1, lo=None, hi=None): if lo is None: lo, hi = self.lo, self.hi res = INF while True: cur = self.line.get(node) if cur is not None: res = min(res, self._f(cur, x)) if lo == hi: return res mid = (lo + hi) // 2 if x <= mid: node, hi = 2*node, mid else: node, lo = 2*node+1, mid+1The insertion logic rests on two lines crossing at most once: after possibly swapping so the better-at-midpoint line stays, the loser can only win on one side, determined by comparing at that side's endpoint.
O(log C) per operation where C is the coordinate range, with no requirement on insertion or query order — which is why it handles the cases the deque-based hull cannot.
Using a dictionary rather than an array makes it a sparse structure, so a coordinate range of 10^{18} costs only O(nlog C) memory.
- Program
Apply divide-and-conquer DP optimisation to a partition problem and verify against the naive version.
Solution
def dc_optimise(n, k, cost): """dp[j][i] = min over m of dp[j−1][m] + cost(m, i); opt is monotone""" INF = float('inf') prev = [cost(0, i) for i in range(n+1)] for _ in range(2, k+1): cur = [INF]*(n+1) def solve(lo, hi, optlo, opthi): if lo > hi: return mid = (lo+hi)//2 best, arg = INF, optlo for m in range(optlo, min(mid, opthi)+1): v = prev[m] + cost(m, mid) if v < best: best, arg = v, m cur[mid] = best solve(lo, mid-1, optlo, arg) solve(mid+1, hi, arg, opthi) solve(1, n, 0, n) prev = cur return prev[n]O(knlog n) instead of O(kn²). Each level of the recursion does O(n) total work across all its calls, because the split ranges partition [0,n] with only O(1) overlap.
The correctness hypothesis is monotonicity of
opt, which is not checked by the code. Verify it explicitly on small inputs: run the naive O(n²) version, record the argmin for each i, and assert it is non-decreasing. If it is not, the optimisation silently returns a wrong answer.Typical applications: partitioning an array into k segments minimising a convex cost, and the "divide the sequence into k groups" family of problems.
- Program
Solve a classic CHT problem: minimum cost to build towers / segment an array with a linear-in-length penalty.
Solution
# dp[i] = min over j < i of dp[j] + (prefix[i] − prefix[j])·c + K # = min over j of (dp[j] − prefix[j]·c) + prefix[i]·c + K # → lines with slope −prefix[j] and intercept dp[j], queried at x = c def segment_dp(a, c, K): n = len(a) pre = [0]*(n+1) for i, v in enumerate(a): pre[i+1] = pre[i] + v hull = CHT() dp = [0]*(n+1) hull.add(0, 0) # line for j = 0 for i in range(1, n+1): dp[i] = hull.query(pre[i]) + K hull.add(-pre[i], dp[i]) return dp[n]The algebra is the whole technique: expand the cost, group the terms depending only on j into a line's slope and intercept, and the term depending only on i becomes the query point.
O(n) or O(nlog n) depending on whether the slopes and queries are monotone — here the slopes -pre[i] are decreasing (for positive a_i) and the queries increasing, so the deque version applies at O(1) amortised.
Always verify against the O(n²) DP for n\le2000. A sign error in the slope produces a hull that is maintained correctly but represents the wrong lines, and the answer is plausibly wrong rather than obviously so.
- Program
Verify the quadrangle inequality for a cost function empirically and use it to justify Knuth optimisation.
Solution
def check_quadrangle(cost, n): """C(a,c) + C(b,d) ≤ C(a,d) + C(b,c) for a ≤ b ≤ c ≤ d""" bad = [] for a in range(n): for b in range(a, n): for c in range(b, n): for d in range(c, n): if cost(a,c) + cost(b,d) > cost(a,d) + cost(b,c): bad.append((a,b,c,d)) if len(bad) > 5: return bad return bad def check_opt_monotone(dp_naive_argmin): return all(dp_naive_argmin[i] <= dp_naive_argmin[i+1] for i in range(len(dp_naive_argmin)-1))O(n⁴), so usable to n≈60 — plenty to gain confidence.
Checking QI is the sufficient condition; checking that the naive DP's argmin is monotone is the necessary and sufficient one for the optimisation to be valid, and it is cheaper (O(n²)). Do both: QI explains why, monotonicity confirms that.
Cost functions that satisfy QI: C(i,j) = (pre[j]-pre[i])², C(i,j) = w(i,j) for any non-negative weight summed over a submatrix, and the entropy-like costs in optimal-BST problems. Ones that do not: anything involving a max or a non-convex transform of the length.
- Program
Implement SMAWK or the simpler "monotone matrix" row-minima algorithm and compare to divide and conquer.
Solution
def monotone_row_minima(n, m, f): """f(i,j) is TOTALLY MONOTONE; returns argmin_j f(i,j) for each row i in O((n+m) log) by divide and conquer on rows""" res = [0]*n def rec(rlo, rhi, clo, chi): if rlo > rhi: return mid = (rlo+rhi)//2 best, arg = None, clo for j in range(clo, chi+1): v = f(mid, j) if best is None or v < best: best, arg = v, j res[mid] = arg rec(rlo, mid-1, clo, arg) rec(mid+1, rhi, arg, chi) rec(0, n-1, 0, m-1) return resThis is the same divide-and-conquer as problem 11, phrased as "row minima of a totally monotone matrix" — O((n+m)log n).
SMAWK achieves a genuine O(n+m) by alternating "reduce" (discard columns that cannot be a minimum for any surviving row) and "interpolate" (solve the even rows recursively, then fill the odd ones between them). It is short but famously hard to reason about, and the constant is large enough that the O(nlog n) version wins below n≈10⁵.
Both require total monotonicity: for every 2×2 submatrix, if the left column wins in the top row then it wins in the bottom row too. Verify that on small instances before relying on either — the failure mode is again a silently wrong answer.
Bit manipulation as arithmetic
A machine word is a vector over GF(2) and simultaneously a binary numeral, and the useful tricks come from operations that respect both readings at once. Knowing a dozen identities turns loops into single instructions.
Why x & (x-1) works
Subtracting 1 flips the lowest set bit to 0 and every bit below it from 0 to 1. ANDing with the original therefore keeps every bit above the lowest set one and clears everything from it down. That is a single observation with an unreasonable number of consequences — Kernighan's popcount, subset enumeration, and the Fenwick tree's index arithmetic all follow from it.
// iterate over the set bits, once each — O(popcount) not O(width)
for (uint64_t m = x; m; m &= m - 1){
int b = __builtin_ctzll(m); // the lowest set bit's index
...
}
// iterate over every SUBMASK of m, in decreasing order — 3ⁿ total (Chapter 4)
for (int s = m; ; s = (s - 1) & m){
...
if (s == 0) break;
}
// Gosper's hack: the next integer with the same popcount
uint64_t next_same_popcount(uint64_t x){
uint64_t c = x & -x, r = x + c;
return r | (((x ^ r) >> 2) / c);
}A bitmask is simultaneously a subset and a number. Operations that make sense in both readings — AND as intersection and as digitwise minimum, XOR as symmetric difference and as carry-free addition — are where the useful identities live. When a bit trick looks magical, re-read it in the other interpretation and it usually becomes obvious.
Two's complement
Bitset acceleration
Any algorithm whose inner loop is a boolean array operation can be made 64× faster by packing it into words. Chapter 46's GF(2) elimination is the archetype, but the technique is general:
| Algorithm | Bitset form | Speed-up |
|---|---|---|
| Transitive closure | O(n³/64) | 64× |
| Subset-sum DP | shift and OR | 64× |
| Bipartite matching (dense) | adjacency as bitsets | 64× |
| Longest common subsequence | bit-parallel (Hyyrö) | 64× |
| Sieve of Eratosthenes | bit array | 8× memory |
| GF(2) Gaussian elimination | XOR rows | 64× |
"Count the set bits" has three answers worth knowing: __builtin_popcountll (one instruction), Kernighan's while (x) { x &= x-1; c++; } (O(popcount)), and a table lookup (O(width/8)). Naming which is fastest and why — the hardware instruction, unless the CPU lacks it — is the actual question.
- Theory
Prove that x & (-x) isolates the lowest set bit, using two's complement.
Solution
Write x = a10^k in binary, where a is the part above the lowest set bit and there are k trailing zeros.
Then lnot x = \bar a01^k, and -x = lnot x + 1 = \bar a10^k — adding 1 propagates a carry through the k trailing ones, turning them back to zeros and flipping the 0 to 1.
ANDing: the high parts a and \bar a have no common set bit; the isolated position has 1 in both; the trailing zeros are zero in both. So x & (-x) = 0^{|a|}10^k. ∎
Uses: the Fenwick tree's
i += i & -iandi -= i & -itraversals, the set-bit iteration loop, and testing for a power of two.Caveat: for signed types, -x with x = INT_MIN is undefined behaviour in C++. Use unsigned types, where the arithmetic is defined modulo 2^n and the identity holds unconditionally.
- Theory
Prove that iterating
s = (s-1) & mfroms = menumerates every submask exactly once, in decreasing order.Solution
Claim: given a submask s ⊆ m with s > 0, the value (s-1) & m is the largest submask strictly less than s.
Why: s-1 clears s's lowest set bit and sets everything below it. ANDing with m restricts to m's bits, so the result agrees with s above the cleared bit, has a 0 there, and has all of m's bits below — which is the largest possible value with that prefix. ∎
Since each step strictly decreases and always lands on a submask, and it produces the immediate predecessor each time, the enumeration is exactly the submasks in decreasing order, each once.
The
if (s == 0) break;at the end of the body rather than in the loop condition is what includes the empty submask — awhile (s)loop would skip it, which is a common off-by-one.Total work over all m is 3^n (Chapter 4 problem 11).
- Theory
Explain Gosper's hack and prove it produces the next integer with the same popcount.
Solution
c = x & -x // the lowest set bit r = x + c // ripple: merge the lowest block of 1s, carry out ones = ((x ^ r) >> 2) / c return r | onesWhy: let the lowest block of consecutive 1s have length k, starting at position p. Adding c = 2^p carries through the whole block, clearing it and setting bit p+k. That is the smallest possible increase that keeps a 1 at a higher position.
But the carry destroyed k ones and created 1, so k-1 must be restored — and to keep the value minimal they go at the very bottom. The expression x⊕ r is exactly the bits that changed (the old block plus the new bit), shifting right by 2 and dividing by c places k-1 ones at positions 0..k-2. ∎
This enumerates all \binom nk masks of popcount k in increasing order, in O(1) per step — far better than filtering all 2^n masks by popcount.
- Theory
Explain why signed right shift rounds toward negative infinity and what to use for truncating division.
Solution
Arithmetic right shift copies the sign bit, so x \gg 1 computes ⌊ x/2⌋ — rounding down. For x = -7: -7 \gg 1 = -4, while -7/2 = -3 in C (which truncates toward zero).
So \gg and / differ for negative operands, and replacing
x/2byx>>1is only safe for non-negative x.Which do you want? Floor division is usually the mathematically correct one — it makes x mod m non-negative, which is what modular arithmetic needs (Chapter 14). Truncating division is what C gives you.
long long floordiv(long long a, long long b){ long long q = a / b; if ((a % b != 0) && ((a < 0) != (b < 0))) q--; return q; }Python's
//and%already floor, which is why Python code translates to C++ incorrectly at exactly these points — one of the most common porting bugs in competitive programming. - Theory
Derive a bit-parallel algorithm for the longest common subsequence.
Solution
Hyyrö's bit-vector LCS: represent one row of the LCS DP table as a bitmask, where bit j records whether dp[i][j] > dp[i][j-1] (a "step" in the row).
For each character of the second string, update the whole row with a constant number of word operations:
for each character c of B: u = V & Match[c] // positions where c matches V = (V + u) | (V - u) // the carry propagation does the DPThe addition's carry chain performs exactly the "take the max of the left and upper neighbours" propagation that the DP requires — arithmetic carry as a parallel prefix operation. ∎
O(nm/64) instead of O(nm). At n = m = 10⁴ that is 1.5×10⁶ word operations instead of 10⁸.
The same idea gives bit-parallel edit distance (Myers' algorithm) and approximate string matching (Shift-Or / Bitap), all exploiting that a carry chain is a free prefix-scan.
- Theory
Show how to compute the XOR of 0,1,…,n in O(1).
Solution
Look at the XOR of four consecutive integers starting at a multiple of 4:
4k ⊕ (4k+1) ⊕ (4k+2) ⊕ (4k+3) = 0,
because the low two bits cycle through 00,01,10,11 (XOR 0) and the high bits appear four times (XOR 0). So the prefix XOR depends only on n mod 4:
n mod 4 = 0 → n n mod 4 = 1 → 1 n mod 4 = 2 → n + 1 n mod 4 = 3 → 0∎ Then \bigoplus_{i=l}^{r} i = f(r)⊕ f(l-1).
This is the standard tool for "XOR of a range" queries and appears constantly in problems that ask for the XOR of a huge arithmetic-like range. The analogous sum has the closed form n(n+1)/2; the XOR's is this period-4 pattern.
- Theory
Explain the SOS (sum over subsets) transform and its complexity.
Solution
Given f on all 2^n masks, compute F[m] = ∑_{s⊆ m}f[s] for every m.
Naively this is O(3^n) by submask enumeration. The SOS transform does it in O(2^n n) by processing one bit at a time:
for i in range(n): for m in range(1 << n): if m >> i & 1: F[m] += F[m ^ (1 << i)]Why it works: after processing bits 0..i, F[m] equals the sum over submasks that agree with m on bits above i and are arbitrary below. Inducting on i gives the full subset sum at the end. ∎
It is a multidimensional prefix sum over the hypercube {0,1}^n — the direct analogue of a 2D prefix sum, done one dimension at a time. The inverse (Möbius) transform is the same loop with
-=.At n=20: 2^{20}×20 = 2×10⁷ versus 3^{20} = 3.5×10⁹ — the difference between feasible and not.
- Program
Implement popcount four ways and benchmark them.
Solution
// 1. hardware instruction int pc_builtin(uint64_t x){ return __builtin_popcountll(x); } // 2. Kernighan — O(popcount) int pc_kernighan(uint64_t x){ int c = 0; while (x){ x &= x-1; c++; } return c; } // 3. SWAR (divide and conquer) — O(log width), branchless int pc_swar(uint64_t x){ x -= (x >> 1) & 0x5555555555555555ULL; x = (x & 0x3333333333333333ULL) + ((x >> 2) & 0x3333333333333333ULL); x = (x + (x >> 4)) & 0x0F0F0F0F0F0F0F0FULL; return (x * 0x0101010101010101ULL) >> 56; } // 4. table lookup, 8 bits at a time int pc_table(uint64_t x){ int c = 0; for (int i = 0; i < 8; i++){ c += TBL[x & 0xFF]; x >>= 8; } return c; }Typical relative timings: builtin 1×, SWAR 3×, table 6×, Kernighan 2–20× depending on the popcount. The builtin compiles to a single
POPCNTinstruction on any x86-64 since 2008 and on ARMv8.SWAR is worth understanding: each step sums adjacent pairs, then nibbles, then bytes, and the final multiply-and-shift sums the eight byte counts in one operation — a parallel prefix computation done with ordinary arithmetic.
Kernighan wins when the popcount is very small (a sparse mask) and loses badly when it is large.
- Program
Implement subset enumeration, submask enumeration and same-popcount enumeration, and verify the counts.
Solution
def all_subsets(n): return range(1 << n) def submasks(m): s = m while True: yield s if s == 0: break s = (s - 1) & m def same_popcount(k, n): """all n-bit masks with exactly k set bits, increasing""" if k == 0: yield 0; return x = (1 << k) - 1 while x < (1 << n): yield x c = x & -x r = x + c x = r | (((x ^ r) >> 2) // c) from math import comb assert sum(1 for m in all_subsets(12) for _ in submasks(m)) == 3**12 assert sum(1 for _ in same_popcount(5, 12)) == comb(12, 5)The two assertions verify the two counting facts: submask pairs number 3^n, and same-popcount masks number \binom nk. Both are exact and catch any off-by-one in the loop termination.
The
if s == 0: breakplacement insubmasksis what includes the empty set; testing that0 in submasks(m)for every m is worth an explicit assertion. - Program
Implement the SOS transform and use it to count, for each mask, the number of array elements that are submasks of it.
Solution
def sos_transform(f, n): F = f[:] for i in range(n): for m in range(1 << n): if m >> i & 1: F[m] += F[m ^ (1 << i)] return F def count_submask_elements(values, n): cnt = [0] * (1 << n) for v in values: cnt[v] += 1 return sos_transform(cnt, n) # F[m] = #{v : v ⊆ m} # the SUPERSET version: iterate the other way def sos_superset(f, n): F = f[:] for i in range(n): for m in range(1 << n): if not (m >> i & 1): F[m] += F[m | (1 << i)] return FO(2^n n). The superset version flips the condition and the direction — both are needed, and confusing them is the usual bug.
Applications: counting pairs with a & b = 0 (for each a, the count of b that are submasks of lnot a), subset-sum convolution (Chapter 30), and any "for each mask, aggregate over its submasks" query.
Verify against the naive O(3^n) double loop for n\le12 — that also demonstrates the speed difference concretely.
- Program
Implement a Fenwick tree and explain its index arithmetic in terms of bit tricks.
Solution
class Fenwick: def __init__(self, n): self.n = n; self.t = [0]*(n+1) def add(self, i, v): i += 1 while i <= self.n: self.t[i] += v i += i & -i # move to the next covering node def prefix(self, i): i += 1; s = 0 while i > 0: s += self.t[i] i -= i & -i # strip the lowest set bit return sThe structure: node i covers the range (i - (i&{-i}), i] — a block whose length is the lowest set bit of i. So
i -= i & -iwalks from i down through a decomposition of [1,i] into O(popcount(i)) disjoint blocks, andi += i & -iwalks up through all the nodes whose block contains i.Both loops run at most log n times, and the whole structure is one array of size n+1 — no pointers, no recursion, excellent cache behaviour. That is why a Fenwick tree beats a segment tree by a factor of 3–5 in practice despite the same asymptotics.
The bit trick is the data structure: the tree shape is exactly the binary representation of the indices, which is why the code is six lines with no explicit tree at all.
- Program
Use bitsets to compute the transitive closure and to speed up a subset-sum DP.
Solution
# transitive closure — O(n³/64) reach = [1 << i for i in range(n)] # each vertex reaches itself for k in range(n): for i in range(n): if reach[i] >> k & 1: reach[i] |= reach[k] # subset-sum feasibility — O(n·S/64) possible = 1 # bit s set ⟺ sum s achievable for a in items: possible |= possible << a possible &= (1 << (S+1)) - 1 # keep only sums ≤ S print((possible >> target) & 1)Python's arbitrary-precision integers are bitsets, and
<<,|,&operate word-parallel internally — so this idiom gives the 64× speed-up with no extra machinery.The subset-sum line
possible |= possible << ais the entire DP: shifting by a represents "also take this item", and OR-ing merges the two options. At n = 100 and S = 10⁵ this is 100×10⁵/64 = 1.5×10⁶ word operations — instant, versus 10⁷ for the boolean-array version.In C++ use
std::bitset<S>for a compile-time size or avector<uint64_t>with a hand-written shift. - Program
Write bit-twiddling routines for: reversing bits, next power of two, and integer log base 2.
Solution
uint32_t reverse_bits(uint32_t x){ x = ((x >> 1) & 0x55555555) | ((x & 0x55555555) << 1); x = ((x >> 2) & 0x33333333) | ((x & 0x33333333) << 2); x = ((x >> 4) & 0x0F0F0F0F) | ((x & 0x0F0F0F0F) << 4); x = ((x >> 8) & 0x00FF00FF) | ((x & 0x00FF00FF) << 8); return (x >> 16) | (x << 16); } uint64_t next_pow2(uint64_t x){ if (x <= 1) return 1; return 1ULL << (64 - __builtin_clzll(x - 1)); } int ilog2(uint64_t x){ return 63 - __builtin_clzll(x); } // x > 0 requiredBit reversal is a butterfly network: swap adjacent bits, then pairs, then nibbles, and so on — log_2 w steps. It is exactly the permutation the FFT needs (Chapter 50), and the same structure explains why that permutation is its own inverse.
next_pow2uses x-1 so that an exact power of two maps to itself — the classic off-by-one. Guard x=0 separately, sinceclz(0)is undefined.ilog2must never be implemented as(int)log2((double)x): for large x the floating-point result can be off by one ulp at exact powers of two, giving a wrong answer that survives most testing (Chapter 11 problem 13). - Program
Write a test harness verifying bit-trick identities exhaustively over 16-bit values.
Solution
for x in range(1 << 16): # lowest set bit if x: assert x & -x == 1 << (x & -x).bit_length() - 1 assert (x & (x-1)) == x - (x & -x) # power of two assert ((x != 0) and (x & (x-1)) == 0) == (bin(x).count('1') == 1) # popcount agreement across implementations assert pc_kernighan(x) == pc_swar(x) == bin(x).count('1') # bit reversal is an involution assert reverse_bits16(reverse_bits16(x)) == x # next power of two if x: assert next_pow2(x) >= x and next_pow2(x) < 2*x for m in range(1 << 12): subs = list(submasks(m)) assert len(subs) == 1 << bin(m).count('1') assert all(s & m == s for s in subs) assert len(set(subs)) == len(subs)Exhaustive testing over 16 bits is 65,536 cases — instant, and it is a proof for that width (Chapter 2: a finite domain fully enumerated is verified, not merely tested).
The identities that generalise to 64 bits do so structurally, so a 16-bit exhaustive check plus a 64-bit random check is strong coverage. The exception is anything involving
INT_MINor width-specific constants, which need explicit 64-bit tests.Keeping this harness is worthwhile: bit tricks are exactly the kind of code that is copied, adapted, and silently broken.
IEEE 754 floating point and error analysis
Floating-point numbers are a logarithmically-spaced finite subset of the reals, and arithmetic on them is exact-then-rounded. Almost every surprising result follows from that one fact plus the loss of associativity.
What breaks
Integers up to 2^{53} are represented exactly in a double, so integer arithmetic in floating point is safe below that — and silently wrong above it. That threshold, about 9×10^{15}, is why JavaScript's Number.MAX_SAFE_INTEGER exists and why 64-bit IDs must not be passed through JSON as numbers.
Catastrophic cancellation
// WRONG when b² much larger than 4ac: one root loses all its digits
x1 = (-b + sqrt(b*b - 4*a*c)) / (2*a);
x2 = (-b - sqrt(b*b - 4*a*c)) / (2*a);
// RIGHT: compute the well-conditioned root, then use x1*x2 = c/a
double q = -0.5 * (b + copysign(sqrt(b*b - 4*a*c), b));
x1 = q / a;
x2 = c / q;
// a = 1, b = 1e8, c = 1:
// naive x1 = 0.0 (all significant digits lost)
// stable x1 = -1e-8 (correct)Subtracting nearly-equal quantities destroys significant digits — the relative error of the difference is the absolute error divided by a tiny result. The fix is always algebraic: rewrite so the subtraction never happens. √{x+1}-√x becomes 1/(√{x+1}+√x); 1-\cos x becomes 2\sin²(x/2); e^x-1 becomes expm1(x).
Comparing floats
"Why is 0.1 + 0.2 != 0.3?" — because 0.1 and 0.2 are not exactly representable in binary, so each is rounded, and the sum of the rounded values rounds to a double slightly above 0.3. The follow-up worth volunteering: the same happens in decimal with 1/3, so this is not a flaw in binary but a consequence of any finite radix.
- Theory
Explain exactly why 0.1+0.2 ≠ 0.3, giving the stored values.
Solution
0.1 in binary is 0.00011 — infinitely repeating, like 1/3 in decimal. The nearest double is
0.1000000000000000055511151231257827…
Similarly 0.2 rounds slightly up. Their exact sum is 0.3000000000000000166533…, which rounds to the double 0.30000000000000004440892…
But the double nearest the literal
0.3is 0.29999999999999998889776… — one ulp below. Hence the inequality. ∎The general fact: a fraction p/q in lowest terms is exactly representable in base b iff every prime factor of q divides b. In binary only powers of 2 qualify, so 0.5 and 0.25 are exact while 0.1 and 0.3 are not.
- Theory
Show that floating-point addition is not associative, with a maximally bad example.
Solution
Take a = 10^{20}, b = -10^{20}, c = 1.
(a+b)+c = 0 + 1 = 1, but a+(b+c) = 10^{20} + (-10^{20}+1) = 0, since -10^{20}+1 rounds back to -10^{20} (the 1 is far below the ulp at that magnitude). ∎
Consequences:
- Summing an array gives different results in different orders. Kahan or pairwise summation reduces the error from O(nε) to O(ε) and O(εlog n).
- Parallel reduction is non-deterministic unless the order is pinned — a genuine reproducibility problem in scientific computing and ML training.
- Compilers may not reorder float arithmetic without
-ffast-math, precisely because it changes results.
Multiplication is also non-associative, but far milder: relative errors compose rather than cancel.
- Theory
Derive the error bound for summing n floats, and explain Kahan summation.
Solution
Naive left-to-right summation incurs a relative error ≤ε per step, accumulating to
|error| ≤ nε∑|x_i| + O(ε²).
The ∑|x_i| factor is the problem: with cancellation the true sum can be tiny while ∑|x_i| is huge, so the relative error is unbounded.
Kahan summation tracks the lost low bits:
s = c = 0.0 for x in xs: y = x - c t = s + y c = (t - s) - y # the part of y that was LOST s = tThe bound improves to 2ε∑|x_i| — independent of n. ∎
Warning: the compiler must not simplify
(t - s) - yto zero, which it is mathematically entitled to do. Avoid-ffast-math, or use pairwise summation, which gives O(εlog n) with no such fragility and is what NumPy uses. - Theory
Explain why tolerance comparison is not an equivalence relation, and what to use instead.
Solution
Define x ~ y \iff |x-y| < ε. Reflexive and symmetric, but not transitive: with ε = 1, 0\sim0.6 and 0.6\sim1.2 but 0\not\sim1.2. ∎
Consequences: it cannot be
- a sort comparator — the strict weak ordering fails, and
std::sortthen has undefined behaviour that can read out of bounds; - a hash-map key — equal-by-tolerance values hash differently;
- a grouping key — the "groups" depend on comparison order.
The fix is to snap to a grid: k = round(x/ε), then compare or hash the integer k. That is an equivalence relation — the kernel of a function (Chapter 5) — at the cost of separating two values that straddle a grid boundary.
For exact geometric predicates the real answer is to avoid floats entirely (Chapter 54).
- a sort comparator — the strict weak ordering fails, and
- Theory
Define the condition number and compute it for subtraction.
Solution
The relative condition number of f at x is κ(x) = |xf'(x)/f(x)| — the factor by which a relative input error is amplified.
For subtraction f(a,b) = a-b: perturbing a by a relative \delta changes the result by a\delta, a relative change of a\delta/(a-b). So
κ = (|a|+|b|)/(|a-b|),
which is enormous when a≈ b — the analytic statement of catastrophic cancellation. ∎
Addition of same-sign numbers has κ=1 (perfectly conditioned); multiplication and division have κ=1 in each argument; √{} has κ=1/2 and reduces error.
So the only genuinely dangerous elementary operation is subtraction of nearly-equal quantities — which is why every stability fix in this chapter is about avoiding one.
- Theory
Explain subnormal numbers and why they can cause a 100× slowdown.
Solution
Subnormals have a zero exponent field and an implicit leading 0 rather than 1, filling the gap between 0 and the smallest normal ≈2.2×10^{-308}, down to ≈5×10^{-324} with progressively fewer significant bits.
Their purpose is gradual underflow: without them x-y could be zero for distinct x and y, breaking the guarantee that x≠ y \Rightarrow x-y\ne0.
The performance problem: many CPUs handle subnormals in microcode or trap to a slow path, costing 100–200 cycles instead of 4. A DSP loop whose values decay into the subnormal range slows by two orders of magnitude with no other symptom.
Mitigations: set the FTZ (flush-to-zero) and DAZ (denormals-are-zero) CPU flags, as audio and game engines routinely do; or add a tiny DC offset. Modern x86 since Skylake handles most subnormal operations at full speed, so the problem is less acute — but it still bites on older hardware and some ARM cores.
- Theory
Explain why
NaN != NaNand enumerate what it breaks.Solution
NaN represents "not a number". IEEE 754 specifies that every comparison involving NaN except
!=returns false, soNaN == NaNis false. The rationale: NaN is not a value, and making all ordered comparisons false letsisnan(x)be written asx != x.What it breaks:
- Sorting: a
<comparator is not a strict weak ordering with NaN present —std::sorthas undefined behaviour and can crash or corrupt memory. - Containers: a NaN key can never be found again, since lookup uses equality.
- Negation:
!(a < b)is nota >= bwhen NaN is involved — both are false, so De Morgan-style rewriting of float comparisons is unsound (Chapter 1 problem 8). - Min/max:
std::min(NaN, x)depends on argument order.
Defence: check for NaN at data boundaries and assert its absence before sorting. One NaN silently poisons every aggregate computed downstream.
- Sorting: a
- Program
Implement Kahan and pairwise summation; compare errors against exact rational summation.
Solution
from fractions import Fraction import math def naive_sum(xs): s = 0.0 for x in xs: s += x return s def kahan_sum(xs): s = c = 0.0 for x in xs: y = x - c t = s + y c = (t - s) - y s = t return s def pairwise_sum(xs): if len(xs) <= 128: return naive_sum(xs) m = len(xs) // 2 return pairwise_sum(xs[:m]) + pairwise_sum(xs[m:]) exact = float(sum(Fraction(x) for x in xs)) for name, f in (("naive", naive_sum), ("kahan", kahan_sum), ("pairwise", pairwise_sum), ("fsum", math.fsum)): print(name, abs(f(xs) - exact) / abs(exact))On 10⁶ values of widely differing magnitudes, typical relative errors: naive 10^{-10}, pairwise 10^{-15}, Kahan 10^{-16},
math.fsumexact (full-precision accumulation, correctly rounded).Fractiongives the exact answer for comparison — which is what makes this a measurement rather than a comparison of two approximations.Try the adversarial input
[1e100, 1.0, -1e100]: naive gives 0.0, Kahan gives 0.0 (it cannot recover a value already lost from the running sum), andfsumgives 1.0. That distinguishes "compensated" from "exact". - Program
Write a stable quadratic solver and demonstrate the naive version failing.
Solution
import math def quadratic_naive(a, b, c): d = math.sqrt(b*b - 4*a*c) return ((-b + d)/(2*a), (-b - d)/(2*a)) def quadratic_stable(a, b, c): disc = b*b - 4*a*c if disc < 0: return None sq = math.sqrt(disc) q = -0.5 * (b + math.copysign(sq, b)) return (q/a, c/q) if q != 0 else (0.0, 0.0) a, b, c = 1.0, 1e8, 1.0 print(quadratic_naive(a, b, c)) # small root comes out 0.0 or garbage print(quadratic_stable(a, b, c)) # (-1e+08, -1e-08) — both correctWith b = 10⁸, √{b²-4ac} ≈ b - 2×10^{-8}, so -b + √{·} subtracts two numbers agreeing in 16 digits — every significant digit is lost.
The stable version computes the root away from the cancellation (
copysignensures the addition never cancels) and obtains the other from x_1x_2 = c/a.Verify by substituting both roots back: |ax²+bx+c| should be at rounding-error level for both, which the naive version fails dramatically for the small root.
- Program
Implement ULP-based comparison and contrast with absolute and relative tolerances.
Solution
import struct def ulp_diff(a, b): """number of representable doubles between a and b""" ia = struct.unpack('<q', struct.pack('<d', a))[0] ib = struct.unpack('<q', struct.pack('<d', b))[0] if ia < 0: ia = (1 << 63) - ia # map to a monotone integer order if ib < 0: ib = (1 << 63) - ib return abs(ia - ib) def almost_equal(a, b, max_ulps=4, abs_eps=1e-300): if a == b: return True if abs(a - b) <= abs_eps: return True # handles ±0 and near-zero return ulp_diff(a, b) <= max_ulpsReinterpreting a double's bits as a signed integer gives a monotone map, so the integer difference counts the representable values between them — a scale-free closeness measure.
Why it is better: an absolute tolerance is wrong for large values (10^{20} and 10^{20}+1 are the same double); a relative tolerance is wrong near zero. ULP comparison is correct in both regimes, with the small absolute guard handling the crossing of zero.
Still not transitive, so it remains unusable as a sort or hash key — a better predicate, not a repair of the underlying non-equivalence.
- Program
Demonstrate order-dependence of summation and quantify the spread.
Solution
import random from fractions import Fraction xs = [random.uniform(-1e6, 1e6) for _ in range(10000)] exact = float(sum(Fraction(x) for x in xs)) results = set() for _ in range(1000): ys = xs[:]; random.shuffle(ys) results.add(naive_sum(ys)) print(len(results), min(results), max(results), exact) print("spread in ulps:", ulp_diff(min(results), max(results)))Typically hundreds of distinct results spread over thousands of ulps. Sorting by increasing magnitude before summing markedly reduces the error (small values are added while the running sum is still small); decreasing magnitude makes it worse.
The practical consequence: parallel reductions, GPU kernels and multi-threaded aggregations are non-deterministic in their last digits unless the reduction order is pinned — the same program on the same data giving different answers on different thread counts.
Fixes: pin the order, use compensated summation, or accumulate in a wider type (
long double,__float128). - Program
Write stable versions of √{x+1}-√x, 1-\cos x and ln(1+x).
Solution
import math def sqrt_diff(x): # sqrt(x+1) - sqrt(x) = 1/(sqrt(x+1) + sqrt(x)) — no subtraction at all return 1.0 / (math.sqrt(x+1) + math.sqrt(x)) def one_minus_cos(x): # 1 - cos x = 2 sin^2(x/2) s = math.sin(x/2) return 2*s*s def log1p_stable(x): return math.log1p(x) # the library function; never write log(1+x)Each rewrite eliminates a subtraction of nearly-equal quantities. At x = 10^{16} the naive √{x+1}-√x gives exactly 0.0 while the stable form gives 5×10^{-9}, correct to full precision.
At x = 10^{-8}, 1-\cos x naively gives 0.0 (since \cos x rounds to 1.0) while 2\sin²(x/2) gives 5×10^{-17}.
The standard library provides
log1p,expm1,hypot,fmaandcbrtprecisely because the obvious formulas are unstable. Use them — writinglog(1+x)orsqrt(x*x+y*y)by hand is a numerical bug, not a style preference. - Program
Implement exact float-to-decimal conversion and verify round-tripping.
Solution
from fractions import Fraction import struct, random def exact_decimal(x): """the EXACT value of a double — usually 700+ digits""" return Fraction(x) # Fraction(float) is exact for _ in range(10**6): bits = random.getrandbits(64) x = struct.unpack('<d', struct.pack('<Q', bits))[0] if x != x or math.isinf(x): continue assert float(repr(x)) == x # shortest round-tripEvery finite double has an exact decimal representation (since 2^{-k} terminates in decimal), typically 700+ digits for small values —
Fraction(0.1)displays the exact stored value.The shortest round-tripping representation is a different, harder problem, solved by Steele–White's Dragon4 and now Ryu and Grisu: the shortest decimal string that parses back to the same double. 17 significant digits always suffice; the shortest is usually 15–17.
Round-tripping 10⁶ random bit patterns verifies this exhaustively — and it is exactly the property that lets you serialise a double to text and back without loss, which naive
%.15gformatting does not guarantee. - Program
Write an audit that detects float-related bugs: NaN, infinities, excessive dynamic range, subnormals.
Solution
import math def audit_floats(values, name=""): problems = [] n_nan = sum(1 for v in values if v != v) n_inf = sum(1 for v in values if math.isinf(v)) if n_nan: problems.append(f"{name}: {n_nan} NaN values") if n_inf: problems.append(f"{name}: {n_inf} infinities") finite = [v for v in values if math.isfinite(v)] nonzero = [abs(v) for v in finite if v] if nonzero: mx, mn = max(nonzero), min(nonzero) if mx/mn > 1e15: problems.append(f"{name}: dynamic range {mx/mn:.1e} exceeds double " "precision — expect cancellation") if any(v < 2.2e-308 for v in nonzero): problems.append(f"{name}: subnormal values present (slow path)") return problemsThree checks that catch most real problems: NaN or infinity entering the data, a dynamic range exceeding what a double holds simultaneously (guaranteeing cancellation in any sum), and subnormals (a performance trap).
Run it at every data boundary — after parsing, after a numerical routine, before serialising. One NaN entering a pipeline silently poisons every aggregate downstream, and catching it at the boundary is vastly cheaper than debugging the aggregate.
Add a review rule that no float is used as a dictionary or sort key without snapping (problem 4) — a grep can enforce it.
Numerical methods: root finding, integration, optimisation
When a closed form does not exist, iterate. The three questions for any numerical method are the same: does it converge, how fast, and what does it cost per step - and the answers are usually a trade between robustness and speed.
Bisection and Newton
double bisect(function<double(double)> f, double lo, double hi){
// REQUIRES a sign change: f(lo) and f(hi) must have opposite signs
for (int i = 0; i < 200; i++){ // fixed count, not an epsilon test
double mid = lo + (hi - lo) / 2; // NOT (lo+hi)/2 — that can overflow
if ((f(mid) < 0) == (f(lo) < 0)) lo = mid; else hi = mid;
}
return lo + (hi - lo) / 2;
}
double newton(function<double(double)> f, function<double(double)> df,
double x, int iters = 50){
for (int i = 0; i < iters; i++){
double d = df(x);
if (d == 0) break; // flat: Newton is undefined
double nx = x - f(x)/d;
if (nx == x) break; // converged to machine precision
x = nx;
}
return x;
}Bisection always converges given a sign change, at one bit per step. Newton converges quadratically or not at all — it can cycle, diverge, or run off to infinity from a bad start. The production answer is a hybrid: take a Newton step, and if it leaves the bracketing interval, fall back to bisection. That is Brent's method, and it is what every library uses.
Integration
Newton in more than one place
Newton's method is not only for scalar roots. The same "linearise and solve" step gives:
| Setting | Newton step |
|---|---|
| Scalar root | x arrow x - f(x)/f'(x) |
| Integer square root | x arrow (x + n/x)/2 |
| Reciprocal without division | x arrow x(2 - ax) |
| Power series inverse | B arrow B(2 - AB) (Chapter 53) |
| Modular inverse mod 2^k | x arrow x(2-ax) mod 2^{2k} (Chapter 17) |
| Systems of equations | x arrow x - J^{-1}F(x) |
| Optimisation | x arrow x - H^{-1}\nabla f |
The reciprocal iteration is why hardware dividers and gmp compute 1/a by Newton on f(x) = 1/x - a — it needs only multiplication, and the precision doubles each step, so a full-precision reciprocal costs about the same as one multiplication at the final size.
"Compute a square root without the library" expects Newton: x arrow (x+n/x)/2, quadratic convergence, about 5 iterations from a decent start. The follow-up worth pre-empting: for an integer square root, floating point is wrong above 2^{53} and you must finish with an integer Newton step plus a correction loop.
- Theory
Prove that Newton's method converges quadratically near a simple root.
Solution
Let r be a simple root (f(r)=0, f'(r)\ne0) and e_n = x_n - r. Taylor-expand about x_n:
0 = f(r) = f(x_n) - e_nf'(x_n) + \tfrac12e_n²f''(\xi).
Dividing by f'(x_n) and rearranging with x_{n+1} = x_n - f(x_n)/f'(x_n):
e_{n+1} = (f''(\xi))/(2f'(x_n))e_n².
So |e_{n+1}| ≤ C|e_n|² with C ≈ |f''(r)/(2f'(r))| — quadratic. ∎
At a multiple root f'(r)=0 and convergence degrades to linear with rate 1-1/m for multiplicity m. The repair is the modified iteration x arrow x - mf/f', or Newton applied to f/f', which restores quadratic convergence without knowing m.
The proof is local: it says nothing about whether the iteration reaches the neighbourhood at all, which is the subject of problem 2.
- Theory
Give three distinct ways Newton's method can fail, with examples.
Solution
- Zero derivative: f(x) = x³-2x+2 from x_0=0 cycles between 0 and 1 forever — a period-2 orbit, not divergence.
- Divergence: f(x) = x^{1/3} has root 0, but Newton gives x_{n+1} = -2x_n — the iterate doubles away from the root every step.
- Overshoot to a different basin: for a polynomial with several roots, a step from near a local extremum can land arbitrarily far away. The basins of attraction form a fractal (the Newton fractal), so an arbitrarily small change in the start can change which root you find.
The unifying cause: Newton assumes the function is well-approximated by its tangent over the whole step, which fails when f' is small or f'' is large relative to the step.
The defence is bracketing: maintain an interval known to contain a root, and reject any Newton step leaving it — falling back to bisection. That gives guaranteed convergence with Newton's speed whenever Newton behaves.
- Theory
Derive Simpson's rule and explain why it is exact for cubics.
Solution
Fit a parabola through (a,f(a)), (m,f(m)), (b,f(b)) with m=(a+b)/2, and integrate it exactly:
\int_a^b f ≈ (b-a)/(6)(f(a)+4f(m)+f(b)).
Why cubics too: the error term involves f''' integrated against a weight that is odd about the midpoint, so it integrates to zero. Formally the error is
-((b-a)^5)/(2880)f^{(4)}(\xi) — depending on the fourth derivative, so any cubic is integrated exactly. ∎
This is the general phenomenon for symmetric quadrature rules: an even number of nodes placed symmetrically gains one extra degree of exactness for free.
Gauss–Legendre pushes this to the limit: choosing both the nodes and the weights optimally gives exactness up to degree 2n-1 with n nodes — twice what a fixed-node rule achieves.
- Theory
Explain Richardson extrapolation and derive Romberg integration.
Solution
If a method has error expansion A(h) = A + c_1h^p + c_2h^{2p}+···, then combining two step sizes eliminates the leading term:
(2^pA(h/2) - A(h))/(2^p-1) = A + O(h^{2p}).
Romberg applies this repeatedly to the trapezoid rule (which has p=2 and only even powers of h, by Euler–Maclaurin):
R[0][k] = trapezoid with 2^k intervals R[j][k] = (4^j·R[j−1][k] − R[j−1][k−1]) / (4^j − 1)The first extrapolation gives exactly Simpson's rule; the second gives Boole's; and R[k][k] has error O(h^{2k+2}). ∎
Practically: 5–6 levels reach machine precision for a smooth integrand, using about 64 function evaluations. Each level reuses all previous evaluations, so nothing is wasted.
The catch: the error expansion must exist, which requires the integrand to be smooth. For a function with a singularity or a kink, Romberg extrapolates a non-existent expansion and can be worse than the plain trapezoid rule.
- Theory
Compare deterministic quadrature with Monte Carlo and identify the crossover dimension.
Solution
A product rule with n nodes per dimension uses n^d evaluations in d dimensions and achieves error O(n^{-k}) = O(N^{-k/d}) for a degree-k-exact rule with N=n^d total points.
Monte Carlo achieves O(N^{-1/2}) — independent of d.
Crossover: N^{-k/d} = N^{-1/2} when d = 2k. For Simpson (k=4) that is d = 8; for trapezoid (k=2), d=4. ∎
So beyond about 4–8 dimensions, Monte Carlo wins — and the gap widens exponentially. That is the "curse of dimensionality", and it is why high-dimensional integrals in physics, finance and Bayesian statistics are all done by sampling.
Quasi-Monte Carlo (Sobol or Halton low-discrepancy sequences) improves the rate to O((log N)^d/N) — better than N^{-1/2} for moderate d, and the practical choice in finance where d ≈ 100 but the effective dimension is much lower.
- Theory
Explain why the secant method's convergence order is φ, the golden ratio.
Solution
The secant method replaces f'(x_n) by the finite difference through the last two points. Its error satisfies
e_{n+1} ≈ Ce_ne_{n-1}.
Positing |e_{n+1}| ~ K|e_n|^p and substituting gives |e_n|^p ~ |e_n|·|e_n|^{1/p}, so p = 1 + 1/p, i.e. p²-p-1 = 0 and
p = (1+√5)/(2) = φ ≈ 1.618. ∎
Why it can still beat Newton: Newton has order 2 but needs two evaluations per step (f and f'). Comparing per-evaluation, secant achieves φ ≈ 1.618 against Newton's √2 ≈ 1.414. So when the derivative costs as much as the function — which is typical — secant is faster.
Newton wins when the derivative is cheap (a polynomial evaluated by Horner gives f and f' in one pass, Chapter 49) or available analytically at no cost.
- Theory
Derive the integer square root by Newton and prove the termination condition.
Solution
For ⌊√n⌋ use the integer iteration
x_{k+1} = ⌊(x_k + ⌊ n/x_k⌋)/(2)⌋.
Claim: from any x_0 ≥ ⌊√n⌋, the sequence decreases until it reaches ⌊√n⌋ and then either stays or oscillates by one.
Why: by AM–GM, (x+n/x)/(2) ≥ √n for all x>0, so every iterate after the first is ≥⌊√n⌋. And while x > ⌊√n⌋ the step strictly decreases. So the standard loop condition "stop when the next value is ≥ the current" terminates exactly at the floor. ∎
def isqrt(n): if n < 2: return n x = 1 << ((n.bit_length() + 1) // 2) # a good starting bound while True: y = (x + n // x) // 2 if y >= x: return x x = yRoughly loglog n iterations from that start. Never use
int(math.sqrt(n)): for n > 2^{53} the double cannot represent n exactly and the result can be off by one — silently, and only for specific inputs. - Program
Implement bisection, Newton, secant and Brent; compare iteration counts on several functions.
Solution
def bisect(f, lo, hi, iters=200): flo = f(lo) for _ in range(iters): mid = lo + (hi - lo) / 2 fm = f(mid) if (fm < 0) == (flo < 0): lo, flo = mid, fm else: hi = mid return lo + (hi - lo) / 2 def secant(f, x0, x1, iters=100, tol=1e-15): f0, f1 = f(x0), f(x1) for _ in range(iters): if f1 == f0: break x2 = x1 - f1 * (x1 - x0) / (f1 - f0) if abs(x2 - x1) < tol * max(1.0, abs(x2)): return x2 x0, f0, x1, f1 = x1, f1, x2, f(x2) return x1Typical iteration counts to 10^{-15} on f(x)=x²-2 from a reasonable start: bisection 52, secant 7, Newton 5, Brent 6.
Then run them on the pathological cases of problem 2 (x³-2x+2 from 0, and x^{1/3}): bisection and Brent still converge, Newton and secant fail. That contrast is the whole argument for using a hybrid method by default.
Use
lo + (hi-lo)/2rather than(lo+hi)/2— the latter can overflow for large bounds and is less accurate near the interval's endpoints. - Program
Implement adaptive Simpson integration with an error estimate.
Solution
def simpson(f, a, b): m = (a + b) / 2 return (b - a) / 6 * (f(a) + 4*f(m) + f(b)) def adaptive_simpson(f, a, b, eps=1e-10, whole=None, depth=50): if whole is None: whole = simpson(f, a, b) m = (a + b) / 2 left, right = simpson(f, a, m), simpson(f, m, b) if depth <= 0 or abs(left + right - whole) <= 15 * eps: return left + right + (left + right - whole) / 15 # Richardson return (adaptive_simpson(f, a, m, eps/2, left, depth-1) + adaptive_simpson(f, m, b, eps/2, right, depth-1))The 15 comes from the error expansion: halving the interval reduces the O(h^4) error by 2^4 = 16, so |L+R-W| ≈ 15× the error of L+R. Adding (L+R-W)/15 is a free Richardson extrapolation to O(h^6).
The
depthguard is essential: a discontinuous or oscillatory integrand can recurse forever, and without a limit the program hangs rather than returning a poor answer.Adaptivity concentrates the evaluations where the function is difficult — integrating 1/√x on [0,1], it spends nearly all its budget near 0 and returns 2.0 to 9 digits, where a uniform rule would need millions of points.
- Program
Implement Romberg integration and show the convergence table.
Solution
def romberg(f, a, b, levels=8): R = [[0.0]*(levels) for _ in range(levels)] h = b - a R[0][0] = h/2 * (f(a) + f(b)) for k in range(1, levels): h /= 2 # reuse all previous evaluations; add only the new midpoints s = sum(f(a + (2*i-1)*h) for i in range(1, 2**(k-1) + 1)) R[k][0] = R[k-1][0]/2 + h*s for j in range(1, k+1): R[k][j] = R[k][j-1] + (R[k][j-1] - R[k-1][j-1]) / (4**j - 1) return R[levels-1][levels-1], RPrinting the table shows the columns converging at increasing rates. For \int_0^1 e^x: column 0 gains 2 digits per row, column 1 gains 4, column 2 gains 6 — reaching machine precision by row 5 with 33 function evaluations.
The
R[k][0]recurrence reuses every previous evaluation, so doubling the resolution costs only the new midpoints. Recomputing the trapezoid rule from scratch each level would double the total work.Run it on |x| over [-1,1] to see the failure mode: the kink breaks the Euler–Maclaurin expansion, and the extrapolated columns are worse than the plain trapezoid rule. That is the caveat in problem 4 made visible.
- Program
Implement an exact integer square root and cube root, and verify against brute force.
Solution
def isqrt(n): if n < 0: raise ValueError if n < 2: return n x = 1 << ((n.bit_length() + 1) // 2) while True: y = (x + n // x) // 2 if y >= x: return x x = y def icbrt(n): if n < 0: return -icbrt(-n) if n < 2: return n x = 1 << ((n.bit_length() + 2) // 3) while True: y = (2*x + n // (x*x)) // 3 if y >= x: return x x = y for n in range(10**6): r = isqrt(n) assert r*r <= n < (r+1)*(r+1) for n in range(0, 10**18, 10**11): r = icbrt(n) assert r**3 <= n < (r+1)**3The assertions state the defining property directly — r²≤ n<(r+1)² — which is stronger and clearer than comparing against a floating-point reference.
The cube-root iteration is Newton on x³-n: x arrow (2x + n/x²)/3. The same pattern gives any integer k-th root with x arrow ((k-1)x + n/x^{k-1})/k.
Compare against
int(n ** (1/3))for n near 10^{18}: the floating-point version is off by one for many inputs, and Python'smath.isqrt(added in 3.8) exists precisely because of this. - Program
Solve a system of non-linear equations with multivariate Newton.
Solution
import numpy as np def newton_system(F, J, x0, iters=50, tol=1e-12): x = np.array(x0, dtype=float) for _ in range(iters): Fx = np.array(F(x), dtype=float) if np.linalg.norm(Fx) < tol: return x Jx = np.array(J(x), dtype=float) # SOLVE, do not invert — Chapter 44 dx = np.linalg.solve(Jx, -Fx) x = x + dx if np.linalg.norm(dx) < tol * max(1.0, np.linalg.norm(x)): return x return xEach step solves JΔ x = -F at O(n³) — using
solverather than forming J^{-1}, which is both slower and less accurate (Chapter 44 problem 13).Convergence is quadratic near a solution where J is non-singular, and the same failure modes as the scalar case apply, amplified: a near-singular Jacobian makes the step enormous.
Practical safeguards: a line search (accept the step only if ‖F‖ decreases, else halve it), and a Levenberg–Marquardt damping (J^TJ+λ I)Δ x = -J^TF that interpolates between Newton (λ\to0) and gradient descent (λ large). Quasi-Newton methods (Broyden, BFGS) approximate J from successive evaluations, avoiding the derivative entirely.
- Program
Compare deterministic and Monte Carlo integration as the dimension grows.
Solution
import random, math, itertools def product_rule(f, d, n): """trapezoid product rule: n^d evaluations""" h = 1.0/n total = 0.0 for idx in itertools.product(range(n+1), repeat=d): w = 1.0 for i in idx: w *= 0.5 if i in (0, n) else 1.0 total += w * f([i*h for i in idx]) return total * h**d def monte_carlo(f, d, N): return sum(f([random.random() for _ in range(d)]) for _ in range(N)) / N f = lambda x: math.exp(-sum(v*v for v in x)) for d in (1, 2, 4, 8, 16): print(d, product_rule(f, d, 8), monte_carlo(f, d, 8**min(d,5)))At d=1 the product rule is far more accurate for the same budget. By d=8 it needs 9^8 = 43 million evaluations for the same nodes-per-dimension, and Monte Carlo with a fraction of that is competitive. Beyond d≈10 the product rule is simply infeasible.
That is the curse of dimensionality made concrete: the same accuracy per dimension costs exponentially more total work, while Monte Carlo's O(N^{-1/2}) does not notice the dimension at all.
Add a Sobol sequence for comparison and observe the quasi-Monte Carlo rate sitting between the two.
- Program
Implement gradient descent with a line search and compare to Newton on a badly-conditioned quadratic.
Solution
import numpy as np def gradient_descent(f, grad, x0, iters=10000, tol=1e-10): x = np.array(x0, dtype=float) for i in range(iters): g = np.array(grad(x)) if np.linalg.norm(g) < tol: return x, i t = 1.0 # backtracking line search fx = f(x) while f(x - t*g) > fx - 0.5*t*g.dot(g) and t > 1e-20: t *= 0.5 x = x - t*g return x, iters # f(x, y) = x² + 100y² — condition number 100 f = lambda v: v[0]**2 + 100*v[1]**2 grad = lambda v: [2*v[0], 200*v[1]]Gradient descent on this needs hundreds of iterations, zig-zagging across the narrow valley; Newton converges in one step, because the function is its own quadratic model.
The convergence rate of gradient descent is ((κ-1)/(κ+1))^2 per step for a quadratic with condition number κ — so κ=100 gives a factor of 0.96 per step, needing about 400 iterations for 10^{-8}.
The practical middle ground: L-BFGS approximates the Hessian from a few recent gradients, achieving near-Newton convergence at gradient-descent cost per step, and is what most optimisation libraries use for smooth problems. For very high dimensions where even that is too much memory, momentum and Adam approximate the same effect stochastically.
Information theory: entropy and coding bounds
Entropy is the average number of bits needed to identify an outcome, and it is a hard floor on compression. The same quantity bounds decision-tree depth, database index size and the number of comparisons a sort requires.
The source coding theorem
Shannon: any uniquely decodable code for a source with entropy H has expected length ≥ H bits per symbol, and a code with expected length < H+1 exists.
The lower bound comes from Kraft's inequality: a prefix code with lengths \ell_i exists iff ∑ 2^{-\ell_i}\le1. Minimising ∑ p_i\ell_i subject to that gives \ell_i = -log_2 p_i — and Gibbs' inequality shows no other choice does better.
Repeatedly merge the two least-frequent symbols into a node whose
frequency is their sum. The resulting tree's depths are the code lengths.
O(n log n) with a heap; O(n) if the frequencies arrive sorted.
OPTIMALITY (exchange argument, Chapter 2):
In some optimal tree the two least-frequent symbols are siblings at the
greatest depth — if not, swapping them there does not increase the cost.
Merging them is therefore safe, and induction finishes.
Expected length: H ≤ L < H + 1. The gap is because code lengths are
INTEGERS; arithmetic coding removes it, reaching H + ε.Entropy is a floor, not a target. No lossless compressor beats H bits per symbol for that model — so all compression progress comes from better models, not better coders. LZ77 exploits repetition, PPM and context mixing exploit conditional structure, and each lowers the effective entropy rather than beating the bound.
Where the same bound appears
| Question | Information bound |
|---|---|
| Comparison sorting | log_2(n!) ≈ nlog_2 n comparisons (Chapter 13) |
| Searching a sorted array | log_2 n comparisons |
| Identifying 1 of n outcomes | log_2 n yes/no questions |
| Weighing puzzles (3 outcomes/query) | log_3 N weighings |
| Lossless compression | H bits per symbol |
| Storing a set of k from n | log_2\binom nk bits |
| Bloom filter (with error ε) | nlog_2(1/ε) bits — beats exact storage |
| Twenty Questions | 20 bits, so 10⁶ objects |
The Bloom filter row is the interesting one: allowing a false-positive rate ε drops the requirement from log_2C(U,n) bits to 1.44nlog_2(1/ε) — independent of the universe size. That is the information-theoretic reason approximate membership is so much cheaper than exact.
"How many weighings to find the odd coin among 12?" is ⌈log_3 24⌉ = 3, because each weighing has three outcomes and there are 24 answers (which coin, and heavy or light). Deriving the bound before constructing the scheme is the mark of understanding — and it immediately tells you 13 coins is the true maximum.
- Theory
Prove that entropy is maximised by the uniform distribution.
Solution
By Gibbs' inequality (or Jensen applied to the concave log), for any distribution p on n outcomes and the uniform q:
D(p‖q) = ∑ p_ilog(p_i)/(1/n) = log n - H(p) ≥ 0,
with equality iff p = q. So H(p)≤log_2 n with equality exactly for the uniform distribution. ∎
The non-negativity of KL divergence follows from Jensen: -D(p‖q) = ∑ plog(q/p) ≤ log∑ p·(q/p) = log 1 = 0.
Interpretation: uncertainty is greatest when nothing is known, and any structure — any deviation from uniform — reduces the bits needed. That is exactly why compression works: real data is far from uniform.
- Theory
State and prove Kraft's inequality.
Solution
Kraft: a binary prefix code with lengths \ell_1,…,\ell_n exists iff ∑_i 2^{-\ell_i}\le1.
(⟹) Embed the code in an infinite binary tree. A codeword of length \ell occupies a node at depth \ell, and the prefix property means no codeword is an ancestor of another — so their subtrees are disjoint. A depth-\ell subtree contains a fraction 2^{-\ell} of the leaves at any deeper level, and disjointness forces the fractions to sum to at most 1.
(⟸) Sort the lengths increasing and assign codewords greedily in binary-counter order, each time skipping past the subtree just consumed. The inequality guarantees there is always room. ∎
Consequence: minimising ∑ p_i\ell_i subject to Kraft gives \ell_i = -log_2 p_i by Lagrange multipliers, and the resulting cost is exactly H. Rounding up to integers gives Shannon–Fano coding with L < H+1.
McMillan's theorem extends the inequality to all uniquely decodable codes, not just prefix ones — so prefix codes lose nothing.
- Theory
Prove that Huffman coding is optimal.
Solution
Lemma: some optimal prefix code has the two least-frequent symbols x, y as siblings at maximum depth.
Why: in an optimal tree, the two deepest leaves are siblings (an unpaired deepest leaf could be moved up). If they are not x and y, swap them with x and y — the cost change is (p_x-p_a)(\ell_a-\ell_x)\le0 since p_x≤ p_a and \ell_a≥\ell_x. So the swap does not increase the cost. ∎
Induction: merging x and y into a symbol of weight p_x+p_y reduces the alphabet by one, and the cost of any tree relates to the merged tree's cost by exactly p_x+p_y — a constant. So an optimal tree for the merged problem yields an optimal tree for the original. ∎
This is an exchange argument (Chapter 2 problem 8), the standard shape for proving a greedy algorithm optimal.
Note the limits: Huffman is optimal among symbol codes with integer lengths. Its redundancy is at most p_{max}+0.086 bits per symbol, which is significant for skewed alphabets — a symbol with probability 0.9 needs 0.15 bits but gets 1.
- Theory
Show that the comparison-sort lower bound is an entropy argument.
Solution
A sorting algorithm must distinguish n! permutations. Each comparison yields at most 1 bit (two outcomes), so at least log_2(n!) comparisons are needed — the decision-tree bound of Chapter 13.
As entropy: the output is uniform over n! possibilities, so H = log_2(n!) bits must be extracted, and each comparison supplies at most H(outcome) ≤ 1 bit. ∎
The entropy framing generalises immediately:
- If the input is partially sorted, the output entropy is lower and fewer comparisons suffice — which is exactly what adaptive sorts like Timsort exploit.
- A comparison with a biased outcome distribution supplies less than 1 bit, so an algorithm making predictable comparisons is wasteful.
- Three-way comparisons supply log_2 3 bits, giving the bound log_3(n!) — which is why ternary comparisons help.
Radix sort escapes because reading a b-bit key supplies b bits at once, not 1.
- Theory
Compute the entropy of English text at several model orders and interpret the numbers.
Solution
Uniform over 27 symbols log₂27 = 4.75 bits/char Order-0 (letter frequencies) ≈ 4.14 bits/char Order-1 (given the previous) ≈ 3.56 Order-2 ≈ 3.3 Order-3 ≈ 3.1 Shannon's human experiments ≈ 1.0–1.3 bits/char Modern neural language models ≈ 0.6–1.0 bits/charEach additional order of context lowers the conditional entropy, because H(X|Y)≤ H(X) always (conditioning never increases entropy).
Interpretation: gzip achieves about 2 bits/char on English, PPM and modern context-mixing compressors about 1.5, and large language models under 1. The gap between 4.14 and 1.0 is entirely a modelling gap — the coder is already optimal for its model.
This is the practical content of the chapter's Principle box: compression research is model research. Shannon's 1951 estimate of 1 bit/char, obtained by having humans guess the next letter, was a prediction of what a perfect model could achieve — and it took 60 years to approach it.
- Theory
Derive the optimal Bloom filter parameters from an information-theoretic viewpoint.
Solution
With m bits, n items and k hash functions, a bit is 0 with probability (1-1/m)^{kn}≈ e^{-kn/m}, so the false-positive rate is
ε ≈ (1-e^{-kn/m})^k.
Minimising over k gives k^* = \frac mnln2 and ε = 2^{-k^*}, hence
m = -(nlnε)/((ln2)²) ≈ 1.44nlog_2(1/ε) bits.
≈ 9.6 bits per element for ε = 1%. ∎
The information-theoretic bound for approximate membership is nlog_2(1/ε) bits, so a Bloom filter is within a factor of 1/ln2 = 1.44 of optimal. Cuckoo filters and quotient filters close most of that gap.
The striking part is what is absent: the universe size U. Exact membership needs log_2\binom Un ≈ nlog_2(U/n) bits, which grows with U. Accepting a 1% error makes the cost independent of how large the keys are — which is why Bloom filters are used for URLs, hashes and arbitrary strings alike.
- Theory
Define mutual information and explain its use in feature selection and decision trees.
Solution
I(X;Y) = H(X)-H(X|Y) = H(Y)-H(Y|X) = D(p_{XY}‖p_Xp_Y) — the reduction in uncertainty about X from learning Y, symmetric and non-negative, zero exactly when X and Y are independent.
Decision trees: at each node choose the feature maximising I(label;feature) — the "information gain" criterion of ID3 and C4.5. It is exactly "which question tells me the most".
Feature selection: rank features by I(label;feature), or better by conditional mutual information given the already-selected ones, which avoids picking redundant features.
The bias worth knowing: raw information gain favours high-cardinality features (an ID column perfectly determines the label and has maximal gain, while generalising not at all). C4.5's gain ratio divides by H(feature) to correct for this — an entropy-based fix for an entropy-based flaw.
- Program
Compute the entropy of a data stream at several model orders.
Solution
from collections import Counter, defaultdict import math def entropy(counts): total = sum(counts.values()) return -sum(c/total * math.log2(c/total) for c in counts.values() if c) def conditional_entropy(text, order): """H(next | previous `order` symbols)""" ctx = defaultdict(Counter) for i in range(order, len(text)): ctx[text[i-order:i]][text[i]] += 1 total = sum(sum(c.values()) for c in ctx.values()) return sum(sum(c.values())/total * entropy(c) for c in ctx.values()) for k in range(5): print(k, conditional_entropy(text, k))The values decrease with the order, and multiplying by the length gives a lower bound on the compressed size achievable with that model.
Watch for overfitting: at high orders most contexts are seen once, so the measured conditional entropy tends to 0 — not because the text is predictable but because the model has memorised it. The honest measure holds out data, or adds a smoothing term (Chapter 36's Laplace smoothing).
Comparing length× H_k against
gzip's output size on the same text is illuminating: gzip typically lands between the order-2 and order-3 bounds. - Program
Implement Huffman coding and compare its length to the entropy bound.
Solution
import heapq from collections import Counter def huffman(freqs): h = [(f, i, s) for i, (s, f) in enumerate(freqs.items())] heapq.heapify(h) nxt = len(h) parent = {} while len(h) > 1: f1, _, a = heapq.heappop(h) f2, _, b = heapq.heappop(h) node = ('node', a, b) heapq.heappush(h, (f1+f2, nxt, node)); nxt += 1 _, _, root = h[0] codes = {} def walk(node, prefix=""): if isinstance(node, tuple) and node[0] == 'node': walk(node[1], prefix + "0"); walk(node[2], prefix + "1") else: codes[node] = prefix or "0" # single-symbol edge case walk(root) return codes codes = huffman(freqs) total = sum(freqs.values()) L = sum(freqs[s]/total * len(c) for s, c in codes.items()) print("Huffman:", L, "entropy:", entropy(freqs), "gap:", L - entropy(freqs))The gap is always in [0,1), and is largest for skewed distributions where the ideal length -log_2 p is far from an integer.
The tie-breaking index in the heap tuple matters — without it, Python compares the node objects and raises. It also makes the tree deterministic, which is necessary if the decoder rebuilds it from the frequencies.
Edge case: a single distinct symbol gives an empty code; the
prefix or "0"guard handles it. That case appears in real data (a run of identical bytes) and crashes naive implementations. - Program
Implement arithmetic coding and show it beats Huffman on a skewed source.
Solution
from fractions import Fraction def arithmetic_encode(symbols, probs): lo, hi = Fraction(0), Fraction(1) for s in symbols: width = hi - lo cum = Fraction(0) for t, p in probs.items(): if t == s: break cum += p hi = lo + width * (cum + probs[s]) lo = lo + width * cum # the shortest binary fraction strictly inside [lo, hi) return shortest_binary_in(lo, hi) def bits_needed(lo, hi): import math return math.ceil(-math.log2(hi - lo)) + 1Arithmetic coding represents the entire message as one number in [0,1), so it does not round each symbol to an integer number of bits. Its length is within 2 bits of -log_2 P(message), giving H+ε per symbol rather than Huffman's H+1.
The gain is largest for skewed sources: with P(a)=0.9, entropy is 0.469 bits/symbol, Huffman must spend 1, and arithmetic coding achieves 0.47 — a 2× improvement.
Real implementations use fixed-point arithmetic with renormalisation rather than
Fraction, emitting bits as the interval narrows so memory stays constant. Range coding is the byte-oriented variant used in modern compressors, and asymmetric numeral systems (ANS) achieves the same compression with far simpler arithmetic, which is why it now underlies zstd and LZFSE. - Program
Solve a weighing puzzle using the information bound to determine the minimum number of weighings.
Solution
import math def min_weighings(n_coins, know_direction=True): """n coins, one counterfeit; a balance has 3 outcomes per weighing""" outcomes = 2*n_coins if know_direction else n_coins return math.ceil(math.log(outcomes, 3)) print(min_weighings(12)) # 3: log₃24 = 2.89 → 3 print(min_weighings(13)) # 3: log₃26 = 2.97 → 3 (the true maximum) print(min_weighings(14)) # 4: log₃28 = 3.03 → 4Each weighing has three outcomes (left, right, balanced), so k weighings distinguish at most 3^k cases. With 12 coins and the direction unknown there are 24 answers, and 3³=27\ge24 — so 3 suffices, and the bound also proves 2 cannot.
13 coins is the maximum for 3 weighings when the direction must be determined (26 answers). With a known-good reference coin available, 14 is achievable.
The construction is a non-adaptive assignment of each coin to a distinct sequence in {L,R,off}^3, avoiding sequences and their reverses (Chapter 13 problem 13) — and the bound tells you exactly how much slack you have.
- Program
Measure the compression ratio of several algorithms and compare against the order-k entropy.
Solution
import gzip, bz2, lzma, math def compare(data): n = len(data) print("raw ", n) for name, mod in (("gzip", gzip), ("bz2", bz2), ("lzma", lzma)): print(name, len(mod.compress(data))) for k in range(4): h = conditional_entropy(data, k) print(f"order-{k} bound", math.ceil(n * h / 8), "bytes")On English text, gzip typically lands near the order-2 bound, bz2 near order-3, and lzma below it — because LZ-family algorithms exploit long-range repetition, which a fixed-order Markov model does not capture, so they can beat the order-k bound without violating anything.
On random data all three produce output slightly larger than the input, which is Chapter 6's counting argument made visible: no compressor shrinks every input, and the overhead is the format's header.
That experiment is worth running once — it makes concrete both that entropy is a real floor for a given model and that better models, not better coders, are where compression gains come from.
- Program
Implement mutual-information-based feature ranking and demonstrate the high-cardinality bias.
Solution
from collections import Counter, defaultdict import math def mutual_information(xs, ys): n = len(xs) px, py = Counter(xs), Counter(ys) pxy = Counter(zip(xs, ys)) return sum(c/n * math.log2((c/n) / ((px[x]/n)*(py[y]/n))) for (x, y), c in pxy.items()) def gain_ratio(xs, ys): hx = entropy(Counter(xs)) return mutual_information(xs, ys) / hx if hx else 0.0 # demonstrate the bias labels = [random.randint(0,1) for _ in range(1000)] useful = [l if random.random() < 0.8 else 1-l for l in labels] # 80% predictive ident = list(range(1000)) # useless ID print("useful MI", mutual_information(useful, labels)) print("ID MI", mutual_information(ident, labels)) # MAXIMAL, and useless print("useful GR", gain_ratio(useful, labels)) print("ID GR", gain_ratio(ident, labels)) # correctly tinyThe ID column achieves maximal mutual information — it determines the label perfectly on the training data — while carrying no generalisable signal at all. Raw information gain ranks it first.
The gain ratio divides by H(feature), which is log_2 1000 ≈ 10 for the ID and 1 for a binary feature, correctly demoting it.
This is why C4.5 uses gain ratio and why any feature-selection pipeline must either penalise cardinality or evaluate on held-out data. It is the information-theoretic face of overfitting.
- Program
Estimate the entropy of a stream in bounded memory using a sketch.
Solution
# Exact entropy needs the full frequency table — O(distinct) memory. # For a stream with too many distinct values, estimate instead: # # 1. Count–Min sketch (Chapter 38) gives approximate frequencies, # but its one-sided error biases the entropy DOWNWARD. # 2. The Chakrabarti–Cormode–McGregor sketch estimates H to (1±ε) # in O(ε⁻² log n) space, using the identity # H = E[log(n / r)] where r is the "run length" of a sampled position def ccm_entropy_estimate(stream, samples=1000): import random n = len(stream) est = 0.0 for _ in range(samples): i = random.randrange(n) s = stream[i] r = sum(1 for j in range(i, n) if stream[j] == s) # count after i est += math.log2(n / r) return est / samplesThe estimator works because sampling a position and counting subsequent occurrences of that symbol gives an unbiased handle on -log p — the "surprisal" whose expectation is the entropy.
The streaming version maintains the counts incrementally rather than rescanning, keeping memory at O(samples) regardless of the number of distinct symbols.
Compare against exact entropy on a stream small enough to hold: the estimate should be within a few percent with 1,000 samples, and the error should shrink as 1/√{samples} — the Monte Carlo rate of Chapter 36, appearing again.
Cryptographic mathematics
Public-key cryptography rests on a handful of problems believed hard - factoring, discrete log, lattice problems - and on one asymmetry: verifying is easy, finding is not. Every chapter of this part reappears here.
RSA
KEYGEN p, q large primes Miller–Rabin (Ch. 23)
n = pq, φ(n) = (p−1)(q−1) Euler's totient (Ch. 20)
e = 65537, d = e⁻¹ mod φ(n) extended Euclid (Ch. 15)
ENCRYPT c = m^e mod n binary exponentiation (Ch. 21)
DECRYPT m = c^d mod n correct by Euler + CRT (Ch. 20, 22)
SPEED-UP decrypt mod p and mod q separately, then CRT — 4× faster (Ch. 22)
SECURITY recovering d from (n, e) is equivalent to factoring n.
"Textbook RSA" as written above is INSECURE — it is deterministic,
malleable (c₁c₂ decrypts to m₁m₂), and leaks for small m.
Real RSA uses OAEP padding; signatures use PSS.Every scheme here is correct by elementary number theory and secure only by conjecture. The correctness proofs in this chapter are complete; the security arguments are reductions — "breaking this is at least as hard as factoring" — and factoring's hardness has never been proved. Knowing which half of a cryptographic claim is a theorem is the point.
Diffie–Hellman and elliptic curves
Hash functions
| Property | Requirement | Attack cost |
|---|---|---|
| Preimage resistance | given h, find m | 2^n |
| Second preimage | given m_1, find m_2 | 2^n |
| Collision resistance | find any m_1≠ m_2 | 2^{n/2} (birthday, Ch. 6) |
The birthday bound is why collision resistance needs twice the output length: SHA-256 gives 128-bit collision resistance, and MD5's 128 bits gave only 64 — which is why practical MD5 collisions were found in 2004 and chosen-prefix collisions in 2019.
"Why is RSA slower than AES?" — because it is modular exponentiation on 3072-bit numbers (thousands of multiplications) versus a few dozen table lookups and XORs. That is why TLS uses public-key crypto only to establish a symmetric key, then switches. Naming that hybrid structure is the answer.
- Theory
Prove RSA decryption is correct, including when gcd(m,n) > 1.
Solution
With n = pq and ed ≡ 1 (mod φ(n)), write ed = 1 + k(p-1)(q-1).
Modulo p: if p | m then both sides are ≡ 0. Otherwise Fermat gives m^{p-1}≡1, so
m^{ed} = m·(m^{p-1})^{k(q-1)} ≡ m ±od p.
Modulo q: symmetric.
Agreeing modulo both primes, CRT gives m^{ed}≡ m ±od n for every m. ∎
The non-coprime case matters: it means RSA is correct for all messages, not just those coprime to n. (Encountering an m sharing a factor with n would itself reveal the factorisation, so it never happens for random messages — but the correctness proof should not depend on that.)
- Theory
Explain why textbook RSA is insecure and what padding fixes.
Solution
- Deterministic: the same plaintext always gives the same ciphertext, so an attacker can build a dictionary. For a small message space (a yes/no vote, a credit-card number) this breaks it entirely.
- Malleable: c_1c_2 = (m_1m_2)^e, so ciphertexts can be combined. This enables a chosen-ciphertext attack: to decrypt c, submit c· r^e for random r, obtain mr, and divide.
- Small message / small exponent: if m^e < n there is no modular reduction at all and m is recovered by an integer e-th root. With e=3 and a 128-bit message under a 2048-bit modulus, this is immediate.
- Håstad's broadcast attack: the same m sent to e recipients with exponent e is recovered by CRT plus an integer root.
OAEP fixes all of these: it pads m with randomness and hashes through a Feistel-like construction, making the encryption randomised and non-malleable, with a security proof in the random-oracle model.
The lesson generalises: a scheme's mathematical correctness says nothing about its security as a protocol.
- Theory
Describe Diffie–Hellman and the man-in-the-middle attack.
Solution
Alice picks a, sends g^a; Bob picks b, sends g^b; both compute g^{ab}. An eavesdropper sees g, g^a, g^b and must solve the computational Diffie–Hellman problem, believed as hard as discrete log.
The attack: Mallory intercepts both messages, sends her own g^m to each side, and establishes two separate keys — g^{am} with Alice and g^{bm} with Bob. She decrypts, reads, re-encrypts and forwards. Neither party can detect it.
The fix is authentication, not more mathematics: TLS signs the DH parameters with a certificate chain, SSH pins the host key on first use, and Signal uses long-term identity keys. ∎
Two further requirements from Chapter 24: p-1 must have a large prime factor (else Pohlig–Hellman), and received elements must be validated to lie in the correct subgroup (else small-subgroup confinement). Both have caused real vulnerabilities.
- Theory
Explain why elliptic curves allow much shorter keys than finite-field discrete log.
Solution
For discrete log in ℤ_p^×, the index calculus method runs in subexponential time L_p[1/3, 1.92] — it exploits the fact that integers factor into small primes, building relations over a factor base.
Elliptic curve groups have no such structure: there is no notion of a "smooth" point, so no factor base exists and index calculus does not apply. The best known attack is generic — Pollard's rho at O(√n).
Consequently a 256-bit curve gives 2^{128} security, matching AES-128, while finite-field DH needs 3072 bits for the same. ∎
The caveats: certain curves are weak — supersingular curves fall to the MOV attack (mapping to a finite field via the Weil pairing), and anomalous curves (where #E = p) fall to Smart's attack in linear time. This is why standardised curves are used rather than freshly generated ones, and why curve parameters are scrutinised for provenance.
- Theory
Explain the birthday bound's role in hash function security.
Solution
For an n-bit hash, finding a preimage requires 2^n work (try inputs until one matches). Finding a collision requires only 2^{n/2}, by the birthday bound of Chapter 6: after 1.18·2^{n/2} random hashes, a collision is more likely than not.
So collision resistance is only half the output length in bits. ∎
MD5 128 bits → 64-bit collision resistance → broken in 2004 SHA-1 160 bits → 80-bit → broken in 2017 (SHAttered) SHA-256 256 bits → 128-bit → secure SHA-512 512 bits → 256-bit → post-quantum marginGrover's quantum algorithm gives a quadratic speed-up on preimages (2^{n/2}) but only a marginal one on collisions, so doubling the hash length restores classical-equivalent security against quantum attacks — which is why SHA-384 and SHA-512 are specified for high-assurance use.
Where collisions matter: signatures (sign a benign document, present a malicious one with the same hash), certificates, and content-addressed storage. Where they do not: HMAC, which remains secure even with a collision-broken hash.
- Theory
Describe Shor's algorithm at a high level and what it breaks.
Solution
To factor n, pick a random a coprime to n and find the multiplicative order r of a modulo n. If r is even and a^{r/2}≢-1, then x = a^{r/2} is a non-trivial square root of 1 and gcd(x-1,n) is a factor (Chapter 22 problem 4). A random a works with probability \ge1/2.
The quantum part is order-finding: prepare a superposition, apply modular exponentiation, and use the quantum Fourier transform to extract the period in polynomial time — where classically period-finding is as hard as factoring.
What it breaks: RSA (factoring), Diffie–Hellman and DSA (discrete log), and ECDH/ECDSA (elliptic-curve discrete log — the same period-finding applies). ∎
What survives: symmetric ciphers and hashes, weakened only quadratically by Grover — so AES-256 and SHA-512 remain adequate. And the post-quantum candidates: lattices (Kyber, Dilithium), hash-based signatures (SPHINCS+), and codes (McEliece).
The practical concern is harvest now, decrypt later: traffic recorded today can be decrypted when a quantum computer arrives, which is why migration is under way before the threat is real.
- Theory
Explain why constant-time implementation matters and give an example attack.
Solution
If the execution time, memory access pattern or power consumption depends on secret data, an attacker who can measure it learns the secret — regardless of the mathematics.
- Square-and-multiply (Chapter 21): the
if (e & 1)branch makes the time depend on the exponent's Hamming weight, and per-iteration measurements can recover individual bits. Practical remote timing attacks on OpenSSL's RSA were demonstrated by Brumley and Boneh in 2003. - Table lookups: AES implementations using S-box tables leak the accessed indices through cache timing, recoverable by an unprivileged process on the same machine.
- Comparison:
memcmpon a MAC returns early on the first differing byte, letting an attacker forge a MAC one byte at a time in 256×16 attempts instead of 2^{128}.
Defences: the Montgomery ladder (Chapter 21) for exponentiation; bitsliced or AES-NI implementations avoiding tables; constant-time comparison that XORs all bytes and checks the accumulated result once; and blinding, which randomises the observable behaviour.
This is why cryptographic code should never be written from a mathematical description alone.
- Square-and-multiply (Chapter 21): the
- Program
Implement RSA end to end with proper key generation, and CRT-accelerated decryption.
Solution
import random def gen_prime(bits): while True: c = random.getrandbits(bits) | (1 << (bits-1)) | 1 if any(c % p == 0 for p in SMALL_PRIMES): continue if miller_rabin(c, 40): return c def keygen(bits=1024): p, q = gen_prime(bits), gen_prime(bits) while p == q: q = gen_prime(bits) n, phi = p*q, (p-1)*(q-1) e = 65537 assert math.gcd(e, phi) == 1 d = pow(e, -1, phi) return (n, e), (n, d, p, q, d % (p-1), d % (q-1), pow(q, -1, p)) def decrypt_crt(c, priv): n, d, p, q, dp, dq, qinv = priv m1, m2 = pow(c % p, dp, p), pow(c % q, dq, q) h = (qinv * (m1 - m2)) % p m = (m2 + h*q) % n assert pow(m, PUB_E, n) == c % n # verify — guards the Bellcore attack return mCRT decryption is about 4× faster: two exponentiations with half-size moduli and half-size exponents (Chapter 20 problem 11).
The verification assertion is not paranoia — it defends against the Bellcore fault attack, where a hardware glitch corrupting one of m_1, m_2 lets gcd(m^e-c, n) reveal a prime factor from a single faulty signature. Re-encrypting before releasing the result costs one cheap exponentiation (e is small) and eliminates it.
Setting the top bit in
gen_primeguarantees the modulus has the full bit length; setting the bottom bit skips evens. - Program
Implement Diffie–Hellman with parameter validation, and demonstrate a small-subgroup attack against an implementation that omits it.
Solution
def dh_keypair(p, g, q): a = random.randrange(2, q) return a, pow(g, a, p) def dh_shared(their_public, my_secret, p, q): # VALIDATE: reject 1, p−1, and anything outside the order-q subgroup if not (1 < their_public < p-1): raise ValueError("out of range") if pow(their_public, q, p) != 1: raise ValueError("not in the prime-order subgroup") return pow(their_public, my_secret, p) # ATTACK on an implementation that skips validation: # send y = p−1 (order 2). The shared secret is 1 if a is even, # p−1 if a is odd — one bit of the private key, per exchange.With a safe prime p = 2q+1 the only small subgroup has order 2, so the leak is one bit. With a non-safe prime whose p-1 has many small factors, repeating the attack with elements of order 3, 5, 7, … recovers the key modulo each, and CRT reconstructs it entirely.
Several TLS and IKE implementations shipped without this check and had practical key-recovery vulnerabilities. The fix is two lines.
RFC 7919 defines named DH groups precisely so implementations stop generating their own parameters — the parameters themselves are a place where subtle weaknesses hide.
- Program
Implement elliptic curve point addition and scalar multiplication over 𝔽_p.
Solution
def ec_add(P, Q, a, p): if P is None: return Q if Q is None: return P x1, y1 = P; x2, y2 = Q if x1 == x2 and (y1 + y2) % p == 0: return None # P + (−P) = O if P == Q: lam = (3*x1*x1 + a) * pow(2*y1, -1, p) % p # tangent else: lam = (y2 - y1) * pow(x2 - x1, -1, p) % p # chord x3 = (lam*lam - x1 - x2) % p y3 = (lam*(x1 - x3) - y1) % p return (x3, y3) def ec_mul(k, P, a, p): """double-and-add — NOT constant time; use a ladder for real keys""" R = None while k: if k & 1: R = ec_add(R, P, a, p) P = ec_add(P, P, a, p) k >>= 1 return RThe group law is the chord-and-tangent construction: three collinear points on the curve sum to the identity, which is the point at infinity (represented here as
None).Each addition needs a modular inverse — expensive. Real implementations use projective coordinates to defer inversions, doing one at the end instead of one per operation, which is a 10–20× speed-up.
ec_mulas written leaks the scalar through timing. Use the Montgomery ladder (Chapter 21), and verify that y² ≡ x³+ax+b for every received point — an invalid-curve attack feeds a point from a weaker curve and recovers the key from the resulting arithmetic. - Program
Implement Shamir secret sharing and verify its information-theoretic security empirically.
Solution
import random P = (1 << 127) - 1 def split(secret, n, k): coeffs = [secret] + [random.randrange(P) for _ in range(k-1)] ev = lambda x: sum(c * pow(x, i, P) for i, c in enumerate(coeffs)) % P return [(i, ev(i)) for i in range(1, n+1)] def reconstruct(shares): s = 0 for i, (xi, yi) in enumerate(shares): num = den = 1 for j, (xj, _) in enumerate(shares): if i != j: num = num * (-xj) % P den = den * (xi - xj) % P s = (s + yi * num % P * pow(den, -1, P)) % P return s # security: with k−1 shares, EVERY secret remains equally likely fixed = split(42, 5, 3)[:2] implied = set() for guess in range(1000): implied.add(reconstruct(fixed + [(99, ...)])) # any 3rd share is consistentThe security demonstration: hold k-1 shares fixed and vary the missing one over the field. Every candidate secret arises exactly once, so the shares carry zero information — H(secret| k-1 shares) = H(secret) in the language of Chapter 68.
That is information-theoretic security, unlike RSA's computational security. No amount of computation helps, and it does not depend on any unproven hardness assumption.
The price: shares are as large as the secret, and there is no integrity — a corrupted share silently produces a wrong reconstruction. Verifiable secret sharing (Feldman, Pedersen) adds commitments to detect that.
- Program
Implement a constant-time comparison and demonstrate a timing attack against the naive version.
Solution
import hmac, time def naive_equal(a, b): if len(a) != len(b): return False for x, y in zip(a, b): if x != y: return False # EARLY RETURN — leaks the position return True def constant_time_equal(a, b): if len(a) != len(b): return False r = 0 for x, y in zip(a, b): r |= x ^ y # no branch, always full length return r == 0 # or simply: hmac.compare_digest(a, b) # the attack: guess byte by byte, timing each candidate def forge(target_len, oracle): guess = bytearray(target_len) for pos in range(target_len): best, best_t = 0, -1 for b in range(256): guess[pos] = b t = min(time_call(oracle, bytes(guess)) for _ in range(200)) if t > best_t: best_t, best = t, b guess[pos] = best return bytes(guess)The naive comparison returns on the first mismatch, so a correct prefix takes measurably longer. The attack recovers a 16-byte MAC in 16×256 = 4096 guesses instead of 2^{128}.
The timing difference is nanoseconds and buried in noise, which is why the attack takes the minimum over many trials — noise only ever adds time, so the minimum is the cleanest signal. Remote versions using statistical methods have been demonstrated across a network.
Use the library function:
hmac.compare_digestin Python,CRYPTO_memcmpin OpenSSL. A hand-written constant-time comparison can still be defeated by compiler optimisation turning it back into a branch. - Program
Estimate the cost of factoring an RSA modulus of various sizes and derive the recommended key length.
Solution
import math def gnfs_cost_bits(n_bits): """log2 of the general number field sieve's operation count L_n[1/3, (64/9)^{1/3}] with c ≈ 1.923""" lnn = n_bits * math.log(2) lnlnn = math.log(lnn) return 1.923 * (lnn ** (1/3)) * (lnlnn ** (2/3)) / math.log(2) for bits in (512, 768, 1024, 2048, 3072, 4096, 15360): print(bits, round(gnfs_cost_bits(bits), 1), "bits of security")RSA bits security status 512 ~50 factored in 1999 768 ~62 factored in 2009 (RSA-768) 1024 ~72 never publicly factored; assumed within state reach 2048 ~93 current minimum 3072 ~112 ≈ AES-128 equivalent — NIST recommendation 15360 ~256 ≈ AES-256 equivalent — impracticalThe subexponential growth is the point: doubling the key size does not double the security. Going from 2048 to 3072 bits adds only about 19 bits of security, and reaching AES-256 equivalence would need a 15,360-bit modulus.
That poor scaling is precisely why elliptic curves won: EC security is n/2 bits for an n-bit curve — linear — so 256 bits gives 128-bit security and 512 gives 256.
Cross-check the model against the RSA-768 and RSA-250 factorisation records: the formula should place them near the effort actually expended, which it does to within a small constant.
- Program
Build a small end-to-end hybrid encryption scheme and identify every assumption it rests on.
Solution
import os, hashlib, hmac def hybrid_encrypt(message, recipient_pub): # 1. random symmetric key k = os.urandom(32) # CSPRNG, not random() # 2. wrap it with the public key (use OAEP in production) wrapped = rsa_oaep_encrypt(k, recipient_pub) # 3. encrypt the payload symmetrically, WITH authentication nonce = os.urandom(12) ct = aes_gcm_encrypt(k, nonce, message) # AEAD: encrypt + MAC return wrapped, nonce, ct def hybrid_decrypt(wrapped, nonce, ct, recipient_priv): k = rsa_oaep_decrypt(wrapped, recipient_priv) return aes_gcm_decrypt(k, nonce, ct) # raises on a bad tagThe assumptions, each of which can fail independently:
- Factoring is hard — unproven, and false on a quantum computer.
- AES is a good pseudorandom permutation — unproven, but heavily analysed.
- The CSPRNG is unpredictable — the most common real failure. Debian's 2008 OpenSSL bug reduced the key space to 32,768; Android's 2013
SecureRandomflaw let Bitcoin keys be recovered. - The implementation is constant-time — problem 12.
- The recipient's public key is authentic — problem 3; without a PKI or pinning, the whole thing is man-in-the-middleable.
- The nonce never repeats — GCM nonce reuse is catastrophic, revealing the authentication key.
Encrypt-then-MAC, always, and prefer an AEAD mode that does it for you. The mathematics in this book is the easy part; every one of the failures above is an engineering failure, and each has caused real breaches.