Tilting at Rare Events
15 min read
How do you measure the probability of something that almost never happens? I keep running into this question with language models: the chance that a model solves a problem at the edge of its ability, or produces a genuinely harmful response, can be \(10^{-4}\) or \(10^{-8}\), and we increasingly want that number with error bars [WH, ’24], [JT+, ’25], [AS+, ’26]. Sampling and counting cannot see it: at a budget of a thousand draws, an event of probability one in a million almost always shows up zero times, and the answer “zero” is wrong without even saying how wrong. The classical fix is to sample from a different distribution, one that produces the event often, and then correct for the cheating. Which distribution to use is the whole design problem. The answer is the exponential tilt, and if you have ever fine-tuned a model against a reward with a KL penalty, you have already built one.
A Coin We Can Check by Hand
An outcome \(x\) is drawn from a distribution \(p\), a score \(s(x)\) measures how extreme the outcome is, and the event of interest is the score clearing a threshold \(\tau\). The quantity we want is the tail probability:
\[ \rho = \Pr_{x \sim p}\big[s(x) \ge \tau\big]. \]
Our running example strips this down to something we can check by hand: \(x\) is a string of 20 fair coin flips, \(s(x)\) counts the heads, and the event is at least 17 heads, so \(\tau = 17\). The answer is exactly computable:
\[ \rho = \frac{\binom{20}{17} + \binom{20}{18} + \binom{20}{19} + \binom{20}{20}}{2^{20}} = \frac{1351}{1048576} \approx 1.3 \times 10^{-3}, \]
about one in 776, which is rare enough to strain plain sampling and small enough that every claim below can be verified with arithmetic.
The naive estimator draws \(m\) strings from \(p\) and reports the fraction that clear the threshold:
\[ \hat{\rho}^{\mathrm{naive}} = \frac{1}{m} \sum_{i=1}^{m} \mathbf{1}\big[s(x_i) \ge \tau\big]. \]
Each indicator is a coin flip that comes up heads with probability \(\rho\), so the average of \(m\) of them has variance \(\rho(1-\rho)/m\), and its standard deviation relative to the quantity being estimated is:
\[ \frac{\sqrt{\rho(1 - \rho)/m}}{\rho} = \sqrt{\frac{1 - \rho}{m \rho}} \approx \frac{1}{\sqrt{m \rho}}, \]
dividing through by \(\rho\) and then dropping the \(1 - \rho\), which is nearly one for a rare event. In words: the error is set by the number of hits \(m\rho\) and not by the number of draws, so ten percent relative error needs about \(100/\rho\) draws. On the coin that is 78,000 draws, and at a real rarity of \(10^{-6}\) it is a hundred million.
Sampling From the Wrong Coin on Purpose
Importance sampling replaces \(p\) with any proposal \(q\) we can sample, and repairs the damage with a weight. Draw \(x_1, \dots, x_m\) from \(q\) and report:
\[ \hat{\rho} = \frac{1}{m} \sum_{i=1}^{m} \mathbf{1}\big[s(x_i) \ge \tau\big]\, w(x_i), \qquad w(x) = \frac{p(x)}{q(x)}. \]
The weight says how many draws from \(p\) each draw from \(q\) stands in for: a sample the proposal over-produces counts for less than one, and a sample it under-produces counts for more. Unbiasedness is a change of measure in four moves:
\[ \begin{aligned} \mathbb{E}_{x \sim q}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\, w(x)\Big] &= \sum_{x} q(x)\, \mathbf{1}\big[s(x) \ge \tau\big]\, \frac{p(x)}{q(x)} \\ &= \sum_{x :\, s(x) \ge \tau} q(x)\, \frac{p(x)}{q(x)} \\ &= \sum_{x :\, s(x) \ge \tau} p(x) \\ &= \rho, \end{aligned} \]
where the first line writes the expectation as a sum and substitutes the weight, the second lets the indicator throw away every outcome below the threshold, the third cancels \(q(x)\), and the fourth is the definition of \(\rho\). The cancellation needs \(q(x) > 0\) only where the sum still has terms, which is on the event. Off the event, \(q\) is free to put no mass at all. Do you see why that freedom is exactly what we want for a rare event?
So any proposal that covers the event gives an unbiased estimate, and the entire design question is variance.
What the Variance Measures
For an average of independent draws, the variance is the second moment minus the squared mean, divided by \(m\). The proposal enters through the second moment:
\[ \begin{aligned} \mathbb{E}_{x \sim q}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\, w(x)^2\Big] &= \sum_{x} q(x)\, \mathbf{1}\big[s(x) \ge \tau\big]\, \frac{p(x)^2}{q(x)^2} \\ &= \sum_{x :\, s(x) \ge \tau} q(x)\, \frac{p(x)^2}{q(x)^2} \\ &= \sum_{x :\, s(x) \ge \tau} \frac{p(x)^2}{q(x)}, \end{aligned} \]
expanding the expectation, applying the indicator to restrict the sum, and cancelling one factor of \(q(x)\).
To read that sum, name the distribution it is secretly about. The conditional distribution given the event restricts \(p\) to the event and renormalizes:
\[ q^{\star}(x) = \frac{p(x)\, \mathbf{1}\big[s(x) \ge \tau\big]}{\rho}. \]
On the event we can therefore write \(p(x) = \rho\, q^{\star}(x)\), and substituting that into the second moment pulls out a factor of \(\rho^2\):
\[ \sum_{x :\, s(x) \ge \tau} \frac{p(x)^2}{q(x)} = \sum_{x :\, s(x) \ge \tau} \frac{\rho^2\, q^{\star}(x)^2}{q(x)} = \rho^2 \sum_{x} \frac{q^{\star}(x)^2}{q(x)}, \]
where the second line extends the sum to all of \(x\), which changes nothing because \(q^{\star}\) is zero off the event. Subtracting the squared mean and dividing by \(m\) gives the master formula:
\[ \begin{aligned} \mathrm{Var}(\hat{\rho}) &= \frac{1}{m}\Big( \rho^2 \sum_{x} \frac{q^{\star}(x)^2}{q(x)} - \rho^2 \Big) \\ &= \frac{\rho^2}{m} \Big( \sum_{x} \frac{q^{\star}(x)^2}{q(x)} - 1 \Big) \\ &= \frac{\rho^2}{m}\, \chi^2\big(q^{\star} \,\Vert\, q\big), \end{aligned} \]
where the second line factors out \(\rho^2\) and the third names the parenthesized quantity, which is the chi-square divergence between \(q^{\star}\) and \(q\).
The proposal is graded against \(q^{\star}\), the distribution of outcomes given that the event happened, and not against \(p\). A proposal can stay very close to the base distribution and still be terrible. What it has to resemble is the event as seen from the inside. And the grading is harsh, because the divergence squares the ratio \(q^{\star}/q\): a region where the proposal supplies half the mass it should contributes twice what it would if the mass were right.
The Perfect Proposal Is the Answer
The divergence is zero when \(q = q^{\star}\), and it is worth watching what the estimator does there. Every draw lands in the event, and every weight collapses to the same constant:
\[ w(x) = \frac{p(x)}{q^{\star}(x)} = \frac{p(x)\, \rho}{p(x)\, \mathbf{1}[s(x) \ge \tau]} = \frac{p(x)\, \rho}{p(x)} = \rho, \]
where the first line substitutes the definition of \(q^{\star}\) and the second uses that the indicator is 1, since every draw from \(q^{\star}\) is in the event. Each term of the average is exactly \(\rho\), so a single draw returns the answer with no randomness left in it.
The catch is the definition of \(q^{\star}\) itself. Its normalizer is \(\rho\), so sampling it, or even evaluating one of its densities, requires the number we set out to estimate. On the coin the trouble is visible without any formulas: given at least 17 heads in 20 flips, the flips are no longer independent, since an early tails forces the remaining flips to carry more heads. So the conditional law is not a coin at all, and simulating it exactly amounts to already knowing the tail.
The job, then, is to get near \(q^{\star}\) without knowing \(\rho\). Notice what kind of object \(q^{\star}\) already is. Multiplying by an indicator is multiplying by an exponential:
\[ q^{\star}(x) \propto p(x)\, e^{h(s(x))}, \qquad h(s) = \begin{cases} 0 & s \ge \tau \\ -\infty & s < \tau. \end{cases} \]
Conditioning is a tilt by a cliff. The cliff is what puts the normalizer out of reach, so the natural relaxation keeps the tilt and flattens the cliff into a slope.
Relax the Cliff to a Slope
Tilting \(p\) by a linear function of the score, \(h(s) = \lambda s\), gives the exponential tilt at strength \(\lambda \ge 0\):
\[ q_{\lambda}(x) = \frac{p(x)\, e^{\lambda s(x)}}{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]}. \]
The normalizer is the moment generating function of the score, finite whenever the score’s tails are light, and, unlike \(\rho\), something we can reason about.
On the coin the family is easy to sample. The score adds over flips, so the tilt factorizes:
\[ e^{\lambda s(x)} = \prod_{t=1}^{20} e^{\lambda\, \mathbf{1}[\text{flip } t \text{ is heads}]}, \]
and renormalizing each factor turns every fair flip into an independent biased flip with heads probability \(e^{\lambda} / (e^{\lambda} + 1)\). Tilting a fair coin is just biasing it. (The same factorization is why tilting an autoregressive model keeps it autoregressive: the tilt distributes over the chain rule.) One structural fact comes free. Because the tilt depends on \(x\) only through \(s(x)\), two outcomes with the same score keep their relative probabilities, so the tilt spends all of its distortion across score levels and none on structure the event cannot see.
Of all the ways to flatten the cliff, why a straight slope? The reason is a variational one. Measure distortion by the Kullback–Leibler divergence:
\[ \mathrm{KL}(q \,\Vert\, p) = \mathbb{E}_{x \sim q}\Big[\log \frac{q(x)}{p(x)}\Big], \]
the average log disagreement of \(q\) with \(p\), in nats, under \(q\)’s own draws.
Claim (Gibbs variational principle): for every \(\lambda > 0\), over all distributions \(q\), the objective \(\lambda\, \mathbb{E}_{x \sim q}[s(x)] - \mathrm{KL}(q \,\Vert\, p)\) is uniquely maximized by the tilt \(q_{\lambda}\), with maximum value \(\log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\).
Proof
Both terms are expectations of logarithms under \(q\), so they merge:
\[ \begin{aligned} \lambda\, \mathbb{E}_{x \sim q}[s(x)] - \mathrm{KL}(q \,\Vert\, p) &= \mathbb{E}_{x \sim q}\Big[\log e^{\lambda s(x)}\Big] - \mathbb{E}_{x \sim q}\Big[\log \frac{q(x)}{p(x)}\Big] \\ &= \mathbb{E}_{x \sim q}\Big[\log \frac{p(x)\, e^{\lambda s(x)}}{q(x)}\Big] \\ &= \mathbb{E}_{x \sim q}\Big[\log \frac{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, q_{\lambda}(x)}{q(x)}\Big] \\ &= \log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big] + \mathbb{E}_{x \sim q}\Big[\log \frac{q_{\lambda}(x)}{q(x)}\Big] \\ &= \log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big] - \mathrm{KL}(q \,\Vert\, q_{\lambda}), \end{aligned} \]
where the first line writes the score term as a logarithm, the second combines the two expectations into one log, the third substitutes \(p(x)\, e^{\lambda s(x)} = \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, q_{\lambda}(x)\) from the definition of the tilt, the fourth splits off the constant normalizer, and the fifth recognizes what is left as a divergence with the sign flipped. The identity holds for every \(q\), and \(\mathrm{KL}(q \,\Vert\, q_{\lambda}) \ge 0\) with equality exactly at \(q = q_{\lambda}\) (Jensen’s inequality on the strictly concave logarithm), which finishes the proof.
In words: among all distributions that stray a given number of nats from \(p\), the exponential tilt reaches the highest average score. It gets the most score per nat of distortion. The variational formula is Donsker & Varadhan’s [DV, CPAM ’75], and the equivalent projection view, the tilt as the least-distorted distribution with a prescribed mean score, is Csiszár’s [Csiszár, Ann. Probab. ’75].
How much does relaxing the cliff give up? On the coin we can compare the relaxation to the real thing exactly. Under the true conditional \(q^{\star}\), each individual flip comes up heads with probability \(0.8586\), and the tilt we are about to select uses a coin of bias \(0.85\). That near-agreement is general rather than a numerical accident: for scores that add across many weakly dependent pieces, the conditional law converges, piece by piece, to a mean-matched tilt (the Gibbs conditioning principle). For such events the tilt is the shape that conditioning itself takes at scale.
Aim the Average at the Threshold
There is one member of the family per \(\lambda\), so which one? Under the tilt the weight has a closed form:
\[ w(x) = \frac{p(x)}{q_{\lambda}(x)} = \frac{p(x)\, \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]}{p(x)\, e^{\lambda s(x)}} = \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda s(x)}, \]
substituting the tilt and cancelling \(p(x)\). The weight is decreasing in the score: a sample that lands deeper into the event carries a smaller weight, so overshooting the threshold is the safe direction. On the event, where \(s(x) \ge \tau\), every weight therefore obeys the cap \(w(x) \le \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}\). The second moment inherits the cap in three moves:
\[ \begin{aligned} \mathbb{E}_{x \sim q_{\lambda}}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\, w(x)^2\Big] &= \mathbb{E}_{x \sim p}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\, w(x)\Big] \\ &= \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, \mathbb{E}_{x \sim p}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\, e^{-\lambda s(x)}\Big] \\ &\le \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}\, \mathbb{E}_{x \sim p}\Big[\mathbf{1}\big[s(x) \ge \tau\big]\Big] \\ &= \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}\, \rho, \end{aligned} \]
where the first line spends one factor of \(w\) to undo the change of measure, the second substitutes the weight’s closed form, the third bounds \(e^{-\lambda s}\) by its value at the threshold, and the fourth recognizes the remaining expectation as \(\rho\).
To make the guarantee as strong as possible, minimize \(\log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big] - \lambda \tau\) over \(\lambda\). The derivative of the first term is the tilted mean:
\[ \begin{aligned} \frac{d}{d\lambda} \log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big] &= \frac{\frac{d}{d\lambda}\, \mathbb{E}_{x \sim p}\big[e^{\lambda s(x)}\big]}{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]} \\ &= \frac{\mathbb{E}_{x \sim p}\big[s(x)\, e^{\lambda s(x)}\big]}{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]} \\ &= \mathbb{E}_{x \sim p}\Big[s(x)\, \frac{e^{\lambda s(x)}}{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]}\Big] \\ &= \mathbb{E}_{x \sim q_{\lambda}}\big[s(x)\big], \end{aligned} \]
where the first line is the chain rule on the logarithm, the second differentiates under the expectation, the third moves the normalizer inside, and the fourth recognizes \(p(x)\, e^{\lambda s(x)} / \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\) as the tilted density. Setting the derivative of \(\log \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big] - \lambda \tau\) to zero, the minimizing strength solves:
\[ \mathbb{E}_{x \sim q_{\lambda}}\big[s(x)\big] = \tau. \]
Tilt until the threshold is the typical outcome. (The tilted mean rises monotonically in \(\lambda\), so the solution is unique; can you see why its derivative in \(\lambda\) is a variance?) This mean-matching rule is as old as the subject, running from Siegmund’s sequential tests [Siegmund, Ann. Stat. ’76] through the rare-event simulation literature it organized [Bucklew, ’04]; Owen’s chapter 9 is the modern treatment of proposal design [Owen, ’13].
On the coin the rule is one line of arithmetic. The tilted mean is 20 times the coin’s bias, so mean matching sets the bias to \(17/20 = 0.85\), which is \(\lambda = \ln(17/3) \approx 1.73\). In the first plot, the fair coin’s head-count distribution puts the threshold far into its tail, and the mean-matched coin plants it at the center of mass: the event has been made typical.
The payoff on the coin, computed exactly rather than simulated: the per-draw relative variance falls from 775 under the fair coin to 2.02 under the \(0.85\) coin, so ten percent relative error costs 203 draws instead of 78,000. The same accuracy, 383 times cheaper, from changing one number.
In the second plot, the cost curve over the coin’s bias tells the design story in one look. The basin around the minimum is broad and forgiving: biases \(0.80\) and \(0.90\) cost 262 and 282 draws against the optimum’s 203, so the rule does not need to be executed precisely. The walls on either side are the two ways to fail. Falling back toward one half recovers the naive 78,000, and pushing the bias toward one explodes just as badly, because a coin that almost never flips tails starves the threshold region where most of the event’s mass actually lives. (At bias exactly one it violates the support condition outright, putting zero mass on 17, 18, or 19 heads.) Aim for the threshold, not past it.
The Chernoff Bound Reappears
The cap \(\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}\) should look familiar. Applying Markov’s inequality to \(e^{\lambda s(x)}\) bounds the tail itself:
\[ \rho = \Pr_{x \sim p}\big[e^{\lambda s(x)} \ge e^{\lambda \tau}\big] \le \mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}, \]
which is the Chernoff bound. So the second-moment bound from the last section says something tidy: the second moment is at most \(\rho\) times the Chernoff bound on \(\rho\). Dividing through by \(\rho^2\) puts it in relative terms:
\[ \frac{\mathbb{E}_{x \sim q_{\lambda}}\big[\mathbf{1}[s(x) \ge \tau]\, w(x)^2\big]}{\rho^2} \le \frac{\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\, e^{-\lambda \tau}}{\rho}. \]
The tilted estimator is inefficient exactly to the extent that the Chernoff bound is loose. On the coin, the bound at the matched strength gives \(4.5 \times 10^{-3}\) against the true \(\rho = 1.3 \times 10^{-3}\), loose by a factor of \(3.5\), so the guarantee promises per-draw relative variance at most \(2.5\); the measured value was \(2.02\).
That connection is also the optimality proof. For light-tailed scores the Chernoff bound is tight on the exponential scale (Cramér’s theorem), so the mean-matched tilt’s second moment is \(\rho^{2 - o(1)}\). No unbiased scheme can beat that exponent, since by Jensen’s inequality the second moment of any unbiased estimator is at least its squared mean, \(\rho^2\). Naive sampling sits at the other extreme with second moment \(\rho\), exponent one. In sample counts, naive needs on the order of \(1/\rho\) draws while the mean-matched tilt needs \(\rho^{-o(1)}\), fewer than any power of \(1/\rho\), which is the property the literature calls logarithmic efficiency. This is also the precise sense in which the linear relaxation of the cliff, rather than a sigmoid or any other softening, is the right one: the linear exponent is the one whose weight cap is the Chernoff bound, so its guarantee closes against the same duality that makes Chernoff tight, with no gap in the exponent.
The light-tail hypothesis is doing real work here. When the score is heavy-tailed, the normalizer \(\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\) is infinite for every positive \(\lambda\), the family is empty, and importance sampling built on tilts fails outright [GW, Ann. Appl. Probab. ’97]. The remedy there is a different mechanism entirely, steering by selection rather than by reweighting, and it gets its own page: prune and clone.
The Tilt You Already Trained
Reread the Gibbs variational principle with different nouns. Call the score a reward, call \(p\) a base language model, and call \(q\) a fine-tuned model. The objective “maximize expected reward minus a KL penalty toward the base model” is the standard RLHF objective, and the claim says its ideal optimum is the exponential tilt of the base model, at strength one over the penalty coefficient [KPB, Findings of EMNLP ’22]. Direct preference optimization takes exactly this closed form as its starting point [RSMEMF, NeurIPS ’23]. Aligning a model to a reward under a KL leash and building the optimal rare-event proposal are the same optimization.
For estimating rare model behaviors, this closes the loop in a practical way. The tilt of a language model by a sequence-level score is intractable to sample directly, since its normalizer \(\mathbb{E}_{x' \sim p}\big[e^{\lambda s(x')}\big]\) is a sum over all strings. But fine-tuning toward it with a KL penalty produces an ordinary autoregressive model: a sampler whose density is a product of per-token softmaxes, evaluable token by token. The importance weight is then \(p/q\) read off the two models directly, and the intractable normalizer never appears, because the network has amortized the normalizer into its per-token conditionals. Training only approximates the tilt, and the framework is built so that this is fine: unbiasedness needs only support, which softmaxes guarantee, and the approximation error is a variance cost, graded by the divergence from \(q^{\star}\) like everything else here. (In practice one also caps the weights by sampling the base model itself a small fraction \(\alpha\) of the time, a sequence-level defensive mixture that bounds every weight by \(1/\alpha\) [Hesterberg, Technometrics ’95].)
The portable lesson: when a rare event is the tail of a score, sample the tilt whose typical draw sits at the threshold. It is the cheapest distortion of the model that makes the event common, it is optimal on the exponential scale, and it stands in for the one proposal we can never have, which is conditioning itself.