Tilting at Rare Events
8 min read
Suppose we want to know how often a language model solves a problem at the edge of its ability, or produces a response its training was meant to prevent [WH, ’24], [JT+, ’25], [AS+, ’26]. If the behavior occurs once in a million generations, a thousand ordinary samples will usually show us none. We need a way to study those rare outcomes more often while still estimating their original probability.
We can sample from a distribution that favors the behavior, then give those oversampled outcomes smaller weights. The weights compensate for the change in sampling, so the estimate still targets the original probability. The best possible sampling distribution would be the original model conditioned on the rare event, but normalizing it requires the probability we are trying to find. Exponential tilting gives an approximation we can work with: gradually favor higher-scoring outputs, then account for that preference in the estimate.
Let \(p\) be the base distribution over complete outputs \(x\), let \(s(x)\) be a score that increases toward the behavior of interest, and choose a threshold \(\tau\). The event and its base probability are:
\[ A=\{x:s(x)\ge\tau\}, \qquad \rho=\Pr_{X\sim p}(X\in A). \]
Every estimator below targets this same \(\rho\) using \(m\) independent draws. They differ only in where those draws come from and how each draw is weighted.
The Naive Estimator
The first thing to try is to count hits, so the question is how many draws that takes. The naive estimator draws \(X_1,\ldots,X_m\) from the base distribution \(p\) and reports the fraction that land in \(A\):
\[ \hat\rho^{\mathrm{naive}} =\frac{1}{m}\sum_{i=1}^m\mathbf 1\{X_i\in A\}, \qquad X_i\sim p. \]
Each indicator is a Bernoulli random variable with success probability \(\rho\). The estimator therefore has:
\[ \mathbb E[\hat\rho^{\mathrm{naive}}]=\rho, \qquad \operatorname{Var}(\hat\rho^{\mathrm{naive}}) =\frac{\rho(1-\rho)}{m}. \]
Dividing by \(\rho^2\) gives its relative variance:
\[ \frac{\operatorname{Var}(\hat\rho^{\mathrm{naive}})}{\rho^2} =\frac{1-\rho}{m\rho} \approx\frac{1}{m\rho}. \]
The expected number of event hits is \(m\rho\). When \(m\rho\) is much smaller than one, the estimate is usually zero and its relative error is large. The way out is to draw from a distribution under which the event is common, and to correct each draw by a weight so that the target stays \(\rho\). The rest of the page is about choosing that distribution: the ideal choice is the base distribution conditioned on the event, which we cannot sample, and the exponential tilt is the sampleable relaxation of it.
The Proposal Estimator
If the draws come from somewhere other than \(p\), why does the estimate still target \(\rho\)? Let \(q\) be a proposal distribution and draw \(X_1,\ldots,X_m\) from \(q\) instead. A likelihood-ratio weight corrects each event hit back to the base distribution:
\[ w_q(x)=\frac{p(x)}{q(x)}, \qquad \hat\rho_q =\frac{1}{m}\sum_{i=1}^m\mathbf 1\{X_i\in A\}w_q(X_i), \qquad X_i\sim q. \]
Claim: if \(q(x)>0\) whenever \(x\in A\) and \(p(x)>0\), then the proposal estimator is unbiased:
\[ \mathbb E_{X\sim q}[\mathbf 1\{X\in A\}w_q(X)]=\rho. \]
Proof
Under the event-support condition, the likelihood ratio leaves the target unchanged:
\[ \begin{aligned} \mathbb E_{X\sim q}[\mathbf 1\{X\in A\}w_q(X)] &=\sum_x q(x)\mathbf 1\{x\in A\}\frac{p(x)}{q(x)} \\ &=\sum_{x\in A}p(x) \\ &=\rho. \end{aligned} \]
The event indicator restricts the sum before \(q(x)\) cancels. For general spaces, the corresponding condition is absolute continuity of the event-restricted base measure with respect to \(q\).
The proposal need not cover outcomes outside \(A\). Its variance is:
\[ \operatorname{Var}(\hat\rho_q) =\frac{1}{m}\left( \mathbb E_{X\sim q}[\mathbf 1\{X\in A\}w_q(X)^2]-\rho^2 \right) =\frac{1}{m}\left( \sum_{x\in A}\frac{p(x)^2}{q(x)}-\rho^2 \right). \]
Every valid proposal has the same expectation \(\rho\). The proposal changes the sum in the variance, so the question becomes which \(q\) makes that sum small.
The Conditional Distribution Is Optimal
Before choosing a proposal we can sample, it helps to know what the best possible proposal is. The base distribution conditioned on the event is:
\[ q^\star(x)=p(x\mid A) =\frac{p(x)\mathbf 1\{x\in A\}}{\rho}. \]
Define the chi-square divergence from this conditional distribution to a proposal \(q\):
\[ \chi^2(q^\star\Vert q) =\sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-1. \]
Lemma: if \(q\) covers the event and the second moment is finite, then:
\[ \operatorname{Var}(\hat\rho_q) =\frac{\rho^2}{m}\chi^2(q^\star\Vert q). \]
Derivation
On the event, \(p(x)=\rho q^\star(x)\). Substituting this into the proposal variance gives:
\[ \begin{aligned} \operatorname{Var}(\hat\rho_q) &=\frac{1}{m}\left( \rho^2\sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-\rho^2 \right) \\ &=\frac{\rho^2}{m}\left( \sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-1 \right) \\ &=\frac{\rho^2}{m}\chi^2(q^\star\Vert q). \end{aligned} \]
The first line substitutes the conditional density, the second factors out \(\rho^2\), and the third recognizes the divergence.
Chi-square divergence is nonnegative and is zero exactly when its two distributions agree. Setting \(q=q^\star\) makes \(\chi^2(q^\star\Vert q^\star)=0\), so the conditional proposal has zero variance and is optimal.
Why not use it, then? Evaluating its density requires the unknown normalizer \(\rho\), and direct conditional sampling is also usually unavailable. Rejection sampling from \(p\) needs \(1/\rho\) base draws per accepted output on average. We therefore need a proposal that moves toward \(q^\star\) without enforcing the event exactly.
Relax Conditioning to an Exponential Tilt
Conditioning multiplies \(p\) by an indicator of the event, which is all or nothing. What if we multiply by a smooth function of the score instead? For a strength \(\lambda\ge0\), define the normalizer and exponential tilt together:
\[ Z(\lambda)=\mathbb E_{X\sim p}[e^{\lambda s(X)}], \qquad q_\lambda(x)=\frac{p(x)e^{\lambda s(x)}}{Z(\lambda)}. \]
This is a probability distribution when \(0<Z(\lambda)<\infty\). Wherever the score is finite, the positive exponential factor preserves the support of \(p\). If \(s(x)>s(y)\), the relative odds \(q_\lambda(x)/q_\lambda(y)\) increase with \(\lambda\).
The most useful way to read \(\lambda\) is through the proposal’s mean score. At \(\lambda=0\), the tilt is the base distribution. When the derivatives exist, increasing \(\lambda\) moves the proposal mean upward according to:
\[ \frac{d}{d\lambda}\log Z(\lambda) =\mathbb E_{X\sim q_\lambda}[s(X)], \qquad \frac{d}{d\lambda}\mathbb E_{X\sim q_\lambda}[s(X)] =\operatorname{Var}_{X\sim q_\lambda}(s(X))\ge0. \]
The first identity translates a desired proposal mean into a value of \(\lambda\). The second says that a stronger positive tilt cannot lower that mean.
The likelihood-ratio weight under this proposal is:
\[ w_\lambda(x)=\frac{p(x)}{q_\lambda(x)} =Z(\lambda)e^{-\lambda s(x)}. \]
The tilted estimator has the same form as the naive estimator, with a different sampling distribution and the correction weight attached:
\[ \hat\rho^{\mathrm{tilt}}_\lambda =\frac{1}{m}\sum_{i=1}^m \mathbf 1\{X_i\in A\}Z(\lambda)e^{-\lambda s(X_i)}, \qquad X_i\sim q_\lambda. \]
Both estimators have expectation \(\rho\). The naive estimator receives mostly zero contributions; the tilted estimator trades those misses for variable likelihood weights.
The Variance as a Function of Lambda
The tilt has one knob, so the next question is how the variance depends on it. Substituting \(q_\lambda\) into the proposal-variance formula gives the exact variance:
\[ \begin{aligned} \operatorname{Var}(\hat\rho^{\mathrm{tilt}}_\lambda) &=\frac{1}{m}\left( \sum_{x\in A}\frac{p(x)^2}{q_\lambda(x)}-\rho^2 \right) \\ &=\frac{1}{m}\left( Z(\lambda)\sum_{x\in A}p(x)e^{-\lambda s(x)}-\rho^2 \right) \\ &=\frac{1}{m}\left( Z(\lambda)\mathbb E_{X\sim p} [\mathbf 1\{X\in A\}e^{-\lambda s(X)}]-\rho^2 \right). \end{aligned} \]
As \(\lambda\) grows, \(Z(\lambda)\) increases while the discounted event mass decreases. Their product determines the variance.
The variance differs from the product of the two \(\lambda\)-dependent factors only by constants. At a differentiable interior minimum, differentiating the log of that product gives:
\[ \begin{aligned} 0 &=\frac{d}{d\lambda}\log\left( Z(\lambda)\mathbb E_{X\sim p} [\mathbf 1\{X\in A\}e^{-\lambda s(X)}] \right) \\ &=\mathbb E_{X\sim q_\lambda}[s(X)] -\frac{ \mathbb E_{X\sim p}[\mathbf 1\{X\in A\}s(X)e^{-\lambda s(X)}] }{ \mathbb E_{X\sim p}[\mathbf 1\{X\in A\}e^{-\lambda s(X)}] }. \end{aligned} \]
The fraction is the mean score inside the event after weighting event outcomes by \(e^{-\lambda s}\). This equation rarely has a closed-form solution because it depends on the rare tail we are trying to estimate.
Match the Mean to the Threshold
If the exact optimum is out of reach, what should we minimize instead? If the proposal mean remains below \(\tau\), the event is still in the proposal’s upper tail and many draws still miss it. Pushing the mean far above \(\tau\) can overshoot the event boundary and increase the spread of the correction weights. A tractable upper bound on the variance identifies the point between those regimes.
For \(x\in A\), the event definition gives \(s(x)\ge\tau\), so:
\[ w_\lambda(x)\le Z(\lambda)e^{-\lambda\tau}, \qquad \rho\le Z(\lambda)e^{-\lambda\tau}. \]
The second inequality is the Chernoff bound. The same quantity also controls the estimator’s second moment.
Why the Chernoff quantity controls variance
One likelihood-ratio factor changes measure back to \(p\):
\[ \begin{aligned} \mathbb E_{q_\lambda}[\mathbf 1_Aw_\lambda^2] &=\mathbb E_p[\mathbf 1_Aw_\lambda] \\ &=Z(\lambda)\mathbb E_p[\mathbf 1_Ae^{-\lambda s}] \\ &\le Z(\lambda)e^{-\lambda\tau}\mathbb E_p[\mathbf 1_A] \\ &=\rho Z(\lambda)e^{-\lambda\tau}. \end{aligned} \]
The inequality uses \(s\ge\tau\) on the event.
At a differentiable interior minimum of this bound, differentiating its logarithm gives:
\[ \frac{d}{d\lambda}\big(\log Z(\lambda)-\lambda\tau\big) =\mathbb E_{X\sim q_\lambda}[s(X)]-\tau =0. \]
The sign has a direct interpretation. If the proposal mean is below \(\tau\), increasing \(\lambda\) lowers the bound; if the mean is above \(\tau\), increasing \(\lambda\) raises it. The minimum occurs when the proposal mean score equals the event threshold. This is the mean-matching rule.
Mean matching makes the event boundary typical under the proposal. It does not imply that the event probability is exactly one half, because the score distribution may be discrete or skewed, and it does not make \(q_\lambda\) equal to the conditional distribution.
When \(Z(\lambda)\) is available in closed form, this equation may give \(\lambda\) in closed form as well. Mean matching optimizes the Chernoff bound, not necessarily the exact finite-sample variance. Under standard light-tail large-deviation assumptions, it has the best possible exponential rate [DZ, ’10], [Bucklew, ’04].
When the exact sequence tilt is hard to sample
For a language model, the base distribution factorizes autoregressively, but a terminal score generally depends on the unseen future. For a prefix \(u\), define its future normalizer:
\[ F(u)=\mathbb E_p[e^{\lambda s(X_{1:T})}\mid X_{1:|u|}=u]. \]
The exact tilted next-token distribution is:
\[ q_\lambda(x_t\mid x_{<t}) =p(x_t\mid x_{<t}) \frac{F(x_{<t}x_t)}{F(x_{<t})}. \]
Computing the ratio requires summing over future completions. Kullback–Leibler (KL)-regularized reward fine-tuning instead fits an autoregressive model \(q_\theta\) that approximates the sequence tilt [KPB, Findings of EMNLP ’22], [RSMEMF, NeurIPS ’23]. Importance sampling then uses the evaluable ratio \(p(x)/q_\theta(x)\). The estimate remains unbiased if \(q_\theta\) has support wherever \(p\) has event mass; top-\(k\) truncation can violate this condition.
Compare the Estimators Interactively
How much does the tilt buy on a case we can compute exactly? The interaction uses 20 fair coin flips only as a final check: the score is the number of heads, and the event is at least 17 heads. Under the base coin, this event has probability \(1.288\times10^{-3}\), or about one hit in 776 draws.
For this additive score, the normalizer is \(Z(\lambda)=((1+e^\lambda)/2)^{20}\), and the tilted coin’s mean head count is \(20e^\lambda/(1+e^\lambda)\). Mean matching sets that mean to the event threshold of 17 heads:
\[ 20\frac{e^\lambda}{1+e^\lambda}=17 \quad\Longrightarrow\quad \lambda=\log\frac{17}{3}, \qquad \frac{e^\lambda}{1+e^\lambda}=\frac{17}{20}=0.85. \]
Compare naive and tilted sampling
Both estimate the fair coin's probability of at least 17 heads.
Mean matching places the event threshold at the proposal mean of 17 heads.
At the mean match, a proposal draw hits the event about \(64.8\%\) of the time. Ten percent relative standard deviation needs about 203 tilted draws, versus about 78,000 naive draws. The target probability is unchanged; the tilt makes useful draws common and the likelihood ratio corrects for that change.