Tilting at Rare Events
7 min read
A thousand generations with no successes tell us little about a behavior that occurs once in a million attempts. That makes rare language-model behaviors difficult to study: a model might occasionally solve a problem at the edge of its ability, or produce a response its training was meant to prevent [WH, ’24], [JT+, ’25], [AS+, ’26]. We would like to make the behavior happen more often in our experiment while still measuring how rare it originally was.
Exponential tilting lets us favor outputs closer to the behavior we want to study. Surprisingly, weighting these samples appropriately still lets us estimate the original probability.
The Naive Estimator
Let \(p\) be the base distribution over complete outputs \(x\), let \(s(x)\) be a score that increases toward the behavior of interest, and choose a threshold \(\tau\). The event and the probability we want are:
\[ A=\{x:s(x)\ge\tau\}, \qquad \rho=\Pr_{X\sim p}(X\in A). \]
The simplest estimate uses \(m\) independent draws \(X_1,\ldots,X_m\) from \(p\) and reports the fraction that land in \(A\):
\[ \hat\rho^{\mathrm{naive}} =\frac{1}{m}\sum_{i=1}^m\mathbf 1\{X_i\in A\}, \qquad X_i\sim p. \]
Each indicator is a Bernoulli random variable with success probability \(\rho\). The estimator therefore has:
\[ \mathbb E[\hat\rho^{\mathrm{naive}}]=\rho, \qquad \operatorname{Var}(\hat\rho^{\mathrm{naive}}) =\frac{\rho(1-\rho)}{m}. \]
Dividing by \(\rho^2\) gives its relative variance:
\[ \frac{\operatorname{Var}(\hat\rho^{\mathrm{naive}})}{\rho^2} =\frac{1-\rho}{m\rho} \approx\frac{1}{m\rho}. \]
The expected number of event hits is \(m\rho\). When \(m\rho\) is much smaller than one, the estimate is usually zero and its relative error is large. The way out is to draw from a distribution under which the event is common, and to correct each draw by a weight so that the target stays \(\rho\).
The Proposal Estimator
Let \(q\) be a proposal distribution and draw \(X_1,\ldots,X_m\) from \(q\) instead. A likelihood-ratio weight corrects each event hit back to the base distribution:
\[ w_q(x)=\frac{p(x)}{q(x)}, \qquad \hat\rho_q =\frac{1}{m}\sum_{i=1}^m\mathbf 1\{X_i\in A\}w_q(X_i), \qquad X_i\sim q. \]
Require \(q(x)>0\) whenever \(x\in A\) and \(p(x)>0\). Under this event-support condition, the sampling probability cancels its weight, giving unbiasedness:
\[ \mathbb E_{X\sim q}[\mathbf 1\{X\in A\}w_q(X)] =\sum_{x\in A}q(x)\frac{p(x)}{q(x)} =\sum_{x\in A}p(x) =\rho. \]
Zero-probability terms are omitted from the sum. For general spaces, the condition is absolute continuity of the event-restricted base measure with respect to \(q\).
The proposal need not cover outcomes outside \(A\). Its variance is:
\[ \operatorname{Var}(\hat\rho_q) =\frac{1}{m}\left( \mathbb E_{X\sim q}[\mathbf 1\{X\in A\}w_q(X)^2]-\rho^2 \right) =\frac{1}{m}\left( \sum_{x\in A}\frac{p(x)^2}{q(x)}-\rho^2 \right). \]
Every valid proposal has the same expectation \(\rho\). We can choose \(q\) to make the sum in the variance small.
The Conditional Distribution Is Optimal
Before choosing a proposal we can sample, it helps to know what the best possible proposal is. The base distribution conditioned on the event is:
\[ q^\star(x)=p(x\mid A) =\frac{p(x)\mathbf 1\{x\in A\}}{\rho}. \]
Every draw from \(q^\star\) succeeds, and its likelihood ratio is \(p(x)/q^\star(x)=\rho\). The weighted estimate therefore equals \(\rho\) on every draw.
Evaluating its density requires the unknown normalizer \(\rho\), and direct conditional sampling is also usually unavailable. Rejection sampling from \(p\) needs \(1/\rho\) base draws per accepted output on average. We therefore need a proposal that moves toward \(q^\star\) without enforcing the event exactly.
To measure how far another proposal is from this ideal, define the chi-square divergence:
\[ \chi^2(q^\star\Vert q) =\sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-1. \]
Lemma: if \(q\) covers the event and the second moment is finite, then:
\[ \operatorname{Var}(\hat\rho_q) =\frac{\rho^2}{m}\chi^2(q^\star\Vert q). \]
Derivation
On the event, \(p(x)=\rho q^\star(x)\). Substituting this into the proposal variance gives:
\[ \begin{aligned} \operatorname{Var}(\hat\rho_q) &=\frac{1}{m}\left( \rho^2\sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-\rho^2 \right) \\ &=\frac{\rho^2}{m}\left( \sum_{x:q(x)>0}\frac{q^\star(x)^2}{q(x)}-1 \right) \\ &=\frac{\rho^2}{m}\chi^2(q^\star\Vert q). \end{aligned} \]
The first line substitutes the conditional density, the second factors out \(\rho^2\), and the third recognizes the divergence.
Chi-square divergence is nonnegative and is zero exactly when its two distributions agree. Setting \(q=q^\star\) makes \(\chi^2(q^\star\Vert q^\star)=0\), so the conditional proposal has zero variance and is optimal.
Relax Conditioning to an Exponential Tilt
Conditioning multiplies \(p\) by an indicator of the event, which is all or nothing. A smooth function of the score relaxes that restriction. For a strength \(\lambda\ge0\), define the normalizer and exponential tilt together:
\[ Z(\lambda)=\mathbb E_{X\sim p}[e^{\lambda s(X)}], \qquad q_\lambda(x)=\frac{p(x)e^{\lambda s(x)}}{Z(\lambda)}. \]
This is a probability distribution when \(0<Z(\lambda)<\infty\). Using it for importance sampling also requires a way to sample from it and evaluate its normalizer; the exponential formula alone does not make those tasks easy. Wherever the score is finite, the positive exponential factor preserves the support of \(p\). If \(s(x)>s(y)\), the relative odds \(q_\lambda(x)/q_\lambda(y)\) increase with \(\lambda\).
The most useful way to read \(\lambda\) is through the proposal’s mean score. At \(\lambda=0\), the tilt is the base distribution. When the derivatives exist, increasing \(\lambda\) moves the proposal mean upward according to:
\[ \frac{d}{d\lambda}\log Z(\lambda) =\mathbb E_{X\sim q_\lambda}[s(X)], \qquad \frac{d}{d\lambda}\mathbb E_{X\sim q_\lambda}[s(X)] =\operatorname{Var}_{X\sim q_\lambda}(s(X))\ge0. \]
The first identity translates a desired proposal mean into a value of \(\lambda\). The second says that a stronger positive tilt cannot lower that mean.
The likelihood-ratio weight under this proposal is:
\[ w_\lambda(x)=\frac{p(x)}{q_\lambda(x)} =Z(\lambda)e^{-\lambda s(x)}. \]
The tilted estimator has the same form as the naive estimator, with a different sampling distribution and the correction weight attached:
\[ \hat\rho^{\mathrm{tilt}}_\lambda =\frac{1}{m}\sum_{i=1}^m \mathbf 1\{X_i\in A\}Z(\lambda)e^{-\lambda s(X_i)}, \qquad X_i\sim q_\lambda. \]
Both estimators have expectation \(\rho\). The naive estimator receives mostly zero contributions; the tilted estimator trades those misses for variable likelihood weights.
The Variance as a Function of Lambda
The tilt strength controls the tradeoff between more frequent successes and more variable weights. Substituting \(q_\lambda\) into the proposal-variance formula gives the exact variance:
\[ \begin{aligned} \operatorname{Var}(\hat\rho^{\mathrm{tilt}}_\lambda) &=\frac{1}{m}\left( \sum_{x\in A}\frac{p(x)^2}{q_\lambda(x)}-\rho^2 \right) \\ &=\frac{1}{m}\left( Z(\lambda)\sum_{x\in A}p(x)e^{-\lambda s(x)}-\rho^2 \right) \\ &=\frac{1}{m}\left( Z(\lambda)\mathbb E_{X\sim p} [\mathbf 1\{X\in A\}e^{-\lambda s(X)}]-\rho^2 \right). \end{aligned} \]
Increasing \(\lambda\) changes both the normalizer and the event-weighted expectation. Their product determines whether the greater frequency of event hits compensates for the variation in their weights.
The variance differs from the product of the two \(\lambda\)-dependent factors only by constants. At a differentiable interior minimum, differentiating the log of that product gives:
\[ \begin{aligned} 0 &=\frac{d}{d\lambda}\log\left( Z(\lambda)\mathbb E_{X\sim p} [\mathbf 1\{X\in A\}e^{-\lambda s(X)}] \right) \\ &=\mathbb E_{X\sim q_\lambda}[s(X)] -\frac{ \mathbb E_{X\sim p}[\mathbf 1\{X\in A\}s(X)e^{-\lambda s(X)}] }{ \mathbb E_{X\sim p}[\mathbf 1\{X\in A\}e^{-\lambda s(X)}] }. \end{aligned} \]
The fraction is the mean score inside the event after weighting event outcomes by \(e^{-\lambda s}\). This equation rarely has a closed-form solution because it depends on the rare tail we are trying to estimate.
Match the Mean to the Threshold
A bound on the variance gives us a more tractable objective to minimize. If the proposal mean remains below \(\tau\), the event is still in the proposal’s upper tail and many draws still miss it. Pushing the mean far above \(\tau\) can overshoot the event boundary and increase the spread of the correction weights. A tractable upper bound on the variance identifies the point between those regimes.
For \(x\in A\), the event definition gives \(s(x)\ge\tau\), so:
\[ w_\lambda(x)\le Z(\lambda)e^{-\lambda\tau}, \qquad \rho\le Z(\lambda)e^{-\lambda\tau}. \]
The second inequality is the Chernoff bound. The same quantity also controls the estimator’s second moment.
Why the Chernoff quantity controls variance
One likelihood-ratio factor changes measure back to \(p\):
\[ \begin{aligned} \mathbb E_{q_\lambda}[\mathbf 1_Aw_\lambda^2] &=\mathbb E_p[\mathbf 1_Aw_\lambda] \\ &=Z(\lambda)\mathbb E_p[\mathbf 1_Ae^{-\lambda s}] \\ &\le Z(\lambda)e^{-\lambda\tau}\mathbb E_p[\mathbf 1_A] \\ &=\rho Z(\lambda)e^{-\lambda\tau}. \end{aligned} \]
The inequality uses \(s\ge\tau\) on the event.
At a differentiable interior minimum of this bound, differentiating its logarithm gives:
\[ \frac{d}{d\lambda}\big(\log Z(\lambda)-\lambda\tau\big) =\mathbb E_{X\sim q_\lambda}[s(X)]-\tau =0. \]
The sign has a direct interpretation. If the proposal mean is below \(\tau\), increasing \(\lambda\) lowers the bound; if the mean is above \(\tau\), increasing \(\lambda\) raises it. The minimum occurs when the proposal mean score equals the event threshold. This is the mean-matching rule.
Mean matching makes the event boundary typical under the proposal. It does not imply that the event probability is exactly one half, because the score distribution may be discrete or skewed, and it does not make \(q_\lambda\) equal to the conditional distribution.
When \(Z(\lambda)\) is available in closed form, this equation may give \(\lambda\) in closed form as well. Mean matching optimizes the Chernoff bound, not necessarily the exact finite-sample variance. Under standard light-tail large-deviation assumptions, it has the best possible exponential rate [DZ, ’10], [Bucklew, ’04].
When the exact sequence tilt is hard to sample
For a language model, the base distribution factorizes autoregressively, but a terminal score generally depends on the unseen future. For a prefix \(u\), define its future normalizer:
\[ F(u)=\mathbb E_p[e^{\lambda s(X_{1:T})}\mid X_{1:|u|}=u]. \]
The exact tilted next-token distribution is:
\[ q_\lambda(x_t\mid x_{<t}) =p(x_t\mid x_{<t}) \frac{F(x_{<t}x_t)}{F(x_{<t})}. \]
Computing the ratio requires summing over future completions. Kullback–Leibler (KL)-regularized reward fine-tuning instead fits an autoregressive model \(q_\theta\) that approximates the sequence tilt [KPB, Findings of EMNLP ’22], [RSMEMF, NeurIPS ’23]. Importance sampling then uses the evaluable ratio \(p(x)/q_\theta(x)\). The estimate remains unbiased if \(q_\theta\) has support wherever \(p\) has event mass; top-\(k\) truncation can violate this condition.
Compare the Estimators Interactively
Consider 20 fair coin flips: the score is the number of heads, and the event is at least 17 heads. Under the base coin, this event has probability \(1.288\times10^{-3}\), or about one hit in 776 draws.
For this additive score, the normalizer is \(Z(\lambda)=((1+e^\lambda)/2)^{20}\), and the tilted coin’s mean head count is \(20e^\lambda/(1+e^\lambda)\). Mean matching sets that mean to the event threshold of 17 heads:
\[ 20\frac{e^\lambda}{1+e^\lambda}=17 \quad\Longrightarrow\quad \lambda=\log\frac{17}{3}, \qquad \frac{e^\lambda}{1+e^\lambda}=\frac{17}{20}=0.85. \]
Compare naive and tilted sampling
Both estimate the fair coin's probability of at least 17 heads.
Mean matching places the event threshold at the proposal mean of 17 heads.
At the mean match, a proposal draw hits the event about \(64.8\%\) of the time. Ten percent relative standard deviation needs about 203 tilted draws, versus about 78,000 naive draws. The target probability is unchanged; the tilt makes useful draws common and the likelihood ratio corrects for that change.