Last lecture’s transformer had no idea where its tokens were, and patched the hole with a learned table of position vectors. Today’s fix is RoPE: rotate the query at position \(m\) by the angle \(m\theta\), rotate the key at position \(n\) by \(n\theta\), and let the in-class identity \(\mathbf{R}_m^\top\mathbf{R}_n = \mathbf{R}_{n-m}\) do the rest. Here we watch its headline consequence. For one repeated token, the attention score matrix is Toeplitz: constant along every diagonal.
We use a small head, \(d = 8\), so the frequency ladder collapses to clean powers of ten, \(\theta_j = 10000^{-2j/8} = 10^{-j}\).
import numpy as npimport matplotlib.pyplot as pltimport scienceplotsfrom matplotlib.colors import LinearSegmentedColormapplt.style.use(["science", "no-latex"])TEAL, CARDINAL, GRAY ="#009090", "#9c1b33", "#c9c9c9"SCORE_CMAP = LinearSegmentedColormap.from_list("course_div", [CARDINAL, "white", TEAL])def rotation(angle):return np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])def block_rotation(m, thetas):# The RoPE matrix Theta_m: one 2x2 rotation block per frequency. d =2*len(thetas) R = np.zeros((d, d))for j, theta inenumerate(thetas): R[2*j:2*j+2, 2*j:2*j+2] = rotation(m * theta)return Rd =8thetas =10000.0** (-2* np.arange(d //2) / d)print("Frequency ladder:", thetas)rng = np.random.default_rng(30)q = rng.standard_normal(d) # one query content vectork = rng.standard_normal(d) # one key content vectorprint(f"||q|| = {np.linalg.norm(q):.6f}")
Frequency ladder: [1. 0.1 0.01 0.001]
||q|| = 3.530342
Rotation never changes a query’s length
In class we proved \(\|\mathbf{R}_m\mathbf{q}\| = \|\mathbf{q}\|\) with a 2x2 computation. The block-diagonal \(\mathbf{\Theta}_m\) inherits the property block by block. Let’s check it at positions far beyond anything we did by hand.
for m in [0, 1, 7, 100, 5000]:print(f"||Theta_{m} q|| = {np.linalg.norm(block_rotation(m, thetas) @ q):.6f}")
Position \(5000\) has wound the fast hand around the circle nearly \(800\) times, and the norm has not budged: rotation changes which keys a query aligns with, never how loudly it speaks.
The score matrix of a repeated token
Now the main event. Plant the same content \(\mathbf{q}, \mathbf{k}\) at sixteen positions (so the scores isolate what position alone contributes), rotate each copy by its own \(\mathbf{\Theta}_m\), and score every query against every key.
n_pos =16rotated_q = np.stack([block_rotation(m, thetas) @ q for m inrange(n_pos)])rotated_k = np.stack([block_rotation(n, thetas) @ k for n inrange(n_pos)])S = rotated_q @ rotated_k.T / np.sqrt(d)vmax = np.abs(S).max()fig, ax = plt.subplots(figsize=(6, 4.4))im = ax.imshow(S, cmap=SCORE_CMAP, vmin=-vmax, vmax=vmax)ax.set_xticks(range(n_pos)); ax.set_yticks(range(n_pos))ax.set_xticklabels(range(n_pos), fontsize=7)ax.set_yticklabels(range(n_pos), fontsize=7)ax.tick_params(which="both", bottom=False, left=False, top=False, right=False)ax.set_xlabel("Key position $n$")ax.set_ylabel("Query position $m$")cbar = fig.colorbar(im, ax=ax, fraction=0.045, pad=0.03)cbar.set_label("Attention score")plt.show()
Stripes, parallel to the main diagonal. The eye says Toeplitz; let’s make numpy say it too, by measuring how much the entries vary along each of the \(31\) diagonals. While we are here, we can also check the class identity directly, entry by entry, against \(\mathbf{q}^\top\mathbf{\Theta}_{n-m}\mathbf{k}/\sqrt{d}\).
diag_stds = [S.diagonal(offset).std() for offset inrange(-n_pos +1, n_pos)]print(f"Largest std along any diagonal: {max(diag_stds):.2e}")identity_err =max(abs(S[m, n] - q @ block_rotation(n - m, thetas) @ k / np.sqrt(d))for m inrange(n_pos) for n inrange(n_pos))print(f"Largest |S[m,n] - q.Theta_(n-m).k/sqrt(d)|: {identity_err:.2e}")
Largest std along any diagonal: 1.43e-16
Largest |S[m,n] - q.Theta_(n-m).k/sqrt(d)|: 4.44e-16
Zero, up to floating point. Every diagonal is constant, and every entry equals the class identity’s prediction \(\mathbf{q}^\top\mathbf{\Theta}_{n-m}\mathbf{k}/\sqrt{d}\): the score depends on \(n - m\) and on nothing else.
The hands of the clock
Each 2D slice of the embedding is a hand advancing \(\theta_j\) radians per position. Let’s track our fastest and our second hand across the sixteen positions.
fig, axes = plt.subplots(1, 2, figsize=(7, 3.6))for ax, theta, caption inzip(axes, [1.0, 0.1], ["Fast hand: $\\theta_0 = 1$", "Slower hand: $\\theta_1 = 0.1$"]): ax.add_patch(plt.Circle((0, 0), 1, fill=False, color=GRAY, linewidth=1)) fast = theta ==1.0for m inrange(n_pos): angle = m * theta near_collision = fast and m in (0, 6) color = CARDINAL if near_collision else TEAL ax.plot(np.cos(angle), np.sin(angle), "o", color=color, markersize=4)ifnot (fast or m %5==0):continue# the slow hand's positions crowd together; label every fifth r_label =1.24if (fast and m ==6) else1.16 ax.text(r_label * np.cos(angle), r_label * np.sin(angle), str(m), ha="center", va="center", fontsize=7, color=color if near_collision else"black") ax.set_xlim(-1.5, 1.5); ax.set_ylim(-1.5, 1.5) ax.set_aspect("equal"); ax.set_xticks([]); ax.set_yticks([]) ax.set_xlabel(caption)plt.show()
The fast hand laps the circle twice, and positions \(0\) and \(6\) (cardinal) land closer together than any two consecutive positions do. One hand repeats itself, which is the aliasing Problem 22 turns into a formula. The slower hand fans the same sixteen positions out in unambiguous order, but into a wedge of only \(1.5\) radians (only every fifth position is labelled, since they crowd together). Fast hands distinguish nearby positions, slow hands distinguish distant ones, and RoPE uses a whole ladder of frequencies.
What if the content varied? (change me!)
The stripes came from position, not from attention in general. Set VARY_CONTENT = True to give every position its own random token. Or leave it False and play with BASE (the ladder’s \(10000\)) or N_POS (the sequence length), and watch the stripes stretch and shrink.
VARY_CONTENT =True# change me!BASE =10000.0# change me too: try 100, or 2N_POS =16# or make the sequence longerthetas_wi = BASE ** (-2* np.arange(d //2) / d)rng_wi = np.random.default_rng(31)S_wi = np.zeros((N_POS, N_POS))for m inrange(N_POS):for n inrange(N_POS): q_mn = rng_wi.standard_normal(d) if VARY_CONTENT else q k_mn = rng_wi.standard_normal(d) if VARY_CONTENT else k S_wi[m, n] = block_rotation(m, thetas_wi) @ q_mn @ (block_rotation(n, thetas_wi) @ k_mn) / np.sqrt(d)vmax = np.abs(S_wi).max()fig, ax = plt.subplots(figsize=(6, 4.4))im = ax.imshow(S_wi, cmap=SCORE_CMAP, vmin=-vmax, vmax=vmax)ax.set_xlabel("Key position $n$"); ax.set_ylabel("Query position $m$")ax.set_xticks([]); ax.set_yticks([])cbar = fig.colorbar(im, ax=ax, fraction=0.045, pad=0.03)cbar.set_label("Attention score")plt.show()diag_stds = [S_wi.diagonal(offset).std() for offset inrange(-N_POS +1, N_POS)]print(f"Largest std along any diagonal: {max(diag_stds):.2e}")
Largest std along any diagonal: 1.38e+00
With varying content the largest diagonal deviation jumps from \(10^{-16}\) to about \(1.4\): the stripes dissolve, because content differences now sit on top of the positional pattern. In a trained model both effects coexist, with content deciding what to attend to and RoPE modulating it by relative position only.
Punchline: rotation turns absolute position into relative position. Each token is stamped with its own position, but the moment two tokens are compared, \(\mathbf{R}_m^\top\mathbf{R}_n = \mathbf{R}_{n-m}\) cancels everything except the offset.
