Sinusoidal position encodings
Easy10 min10 hidden tests
Attention by itself cannot tell token order, so a translation model adds a position vector to each token embedding. Write sinusoidal_positions(positions, d_model) that returns the fixed sine and cosine encodings from the original transformer paper, one row per position. Pair i of columns (2i and 2i + 1) shares one frequency, 1 / 10000 ** (2i / d_model): column 2i holds the sine of position times frequency and column 2i + 1 the cosine. Position 0 is always [0, 1, 0, 1, ...].
Tap a position
pair i: sin and cos of p / 10000^(2i / d_model)Position 1: pair 0 (frequency 1): sin 1 = 0.8415, cos = 0.5403; pair 1 (frequency 0.1): sin 0.1 = 0.0998, cos = 0.9950; pair 2 (frequency 0.01): sin 0.01 = 0.0100, cos = 1.0000; …. Left pairs turn fast and right pairs slowly, so every position gets its own pattern. Position 0 is always [0, 1, 0, 1, …].
Position 1 encoded with 8 numbers.
Examples
Position 0 is [0, 1, 0, 1]; the second pair turns 100 times slower
sinusoidal_positions([0, 1], 4)
returns [[0, 1, 0, 1], [0.841471, 0.540302, 0.00999983, 0.99995]]
Width 2 has a single pair with frequency 1
sinusoidal_positions([2, 0, 5], 2)
returns [[0.909297, -0.416147], [0, 1], [-0.958924, 0.283662]]
Requirements
positionsis a list of non-negative integers (any order, any length of 1 or more).d_modelis an even integer of 2 or more.- For pair i in 0 to d_model / 2 - 1, let
angle = p / 10000 ** (2 * i / d_model). - Column 2i is
sin(angle)and column 2i + 1 iscos(angle), interleaved in that order. - Return a float array of shape (len(positions), d_model), row k for
positions[k].
The idea behind it
Self-attention treats its input as a set: shuffle the tokens and each output just moves with its token. Word order has to be added. The original transformer, "Attention Is All You Need" (2017), adds a fixed vector to each embedding. Column pairs are sine and cosine waves of the position, with wavelengths that grow geometrically from 2 pi toward 10000 times 2 pi. Fast waves tell neighbors apart and slow waves tell far positions apart, a little like the hands of a clock. Because sin(a + b) and cos(a + b) are linear in sin(a) and cos(a), the encoding of position p + k is a fixed rotation of the encoding of p, which makes relative offsets easy for the model to learn. Many modern LLMs use RoPE instead, which applies the same rotations directly to queries and keys.