How I* built Twos

A game of 2048 with a trained model playing the same tiles beside you. This is the path from one question about search versus learning to five models running in your browser, including the version I threw away.

*and Claude

It started in 15-112

In the fall of 2025 I took 15-112, Carnegie Mellon's introductory programming course, with Professor David Kosbie. This was a paradigm changing class for me, which culminated in a term project where we coded a game from scratch (no AI).

Watching Code Bullet beat it

Around the same time I watched Code Bullet's AI learns to play 2048. He builds the game from scratch and then an AI that plays it, and his project describes the approach as a look-ahead algorithm in which you try the moves, see where they lead, pick the best.

Code Bullet's video was the spark. His AI looks ahead rather than learning from experience.

His AI searches, whereas most of the reinforcement learning I had read about learns from experience instead, playing thousands of games and slowly getting better. I wanted to see which one actually works better on a game like this. And could you combine them?

The rules are trickier than they look

2048 looks simple until you write it down. A tile can only merge once per move, so a row of 2 2 2 2 slides left to 4 4, not 8. A row of 2 2 4 becomes 4 4, because the new 4 doesn't get to merge again. When three equal tiles line up, the pair nearest the wall merges first. A move that changes nothing doesn't count, and no new tile appears. I wrote each of those as a unit test before writing the code that passes it, which is the most 112 thing in this whole project.

For speed I store each tile as its exponent (2 is 1, 4 is 2, 2048 is 11), which fits in four bits. A row of four tiles then packs into a 16-bit number, and there are only 65,536 possible rows, so I solve every one of them once when the program starts. After that, every move in every direction is a flip or transpose of the board plus four lookups. One game runs at about 23,000 moves a second in Python, and the neural networks later trained on batches of 32 to 64 games at once.

The row 2, 2, 4, empty becomes the key 4,384, which looks up the slid row 4, 4, empty, empty and a reward of 4 points.
Every slide is a table lookup. The table is built once from the same merge rules the tests check.

Implementing my strategy

The way most people learn to play 2048 (and the way I played during boring high-school classes) is to pick a corner, park the biggest tile there, and keep a descending chain behind it so merges keep cascading toward the corner. I turned that into a heuristic: a score for any board that rewards big tiles near the corner along a snake-shaped path, empty cells, and neighboring tiles that can merge, and penalizes big jumps between neighbors. The player looks one move ahead and picks whichever move leads to the best-scoring board.

It turned out that by random moves alone you can achieve an average score of about 1,000 points. The heuristic approach averages 10,007 and reaches 2048 in 7% of games.

Five ways to play

Every approach plays the same game, framed the same way: a Markov decision process. The state is the board, the action is one of four moves, and the reward is the points from whatever merged. What separates them is how they decide which move to make.

1. The heuristic

Covered above: a score I designed by hand, and a one-move lookahead. It's in the lineup as the floor.

Nothing is learned here: every weight in the score is a number I picked and tuned by watching it play.

2. The N-tuple network, trained with temporal-difference learning

This is the approach most 2048 research builds on, from Yeh et al. (2016). Instead of a neural network, it scores a board by looking at a few small groups of cells and adding up numbers from a lookup table for each group. Training is just adjusting those numbers.

Four 4 by 4 grids, each with six highlighted cells: the top row plus two cells below, the second row plus two, a 2 by 3 block in the corner, and a 2 by 3 block one row down.
The four patterns, drawn straight from the code. Sharing each one across all eight symmetries means one game teaches every orientation at once.

It scores the board right after my move, before the random new tile appears, which takes some of the luck out of what it has to learn.

It learns with temporal-difference learning: after each move, it nudges its score for the old board toward what actually happened next.

It learned fast. In a 60-minute run it played 55,022 games against itself, and the share of its recent games reaching 2048 went from zero to 90%.

Line chart: the N-tuple network's 2048 rate over its last 100 training games climbs from 0% to about 50% by minute 10, 75% by minute 25, and 90% at minute 60.
Share of the last 100 training games that reached 2048, logged every ten seconds.

3. The deep Q-network

Deep Q-learning is the method made famous for learning Atari games. A neural network looks at the board and estimates how many points each of the four moves will lead to, then picks the highest.

Diagram: the board as 16 one-hot planes feeds two 2 by 2 convolutions and a dense layer of 256, which splits into a DQN head of four Q-values or PPO heads of four probabilities plus a value.
DQN and PPO share the same body and differ only in what comes out of it.

It trains by playing lots of games at once, saving recent moves, and learning from them in batches. It skips illegal moves, and it shrinks big rewards so late-game merges don't throw the training off.

The first version trained very slowly. Rewriting one piece of the network's math in an equivalent but faster way made training about 7× faster.

Bar chart: one training step takes 16 milliseconds with nn.Conv2d and 2.16 milliseconds with PairConv.
Same network, same math, one forward and backward pass on 256 boards.

After 30 minutes and about 15,500 games, it averaged 7,687 in the benchmark and reached 2048 in 3% of games. Below my heuristic.

4. PPO

Proximal policy optimization learns which move to make directly, as a probability for each move, along with a guess at how good the position is. It plays a batch of games, learns from them, and repeats.

Like the DQN, it ignores illegal moves, which mattered a lot late in games when only one or two moves are possible.

After 30 minutes and about 52,700 games, it averaged 7,351 and never reached 2048. Also below my heuristic.

5. Search on top of learning

This is where Code Bullet's idea came back. Expectimax search looks a move or two ahead, like his AI did, and averages over where the next random tile could land. Instead of searching all the way to the end of the game, it stops and asks the N-tuple network how good the board looks.

Tree diagram: the board branches into up, left and right moves; each move branches into every possible new tile, weighted by probability; the leaves are scored by the N-tuple network.
Max over my moves, average over the game's tiles, and learned values where the search stops.

It looks further ahead when the board is crowded, since that's when a mistake costs the most.

It doesn't learn anything new on its own. It just combines what the N-tuple network learned with a little lookahead.

How they compare

Line chart of average score over the last 100 training games. The N-tuple network climbs steadily from about 8,000 to 57,800 over 60 minutes. DQN and PPO rise quickly to around 7,000 and flatten, ending at 7,800 and 6,300 after 30 minutes.
Average score of recent training games. All three trained at the same time on one laptop CPU.

The neural networks got half the training time, so this isn't a perfectly fair comparison. But the N-tuple network was still improving at the hour mark, while the neural networks had leveled off.

Bar chart of mean score over 30 seeded games: Search plus N-tuple 109,744; N-tuple network 55,334; Heuristic 10,007; Deep Q-network 7,687; PPO 7,351.
Thirty seeded games each, played by the same JavaScript the site runs. Search plus learning reached 2048 in 97% of games, and 4096 in all of those.

So the answer to the question Code Bullet left me with: search and learning are better together. Learning alone scored five and a half times my heuristic, and adding a shallow search on top doubled that again.

Version two: a game first

The current site: your board on the left early in a game, the AI's board on the right at about 12,000 points with a 1024 tile, and the AI's move ratings open below it.
The page as it is now. The model has been playing for under a minute.

One detail I'm proud of: both boards draw their new tiles from the same seeded random sequence. You and the model get the same luck, as far as two different games allow, so the comparison means something. Press New game and both boards restart from one new seed. And when the model finishes, Run again replays the exact same tiles, which makes switching models a fair head-to-head.

The AI board with a soft glow on three tiles, and bars rating each move: up 33.9k, down 33.9k, left 26.9k, right 34.0k, with right highlighted.
Open the panel and you see each move's rating before it's played, plus a glow on the tiles that drove the choice.
Your board with a 2048 tile and the message 'You made 2048', with Keep going and New game buttons.
The rules and flow match the original game, including the choice to keep going after 2048.

For the glow, each model explains itself in its own terms. For the N-tuple models, every active table weight is spread over the tiles its pattern covers. For the neural networks I use occlusion: take one tile away, run the network again, and see how much its answer moves.

Running every model in your browser

Version one needed a Python server for every move. Version two has no server at all. I trained the models in Python, exported their weights to small files, and rewrote the game and all five players in plain JavaScript that runs in a background thread (a web worker), so the animations never stutter while the search is thinking.

Fitting the model through a phone connection

The full N-tuple tables would be 268 MB. Training only ever touches a fraction of the entries, so I ship just those: sorted keys stored as the gaps between them, which compress well, and weights stored as 16-bit integers with one scale per table. That still left 10 MB. So I ran a sweep, dropping every weight below a threshold and measuring the search at each step. The answer surprised me: weights under 50 points are close to noise. Dropping them cut the file to 1.6 MB with no measurable loss in strength.

Line chart of mean search score against file size: from 10.2 MB down to 1.6 MB the score stays between about 91,000 and 104,000.
Each point drops more small weights. The score barely moves while the file shrinks by 84%.
The site on a phone: the Twos title, a short description, links, and your board with a swipe hint.
On a phone the boards stack and you swipe to move.

What's next

The N-tuple network was still improving when I stopped it, so the obvious next step is to train longer, and to try the multi-stage version from the Yeh paper, which uses separate tables for the early and late game. I'd also like to give the neural networks a fair rematch with equal training time and a few upgrades, like multi-step returns, and see how close they can get. And if a deeper search can stay fast enough in the browser, I want to know how often it reaches 8192.