I built this in a day, working with coding agents, at the California Institute for Machine Consciousness hackathon "Build a Predictive Mirror", in San Francisco on 30 October 2026. A predictive mirror shows you yourself, generated entirely from a model's guesses: delay the camera, and ask the model to fill in the present from what it has seen so far. My version runs in a browser tab on a laptop. It tracks your face and hands, runs seven predictors on your motion (four of them train from scratch while you watch) and scores each guess against the real frame when it arrives. You can try it with a webcam. The video stays in your browser.
A webcam session is an easy test, because most of the time you sit still. Sign language is much harder. The hands carry the meaning and move fast: in this video they travel 67.1 pixels in an average second, about a fifth of the frame's height. If a model could anticipate what was about to be signed, it would show up directly as low hand error one second ahead. So I took the mirror's models offline, gave them the first 20 minutes of a signed video to learn from and asked them to predict the last 6.
The four models that learn from the video cut hand error from 67.1 to between 53.0 and 55.7 pixels, about a fifth. The graph neural cellular automaton (NCA) has the lowest average, but its lead over the other three is within the noise of this test. Most of what they learn is where the hands come back to rest. Replaying remembered motion does worse than doing nothing. You can see both in the demo above.
After the hackathon I ran ablations in PyTorch, on this video and on How2Sign, a much larger American Sign Language dataset. Removing the graph's edges changed nothing. Letting a model make several guesses looked like a big gain until I compared it with mean reversion plus a fixed set of offsets, the same at every moment, which got most of the same gain. More data removed overfitting but didn't lift the ceiling. From motion alone, these models predict only a few per cent of the next second beyond the drift back to rest. The details are in the follow-up below.
The setup
The video is a 26-minute British Sign Language translation of the Universal Declaration of Human Rights, signed against a plain backdrop by two signers who alternate by article. For every frame I extract 478 face landmarks and 21 per hand with MediaPipe6, the same models the live mirror uses. The first 20 minutes are the training split. After a 5-second gap, the last 6.1 minutes are the test.
At each test moment, every model sees only the video up to one second before and predicts the pose at that moment, exactly as it would live. I score the hands separately, because face landmarks outnumber hand landmarks 478 to 42 and would otherwise decide every result.
The seven predictors run unchanged from the live mirror. They range from doing nothing at all to a learned neural cellular automaton. Four of them learn from the video: mean reversion, the two PCA models and the graph NCA.
- hold last pose. Assumes you froze: its guess for now is exactly the pose from the past moment. It is the baseline every other model has to beat.
- damped velocity. Takes how fast each point was moving just before the past moment and keeps it going that way, slowing to a stop. Good at finishing a movement already under way, blind to anything new.
- mean reversion. Guesses you will drift back toward your average pose, by an amount learned from how quickly you have returned to it before. The further ahead it looks, the more it bets on your usual resting position.
- nearest-neighbour replay. Searches your history for the two seconds that look most like the two seconds before the past moment, then replays what you did next. It learns nothing and relies on memory alone.
- PCA linear direct. Compresses your whole pose into 16 numbers, then learns one straight-line rule from their recent values to their values one lag later. It refits every two seconds on your own motion.
- PCA linear rollout. Uses the same 16 numbers but learns a rule for just the next tenth of a second, then applies it over and over to reach the lag. Small errors pile up, which in the live mirror usually makes it lose to the direct version. On this video the two finish within half a pixel.
- graph NCA. Each cell is a landmark running the same tiny neural network, and it can only sense itself, its skeleton neighbours and a global average. Trained on your own motion, the cells watch one second of you, then move together to imagine the rest.
The graph NCA
The graph NCA treats the skeleton as a colony of cells. It follows Growing Neural Cellular Automata1 and its extension to graphs2. Each cell is one landmark (48 picked from the face mesh and 21 on each hand), and 102 edges link them along the anatomy. Every cell runs the same tiny neural network, 1,938 parameters in total. A cell can sense only itself, its neighbours on the skeleton and an average over all cells.
To forecast, the cells first watch one second of real motion, which fills their memories, then run freely across the lag. Training compares where they end up with where the landmarks really went. I pretrained it for 3,000 steps on the training split (about six minutes on a laptop CPU) and let it keep learning during the test at the live demo's pace. Each of the face's 478 landmarks moves with its nearest face cell.
Learning beats standing still by a fifth, but no learner clearly wins
Holding the last pose misses by 67.1 pixels on the hands, which is just how far hands move in a second. On the 1,185 moments where every model has hands to score, the four learning models each do 10 to 12 pixels better, and a block bootstrap over 10-second stretches of the test puts every one of those gaps well clear of zero. Between the four, the differences are about a pixel and every interval spans zero. The graph NCA is ahead of the other three by 0.8 to 1.4 pixels, with 95% intervals running from about −3 to +1. This test can't rank them. (The chart and table below average each model over every moment it could score, which is why the NCA's lead looks bigger there.)
Two models are worse than doing nothing. Damped velocity (39% worse) extends motion that has usually reversed, because signing hands change direction faster than once a second. Replay (10% worse), a simple form of motion matching4, commits to one remembered gesture, and when the memory is wrong it is far wrong. With two signers and only minutes of history, exact repeats are rare. On the face, every model lands between 10 and 17 pixels, because the face barely moves.
| predictor | hands, px | vs hold last pose | whole pose, px | face, px | ms per prediction |
|---|---|---|---|---|---|
| graph NCA | 53.0 | 21% better | 13.7 | 11.7 | 7.7 |
| PCA linear rollout | 55.4 | 17% better | 13.7 | 10.5 | 13.1 |
| mean reversion | 55.6 | 17% better | 14.4 | 11.3 | 0.4 |
| PCA linear direct | 55.7 | 17% better | 13.8 | 10.6 | 12.2 |
| hold last pose | 67.1 | baseline | 16.3 | 12.6 | 0.0 |
| nearest-neighbour replay | 73.5 | 10% worse | 18.5 | 14.6 | 1.7 |
| damped velocity | 93.4 | 39% worse | 21.9 | 16.7 | 0.3 |
Over time, every model finds the same moments hard
Errors rise and fall together across models. Fast passages are hard for everyone and calm ones are easy for everyone. If I rank the models within each 10-second stretch, the graph NCA comes first in 18 of 36, more than any other. Mean reversion is first in 9 and the PCA rollout in 5.
The demo shows how they get there. The learning models hedge: their hands drift part of the way back toward the resting position instead of continuing the stroke. That lowers average error, but it produces a softer, slower signer than the real one.
What the mirror can and can't see coming
One second ahead, the predictable part of signing is its envelope: hands return to a resting zone, the face stays put and strokes slow down before they turn. All four learning models capture that, which is worth about a fifth of the error. None of them anticipates what is signed. That would need the language and the content, and 20 minutes of one text is far too little to learn either. This matches what Martinez, Black and Romero found for body motion3: standing still is a strong baseline.
At the hackathon I wrote down three things to try next: shorter horizons, forecasts that make several guesses and are scored by the best one (so hedging stops looking like skill), and conditioning on the text being signed. I tested the second afterwards, along with a set of ablations.
Follow-up: the ablations
I ported the graph NCA to PyTorch (the same 1,938 parameters) to find out where its small lead came from. Each run trains once on the same 20-minute split and is then frozen, with no online learning during the test, and every setting runs with 3 seeds. Scoring is on a 10 Hz grid with 2,421 test moments, so the numbers sit a few pixels away from the table above: holding the last pose scores 63.7 pixels and mean reversion 52.6.
The graph contributes nothing yet
The full model scores 50.6 pixels. Cutting all 102 edges, so each cell sees only itself and the global average, gives 50.9. Removing the global average instead gives 51.3, and removing both gives 51.0. With the full 20 minutes of training data, every NCA variant I tried lands between about 50 and 53 pixels. Training longer (1,500 to 12,000 steps) doesn't help, and neither does more context (I tried 0.3 to 4 s). Bigger networks fit the training split better and the test no better: at 35,466 parameters it scores 47.0 on training and 53.0 on test. Less data does hurt (56.7 pixels with 2.5 minutes). A GRU over the whole pose sequence memorises, at about 23 pixels on the training split and 60 on the test.
Best-of-K gains come mostly from fixed guesses
I gave the NCA K output heads, trained winner-take-all and scored by the best of the K. With 2, 4 and 8 guesses it scores 41.8, 33.8 and 28.1 pixels, against 50.6 for a single guess. That looks like a big gain. The control is the mean-reversion forecast plus K fixed offsets (k-means centres of its errors on the training split), the same K offsets at every moment whatever the signer is doing. It scores 42.5, 36.6 and 30.7. At K = 8 the fixed guesses get within 2.6 pixels of the learned ones, so most of the apparent gain comes from covering the places hands tend to go.
A ranking head trained to pick the winning guess in advance does about as well as always picking the guess that won most often in training: it is right 33% of the time against 33% with 4 guesses, and 22% against 19% with 8. Its committed pick averages about 56 pixels, worse than a single guess. I'd now report any best-of-K number next to a fixed-guess control like this one.
More data removes overfitting, but the ceiling stays
On How2Sign7 (American Sign Language from many signers, with OpenPose keypoints that include the arms) I trained a GRU on the training split, about 20 hours, and tested on the validation split, which uses different videos. Test error falls from 44 pixels with 1/64 of the training data (about 0.3 hours) to 34.8 with all of it, which is only just below mean reversion's 35.5. Holding the last pose scores 37.6. Longer context (0.5 to 3 s) and a bigger GRU (512 units, 35.6 to 36.0) give nothing further, and with 4 ranked guesses the committed pick is worse, at about 37.
From body motion alone, these models predict only a few per cent of where the hands will be one second ahead, beyond drifting back to rest: about 4% better than mean reversion on the British Sign Language video and 2% on How2Sign.
What I'd try next
The text being signed is known, and How2Sign ships translations, so conditioning on the sentence is the direct way to ask whether a mirror can learn what someone is about to say. Shorter horizons, a quarter to half a second, would test whether models can finish a stroke already under way, which is where motion should be most informative. A sweep from 0.2 to 2 seconds would show where that stops.
Method details
Landmarks: MediaPipe Face and Hand Landmarkers (float16 models) on every frame of the 25 fps, 854×480 video, in 8 parallel processes. Each process restarts tracking at its chunk boundary. Split: train 0 to 20 min, 5 s gap, test 20:05 to 26:12. Evaluation: every 5th test frame (1,601 moments), lag 1 s. Each predictor receives only keyframes up to the cutoff. Error is the mean Euclidean distance per landmark in a 640×360 frame. Hand error counts a hand only when both the prediction and the truth contain it. Paired differences use a block bootstrap with 10-second blocks and 4,000 resamples, over the 1,185 moments where every model predicted visible hands.
Predictors keep their live-demo settings. The PCA models fit on the last 3 minutes, with principal components from a randomised SVD5. Mean reversion also fits on the last 3 minutes, and replay searches the last 5. The graph NCA has 8 memory channels and a 32-unit hidden layer, with velocity damping 0.9. It trains with Adam (learning rate 3·10⁻³, batch 8, gradients clipped to norm 1), rollouts of up to ten 100 ms steps, and 9 online steps per second of test video, running on TensorFlow.js's WebAssembly backend.
Code, scripts and raw results, including the follow-up ablations (ablate.py, seq.py,
h2s.py), are in
experiments/sign-offline
on GitHub. The live mirror is at athonmillane.com/mirror.
Acknowledgements and licences
Thanks to Joscha Bach and the California Institute for Machine Consciousness for running the hackathon.
The video is The Universal Declaration Of Human Rights - A British Sign Language Translation by m ross, licensed under CC BY 3.0 (via Wikimedia Commons). The demo above shows a one-minute excerpt, trimmed, scaled to 640 pixels wide and silent, with skeletons drawn over it. Landmarks come from Google's MediaPipe Face and Hand Landmarkers.
The follow-up uses OpenPose keypoints from How2Sign7 by Duarte et al., licensed under CC BY-NC 4.0 for research use. Only error numbers computed from them appear on this page.
References
- Mordvintsev, A., Randazzo, E., Niklasson, E. and Levin, M. Growing Neural Cellular Automata. Distill, 2020.
- Grattarola, D., Livi, L. and Alippi, C. Learning Graph Cellular Automata. NeurIPS, 2021.
- Martinez, J., Black, M. J. and Romero, J. On human motion prediction using recurrent neural networks. CVPR, 2017.
- Clavet, S. Motion Matching and The Road to Next-Gen Animation. GDC, 2016.
- Halko, N., Martinsson, P.-G. and Tropp, J. A. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 2011.
- Lugaresi, C. et al. MediaPipe: A Framework for Building Perception Pipelines. 2019.
- Duarte, A., Palaskar, S., Ventura, L., Ghadiyaram, D., DeHaan, K., Metze, F., Torres, J. and Giró-i-Nieto, X. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. CVPR, 2021.