Building the game: iterative learning for deterministic robot programs

06 Oct, 2026

In my last post, I shared a digital twin with a short list of rules that must never break. This post goes one step further. What happens when those rules become a game, and a computer learns to play it? Can an algorithm skilled this way produce deterministic and repeatable code? How can generative AI be used wisely towards this purpose? I explore these in this blog.

Over the last month I worked on two projects that let a computer write robot programs. The first one trains a small model from zero. Its only teacher is a physics simulator. This is more like AlphaZero without a game. The second one uses a large language model that already exists. It can write programs for industrial robots, for example, and the robot's own software checks every program on a simulated controller on my PC. This is more like AlphaProof. As before, I built both by directing an AI coding agent (Claude Code). I set the rules and made the calls.

Both projects offered the same insight. The model was the smallest part of the work. Most of the work went into the game.

AlphaGo learned to play Go better than any person. But it did not start from nothing. It got the game first: a board, the rules, a referee that never makes a mistake, and a clear result at the end. Win or lose. Robot programming does not come with a game. Nobody gives you the board, the rules or the score. You have to build them.

So the learning happens at two nodes. First, the person learns how to build the game. Then the computer learns how to play it.

Is this really new?

Short answer is a 'no'. A computer that learns inside a game with exact rules is the whole idea behind AlphaGo and AlphaZero [1, 2]. A model that writes an answer and lets a strict checker judge it is how AlphaProof does mathematics [3]. Coding models that write many programs and keep the ones that pass the tests are also not new [4].

But there is a difference in where the effort goes. A model like AlphaCode first learns from a very large amount of code (pretraining). Then it gets more training for the task (fine tuning). This blog puts the effort into the game itself. In the second project, nobody trains the model at all. The referee's answers teach the computer how to write correct code.

Two older ideas are also close, and it helps to say how this work is different.

The first is reinforcement learning with ready-made environments [5, 6]. OpenAI's Gym and Roboschool gave researchers simulated robots to train on. The environment came ready. The job was to find a good control strategy inside it. In my approach, the environment is not given. I build the rules and the environment together, and each failure can change both.

The second is model reference adaptive control (MRAC) [7]. Here, a controller adjusts itself so that a machine follows a reference model. But you must know the 'model' machine (the reference 'plant') well enough to write that model first. In my approach, I do not start with a known plant. In a way, I build the plant itself. The moves, the referee and the score are the thing the computer learns against. The intended outcome is robot code.

The claim is narrow. For robot programs, the game is not given. Somebody must build it from manuals, simulators and failures. That building is slow, and I think it is the real product. The model that plays inside the game can change. The game stays. And all this with a focus on industrial robotics. I am using the same tools as shared in my previous blogs.

What a game needs

A game that a computer can learn from needs four parts:

1. Legal moves. What the player can do.
2. A referee. Who decides if a move worked.
3. A score. How good the result is.
4. A supply of games. Something to practice on.

Go gives you all four. In industrial robotics, you must provide for each one yourself. Here is the overall idea of what this framework looks like with suggested tech stack.

The moves

The first decision was the most important one. The computer does not write program text.

A robot program is text. It is easy to let a model write text. It is also easy for that text to be wrong in a thousand small ways: a missing comma, a word the robot does not know, a command in the wrong place.

So I made the moves smaller. In the first project, the computer chooses from a short list of actions: move above an object, go down, grip, lift, release. In the second project, the computer fills in a structured form. Each line of the form must match a command from the robot maker's manual, with the page it came from. A separate program, namely the composer, then turns the form into the final text. That composer program never guesses.

Composer is the first entity that makes the framework deterministic.

This makes the space small enough to check. Every move the computer can make is a move the referee can judge.

There is a trap here. It is tempting to block every bad move in advance. For example, why allow "release" when the gripper is high in the air? The object will fall.

I did not block it. The list of moves says what is possible, not what is good. If I block "release at height", the computer never learns why it is a bad idea. The rule moves into my code and out of its learning. Good and bad must come from the score, not from the list of moves.

The referee

The referee must be exact. It must give the same answer every time.

Referee is the second entity that makes the framework deterministic.

In the first project, the referee is a physics simulator. It drops the box, slides it, and tells you where it stopped. It does this in five slightly different worlds, with different friction, weight and start positions. The computer sees three of these worlds. The other two stay hidden. A program passes only when it works in all five. This stops the computer from learning one world by heart.

In the second project, the referee is the robot's Digital Twin (think of MuJoCo, Isaac Sim or ABB Robot Studio). It reads the program, links it and runs it on a simulated robot. It does not care what the manual says. It only cares what the robot does. Each time, the referee was right. So the order of trust became clear: the referee is above the rule book. The manual is a good first guess. The robot physics is the truth.

The score

The score is the part that breaks most often. An adaptive algorithm is very good at finding the cheapest way to get points.

The second project uses steps instead of points:

1. The program follows the manual.
2. The robot's software accepts it.
3. It runs on the simulated (or Twin) robot without errors.
4. A human expert watches it and says "yes, this is right".

The last step is important. A program can pass every check and still do the wrong thing for the cell. Only a person can judge that. The computer can climb to step 3. Only a person can give step 4. Seen from the Reinforcement Learning viewpoint, the code generator learns from human experience.

A supply of games

Go can play itself millions of times. Large physical industrial Robots cannot afford this. Each run takes time, and on real hardware each run has risk. So you must make practice games cheaply.

The first project uses a simple trick. Write a random program. Run it in the simulator. Look at where the boxes ended up. Now you have a solved task, for example, "put the boxes here", with a program that does it. About 4 in 100 random programs give a useful task. That is enough to train on, and no person had to write an example.

The second project makes its practice from the manual. For every command, it writes a small test program that should pass, and some that should fail. Then it asks the robot's software. About 1,000 commands went through this check. Each one now carries a label: accepted, needs an extra option, or rejected.

Two learners, one loop

There are two learners here, and they learn in different ways.

The person learns to build the game. Each failure teaches one rule. I write the failure down, find the cause and add a check that catches it next time. This is not trial and error. These failure sit at the boundary between the known and the unknown due to lack of exact match between reality and digital twin. The new check runs earlier and costs less than the failure did. I never remove a rule without knowing which failure it came from.

The computer learns to play. In the first project, it learns in the usual AlphaZero way, except the game is given by human. It plays the practice games, and its small model gets better at choosing moves. With that model to guide the search, it solved 166 of 200 tasks it had never seen. The same search without the model solved none. Random programs at the same cost also solved none.

In the second project, the model does not change at all. It is a large model that I cannot train. But the results still get better over time, because the game gets stricter. Each mistake that the robot's software catches becomes a check that runs before the robot sees the next program. That mistake never reaches the robot again. In the last test, ten of ten programs passed on the first attempt.

So the two learners feed each other: The person makes the game stricter. A stricter game makes the computer better, with or without new training. The computer's new mistakes show the person where the game is still weak.

Debatably, this is Reinforcement Learning with Human Feedback (RLHF) in some way, except the game and the rules are in human control not some large language model.

The collaborative learning is the third entity that makes the framework deterministic.

What I would tell someone who starts this work

• Build the referee first, before any model and before any training. If you cannot judge a program, the algorithm cannot learn from it.
• Make the moves small and typed. Do not let the computer write free text that a machine is left to trust.
• Keep good and bad out of the move list. Put them in the score.
• Give zero for doing nothing. And confirm it.
• Hide some worlds when you build the game. A result that works only in the worlds the computer saw is not a result. Basic but good to remember.
• Write down every failure and the rule it gave you. That list is the real product.
• Human has to have the ultimate judgment call, for large machines anyway. The computer can prove that a program runs. Only a person can say that it does the right job before anything hits the floor.

Also use of an LLM is advised for automating and heavy lifting of building the framework, not for program synthesis.

It is tempting to generalize this exercise for all physical worlds governed by deterministic laws. However, appropriate games must be orchestrated.

References

1. D. Silver et al., Mastering the game of Go with deep neural networks and tree search, Nature, 2016 (AlphaGo).
2. D. Silver et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, Science, 2018 (AlphaZero).
3. T. Hubert et al., Olympiad-level formal mathematical reasoning with reinforcement learning, Nature, published 12 November 2025 (AlphaProof).
4. Y. Li et al., Competition-level code generation with AlphaCode, Science, 2022.
5. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.
6. OpenAI, Roboschool: Open-source software for robot simulation, integrated with OpenAI Gym, 2017.
7. K. J. Åström and B. Wittenmark, Adaptive Control, 2nd edition, Addison-Wesley, 1995.