Nicole Junkermann on Learning by Trial: A Plain Guide to Reinforcement Learning

What the phrase actually means

Reinforcement learning sounds far more forbidding than the idea underneath it. Take the vocabulary away and what is left is ordinary: trying something, finding out how it went, and trying again with that knowledge in hand. Nicole Junkermann keeps returning to it because it is one of the few corners of the subject that can be explained without any jargon at all.

Most of us learned to catch a ball this way. No instruction works. There is a throw, a miss, a small adjustment nobody could put into words, and another throw. The correction arrives as an outcome rather than an explanation, and after a few hundred repetitions something that cannot be written down has been learned anyway.

Learning without being told the answer

The difference from other kinds of machine learning is worth holding onto. Usually a system is shown thousands of correct answers and asked to find the pattern connecting them. Here there are no answers to show. There is a situation, the freedom to act, and a number afterwards saying how well that went.

A higher number suggests the behaviour is worth repeating. A lower one suggests it is not. Run that loop often enough and something appears that nobody wrote down and nobody could easily describe. The system has not been taught. It has practised.

Why so much of it is failure

What the retelling usually leaves out is how much of the process is simply getting it wrong. The overwhelming majority of attempts fail, and they have to, because a system that only ever repeats what already works will never find anything better. Early on the results look worse than useless.

There is no way to skip that stretch, and no reliable way to tell from inside it whether the approach is sound. The only honest response is more attempts, and a willingness to read the record afterwards without flattering it.

• Act, even when there is no reason to think the action is a good one.
• Take the score as it comes, rather than as it was hoped for.
• Change a little, not everything, so it stays clear what the change did.
• Keep the record honest, because the record is the only teacher in the room.

The problem of a late score

A second difficulty makes the patience non-negotiable. The score often arrives long after the decisions that earned it, and the credit has to be spread backwards across many small choices, most of which were neither good nor bad on their own.

Working out which parts of a long sequence deserve the credit is genuinely hard, and it is hard for people too. Anyone who has tried to decide which of a year's decisions produced a good year has met the same difficulty, which is part of why a written record matters so much in the writing kept here.

What it is not

It is worth being plain about the limits. None of this says what such systems will be able to do, or when. It is a description of a method, not a promise and not a forecast. What the method teaches is older than any of it: a run of failures is sometimes the only available route, and the people who can sit calmly with that are rare. The same thought runs through the note on the small changes worth welcoming, and through the slower habits the reading pages argue for.