Once upon a time I was a flashcard freak. I spent a decade hoarding facts and ideas into a collection of 30,000 cards that I reviewed obsessively with spaced repetition algorithms. Then I spent another three years recording podcasts about it.

We called the podcast “Golden Nuggets” because a recurring topic of conversation was the problem of knowledge valuation - how should you weigh the expected value of a flashcard you are considering adding to your collection?

I learned that people are as varied in their taste in flashcards as they are in their taste in music. Some like to extract insights from the biographies of great historical figures, others prefer the foundations of mathematics. Some are highly selective, even delicate connoisseurs of perfectly packaged factoids, while others prefer a more evolutionary approach, creating hundreds of flashcards in a sitting and letting future review sessions perform the Darwinian filter.

It’s difficult to know in advance how useful a particular flashcard will end up being. Scrolling back with the benefit of hindsight I can make rough judgements about which cards were worth creating: from the German vocabulary cards I never used and the extracts from random Wikipedia articles that ended up orphaned from the rest of my collection, to the programming cards that helped me find my first job.

What if I had been able to see every counterfactual world in which I added or removed a particular flashcard? What if I could have tried every possible combination and chosen between them based on how my life actually turned out? It would surely have told me something about what kind of knowledge I ought to value.

There’s a wonderful paper from 2021 called DreamCoder where the authors built a system that does something analogous to this, but with programs instead of flashcards. Rather than deciding in advance which concepts are worth knowing, the system discovers their value by testing them across thousands of problems and seeing which ones keep proving useful.

DreamCoder uses a so-called “wake-sleep” algorithm. During the waking phase, given a handful of input-output examples like [3,1,2] → [1,2,3], it searches over combinations of tokens in a simple functional programming language until it finds an executable program that can successfully transform all of the inputs to the outputs.

During the “sleep phase”, DreamCoder then refactors solutions to the problems it solved, abstracting out reusable functions into a shared library of concepts. The guiding principle for the refactoring is compression: a concept earns its place in the library if it makes solutions shorter across the entire corpus of tasks it was trained on.

Once DreamCoder has built a library, it can search over combinations of library functions rather than raw primitives. Since library functions are higher-level, solutions expressed in them are much shorter, and shorter solutions are exponentially easier to find. A program to sort a list expressed in raw primitives requires 32 function calls and would take 10^73 years to find by brute force, but expressed using the learned library it requires only 5 function calls and is found in under 10 minutes.

I realised when re-reading the paper recently it provides an interesting perspective on my question. It suggests that one of the ways you can value a golden nugget is in how well it reduces the search space for problems you haven't encountered yet. This would allow you to compare the value of pieces of knowledge - how much harder do remaining problems become when you remove it from your library?

What excites me is that LLMs allow us to run DreamCoder-style experiments over anything representable as text - not just programs, but theories, explanations, historical narratives, scientific concepts. We could investigate how individual ideas affect the search space for problems we haven’t yet encountered.

Take for instance DreamCoder’s “growing a language for vector calculus and physics” experiment:

It reminds me of what Demis Hassabis has called the “Einstein test” for AGI - can an AI system trained only on data up to 1900s rediscover special relativity?

What if we inverted that experiment? Give the system all the conceptual ingredients Einstein needed and remove them one-by-one. Which are accidents of how Einstein happened to think about the problem? Could a system reconstruct the same physics from a different set of primitives entirely?

Beyond science, what if we could simulate the conditions of ancient Athens - a city of perhaps 100,000 people where the density of interesting minds per acre of the Agora may have been the highest in history - and test which ingredients were essential to that explosion of ideas?

Might it be possible to turn epistemology into a science?

There wasn’t time for a proper debate at the event, so I’ve tried to steelman Lee’s argument, which I believe centres around search, possibility, and probability. I think Lee is arguing from a Deutschian perspective and have interpreted his claims through that lens - but I’m no expert on the philosophy of science, so corrections are welcome.

Search and Learning

One point that Lee made during the event was that the space of chemical reactions is too large to search, and that search alone is not enough to create an AI scientist.

Let’s compare the search space of chess to chemical reaction space. In chess, each turn you have a choice of around 38 moves on average. For every one of those, your opponent has ~38 responses. This gives a game tree with a branching factor of 38 and roughly 10^40 legal game states which is more than the number of stars in the observable universe.

Chess-playing AI engines overcome this by using a trained approximator (eg. a neural network) to recognise patterns similar to what they saw during training, combined with multi-step lookahead search to correct for the data distribution shift that arises from never having seen the exact position before. It's very unlikely you'll play the same chess game twice, but the trained approximator is still useful because chess games share recurring structural features (forks, pins, open files, king safety patterns) that generalise across games.

Chemistry has a far larger search space. The “board” is all possible molecules; the “moves” are reactions that transform one into another. Even a small number of starting materials produces an enormous combinatorial explosion: 18 reagents yield ~1,000 possible reaction combinations; allow multi-step synthesis with varying conditions and that number quickly exceeds 10 billion from fewer than a dozen inputs. And this is still tiny - the space of all possible molecules up to 30 atoms using just carbon, oxygen, nitrogen, and sulphur is estimated at 10^60.

Lee’s group has published extensively on using search algorithms to prune down the enormous chemical search space.1234567 So in this respect Lee is clearly not anti-search, in fact his approach of scaling autonomous chemistry labs and performing high throughput search appears consistent with the mainstream “bitter lesson-pilled” view of search and scaling in AI.

So why shouldn’t Lee be excited about scaling search and learning to achieve superhuman AI chemists the same way we achieved superhuman AI chess players? After all, both chess and chemistry have search spaces too large to enumerate, but also learnable structures and regularities that an AI can exploit to guide its search efficiently. Shouldn't we expect the same approach to work for chemistry?

I found the clearest articulation of the Cronin Group's objection in this article from Lee’s collaborator Sara Walker.

Chemical space cannot be computed, nor can the full space be experimentally explored, making probability assignments across all molecules not only impossible but unphysical; there will always be structure outside our models which could be a source for novelty. - Why the Physics Underlying Life is Fundamental and Computation is Not by Sara Walker

Sara’s argument is that chemistry has an open-endedness that chess does not - new molecules can create new conditions (catalysts, environments, physical contexts) that enable reactions that were impossible before those molecules existed. A novel catalyst could open up a region of reaction space that has no structural similarity to anything in the training data, because it didn’t exist as a possibility before.

Imagine a variant of chess where pieces that reach certain configurations can combine to create entirely new piece types with movement rules that didn’t exist at the start of the game. Your neural network might still handle the opening - but mid-game, players start producing pieces it’s never seen, with abilities that couldn’t have been predicted from the starting rules. You can’t assign a probability to a threat from a piece type that doesn’t exist yet.

The Walker/Cronin claim, as I understand it, is an argument about the difference between probability and possibility. A probabilistic model assigns likelihoods over a space defined by what it's seen - it can interpolate within that space and extrapolate along familiar dimensions - but it can't represent possibilities that don't yet exist in any form in its training data. And for Lee and Sara, it's precisely those new possibilities that constitute true scientific discoveries.

Possibility and Probability

If the open-endedness of chemical space is a constraint on every agent navigating it, what is it that human scientists supposedly do that an AI can’t? From Lee’s tweets, I get the sense that his answer is: to make genuine discoveries, you need to be able to imagine completely new possibilities - to invent a possible world, not just search within the existing one.

On the surface, LLMs can already do this - you can easily get them to propose hypotheses, design experiments, and analyse results autonomously. But Lee’s deeper claim, I think, is that truly creative hypothesis generation requires reasoning over logically possible worlds, not merely probable ones.

To create a chess-playing AI engine, humans provide the rules of chess and the model teaches itself to play at a superhuman level through self-play. But we’d never expect it to invent anything outside the sandbox universe we created for it. It can’t propose a new game, or suggest a rule change to make chess more interesting. It is, in Popper’s terms, an induction machine:

…we may consider the idea of building an induction machine. Placed in a simplified “world” such a machine may through repetition “learn”, or even “formulate”, laws of succession which hold in its “world”… In constructing an induction machine we, the architects of the machine, must decide a priori what constitutes its “world”; what things are to be taken as similar or equal; and what kind of “laws” we wish the machine to be able to “discover” in its “world”. In other words we must build into the machine a framework determining what is relevant or interesting in its world: the machine will have its “inborn” selection principles. The problems of similarity will have been solved for it by its makers who thus have interpreted the “world” for the machine. - Conjectures and Refutations by Karl Popper

The real question for AI science is whether the same constraint applies to frontier language models. After all, their “world” and “inborn selection principles” are much more general, having been pre-trained on essentially the sum of human declarative knowledge. Are they ultimately bounded by those inborn facts and rules, or are they capable of imagining possible worlds beyond them?

Consider Demis Hassabis’ “Einstein test”: train a foundation model with a knowledge cutoff at 1911 and see if it could produce general relativity. A model trained on pre-1911 data would have learned that Newtonian mechanics was one of the most successful theories in history. Every well-designed experiment had corroborated it. A probabilistic reasoner would have assigned near-certainty to Newton being correct. It wasn’t until 1919, when Eddington measured starlight bending near the sun at roughly double the Newtonian prediction, that Einstein’s theory was accepted - despite directly contradicting centuries of established mechanics.

The real paradox is that Newtonian mechanics was shown to be false at precisely the point when a probabilistic model would have been most confident it was true. New scientific theories are, almost by definition, improbable or even absurd by the standards of incumbent theories.

Beren Millidge makes the general version of this argument in machine learning terms:

The reason scaling pretraining (aka unsupervised learning on a fixed corpus of e.g. web data) does not scale to AGI or imply omniscience is pretty simple. The object that pretraining optimizes is the approximation to the true posterior over token sequences taken from the internet. Better scaling means better approximating the distribution of common-crawl. The distribution of common-crawl does not contain superintelligent behaviour and hence scaling alone will not reach it… To see this more clearly, let’s go back to the example of prompting the pretrained model with ‘The solution to alignment is: ‘. Suppose the model somehow magically generalized deeply and actually knew the solution to alignment. Even if this was the case, actually completing this prompt with the solution to alignment would be incredibly unlikely for the model…. - The Limit of Prediction is not Omniscience by Beren Millidge

Even if a model somehow "knew" a revolutionary new theory, producing it would be incredibly unlikely under a distribution trained on a world where that theory doesn't yet exist. That, I think, is what Lee is getting at by pointing at the asymmetry between probability and possibility: probability tells you what's likely within a known world, only possibility can allow you to invent a new one.

I’ve tried to steelman Lee’s argument, and I think it raises genuine philosophical questions that the AI scientist teams haven’t fully grappled with: can a system trained on existing knowledge propose possibilities that lie outside of its inborn facts and rules, or is it fundamentally limited to recombining what it’s already seen? I don’t know the answer. Whether this philosophical distinction has practical consequences, or whether it describes a limit that engineering will route around, remains to be seen. My view nowadays is that I’ve been humbled (or “bitter lessoned”) over and over again by asserting on lofty philosophical grounds that scaled search and learning won’t smash through whatever capability we set beyond the reach of AI models, but I still find the arguments and paradoxes interesting. I'd love to be a part of a more structured debate between Lee and a representative from the AI scientist teams to explore these disagreements further.

After spending the first half of the year teaching myself maths, grinding flashcards, attempting machine learning bounties and various LLM experiments, I decided I needed to change my strategy. I was quite miserable that I hadn’t really achieved any concrete “wins”.

Had I persisted with the maths courses would I feel better off now? Certainly I would have gotten some grades or some other form of tangible feedback from the courses or textbooks to tell me that I was making progress. But progress towards what? I want to be able to solve difficult problems, create useful ideas, become more technical, build a company, do something interesting. - Progress (or lack thereof)

After finishing that post I took a break and decided that my main priority was to get out of Oxford where I had been living for the last three years. Despite my best efforts I hadn’t been able to find or create a social group. It basically just felt fucking depressing living there.

I’ve learned it’s just extremely mentally taxing to something like this by yourself. Starting a small Discord community with friends I met from Twitter and other corners of the internet certainly helped, but I think I need to find similar people IRL, ideally in London where I intend to move very soon.

During the period where I was viewing flats and preparing to move I randomly met a few biology PhDs. They were complaining about wet lab work. They told me they spend hours per day manually moving small volumes of liquids between test tubes using pipettes and often have to babysit experiments on the weekends. I even met a biologist who had developed repetitive strain injury in their thumb from pipetting so much!

They said that the equipment for automating pipetting is very expensive. A Hamilton (as much as $2 million) or Teccan (~$600k) pipetting robot costs well beyond the budget of all but the most well resourced labs. Even a more reasonably priced Opentrons Flex (£60k with all the bells and whistles) is much more expensive than it should be. And this is for a machine that automates just a single operation in a biologist’s workflow.

It started to seem like a promising problem to work on. While I didn’t know much about hardware or electronics the one lesson I did successfully internalise over the last 6 months is that LLMs have made it possible to learn anything. So I started teaching myself the basics of electronics and microcontrollers using a cheap kit from AliExpress and getting GPT to give me little challenges.

I got the hang of 3D design by drawing over reference designs and making small tweaks.

When I moved to London, I moved into a tiny room in a seven person EA house share and turned it into mini factory for hardware projects. My housemate kindly leant me his 3D printer.

By this point I had taught myself enough 3D design, electronics and soldering that I was actually able to build some interesting pieces of equipment. I wrote up a quick post brainstorming ideas for a cheap, automated DIY Bio Lab.

Less than 300 people read that post, but out of the blue someone I’d met at a reading group in London messaged me asking to collaborate.

I started working with them in their engineering lab every day hacking on 3D printable automated lab equipment. Since then we’ve won a robotics hackathon, built prototypes for three pieces of equipment and made a lot of progress towards getting our work in the hands of real researchers.

It’s quite likely that a few of the parts born in my bedroom are going to be used by actual scientists to do real research! That’s pretty hilarious to me.

Returning to the question I asked myself six months ago:

Had I persisted with the maths courses would I feel better off now?

No. The most promising opportunities arise from working on interesting problems, sharing what you made with the world and finding like-minded people to work with. This is the second time I’ve done a year of open-ended search to find what I want to do and it’s exactly the same lesson I learned the first time.

Summary of learnings:

• “People, ideas, machines - in that order!” is literally true in my experience.

• Posting online is valuable even if barely anyone reads your posts.

• Upside from working on interesting problems is much higher than courses.

Can we improve upon the humble boiled egg recipe using precision cooking to guarantee ourselves perfect eggs every time without investing in an overpriced laboratory-grade heat plate?

Egg Science

Egg cooks are challenged by a two-phase structure: albumen and yolk require two cooking temperatures…. the white and the yolk contain different blends of proteins, so the white gels at a higher temperature and a different rate than the yolk does - Periodic cooking of eggs

A popular recipe in restaurants that accounts for this is the so-called sous vide egg. The egg is immersed in water maintained at a precise temperature point (generally between 60 and 70°C) for one hour. The result is an egg where both whites and yolk have similar consistencies. My challenge is to create a device capable of cooking such an egg for the lowest possible cost.

Ingredients

Laboratory-grade temperature controlled heat plates are expensive. But the components required to build one are actually very cheap. For the heat source, I settled on this desoldering plate from AliExpress because it’s cheap, heats rapidly and requires zero drilling or assembly.

While it doesn’t have built-in temperature control, this can be added manually using a solid state relay which pulses power into the heat plate based on signals from an ESP32 microcontroller.

Finally, I wanted an automated stirrer for even heat distribution, so I placed a small PC fan under the hot plate to spin magnets to create a vortex inside the beaker.

Visually it looks like a mess, but all of the electronics can easily be packaged away inside a 3D printed box, and thanks to the shockingly low cost of electronics on AliExpress, the total cost is less than £20!

Assembly

During my electronics and hardware meet-up we made a first attempt at assembling the circuit. While we were easily able to get the temperature sensors working, controlling the heater turned out to be much more difficult.

The issue is that the heater can reach 260 degrees in under 30 seconds. When I ran it in my room, it melted the the kapton tape I’d used to strap the temperature sensor to the hotplate and I had to evacuate my room until the toxic plastic fumes cleared…

The way this is handled in industrial applications is by using a PID loop which constantly compares the target temperature to the actual temperature, and adjusts the heater power in real-time to control the heating process.

After integrating a PID auto-tuning library, heating the water bath was much more controllable. It’s very convenient because it auto adjusts its parameters based on the volume of water, the number of eggs and the material of the container.

Testing

The heat plate is nicely calibrated. It can easily stay within +-0.5 degrees celsius of the target temperature. It was quite slow to heat up though, taking around an hour minutes to heat 300ml of water to 65 degrees. Also when I placed the egg inside the heat bath the temperature dropped 5 degrees and it took some time to recover to 65. These issues were both because I constrained the heater’s max power output as a safety precaution.

Here’s the cooked egg. It tasted pretty good!

Next Steps

The egg experiment was fun, but I’m not really interested in using this to cook eggs. My real goal is to use the heat plate as one component of an automated fluidic system capable of automating basic biology experiments, with my first goal being automated gradient PCR.

Precise temperature-controlled heat plates with magnetic stirrers are essential for this - I will need three of them set to precise temperatures to perform the thermo-cycling steps of PCR. Later I may use it for other purposes like incubation.

There are many quality of life improvements I’d like to add:

• Solder and wire the circuit into a PCB.

• Make a 3D printed case to hold the system together and hide away the cables.

• Add a mini screen to track the temperature on the heat plate.

• Wire up circuitry to power the ESP32, fan and heater from the same power source.

• Better code to give me the confidence that it can run unattended without burning the house down.

• Open source list of equipment, assembly instructions, software and CAD designs so anyone can experience the joy of perfectly cooked eggs!

The next component I’m planning to work on is a thermal flow meter to non-invasively track the flow rate of fluid through capillary tubing.

If you are in London and want to help me build it, I started an electronics and hardware meet-up where I’ve been working on projects like this with friends. Here’s a video from a session where we built a syringe pump:

If that looks fun, DM me on Twitter and I will send you the details!

I’ve written a simple guide with video on how to get productive with electronics as fast as possible. Between this and ChatGPT, I think you can easily know everything you need to work on an interesting project in a weekend, even assuming zero coding or existing electronics knowledge.

I’ll tell you what to buy, teach you basic electronics and programming through six mini projects, then show you how to use AI to plan and buy parts for more complicated projects.

Equipment

The first thing you need is a basic starter kit. Something that looks like this picture.

I suggest buying a minimalist one because these kits typically contain a bunch of components you will never use. To learn the fundamentals of electronics make sure you get the following components (I included example Amazon links at the end):

1. ESP32

The ESP32 is a cheap ~$5 microcontroller (small computer) that can control and read data from hardware components connected to its pins. Programming them is easy, and we’ll use an AI programming environment to make it even easier. Note that there are many variations of the ESP32 board. I bought the ESP32 WROOM-32 30 pin version with a USB type C power source. You can buy starter kits that come with a microcontroller board or just the components by themselves.

2. LEDs, buttons, a potentiometer and 220 Ohm resistors

More involved projects build off of the simple exercises you can do with these parts. As mentioned above, you can buy starter kits that include all of them at once.

3. Male to male jumper wires

These are used to connect the ESP32’s pins to your components to create circuits. They will be included in most starter kits.

4. A breadboard

This is a rectangular board with slots for jumper wires and components. Every starter kit will include one of these.

5. Breakout Board

An annoying thing about most breadboards is that when the ESP32 is inserted you can only access one row of pins along the side. The breakout board lets you easily plug jumper wires into the slots around the full perimeter of the board. Make sure you buy a breakout board for an ESP32 with the correct number of pins (eg. 30 pins). Note that this generally doesn’t come with most starter kits. You will likely need to order it separately.

Example Amazon Links

• A starter kit (doesn’t include an ESP32)

• An ESP32 and breakout board

• Total cost: ~24 GBP

Practical Guide

Here are six projects that require hardly any components but teach you the widest range of useful concepts. For the first three projects, this is the workflow I suggest copying and pasting this prompt into an LLM followed by the project title and goal:

“You are a tutor that always responds in the Socratic style. You *never* give me the answer, but always try to just ask the right question to get me to learn and think for myself. You should break down the problem into simple parts. *Ask one question at a time.* Never give me the answer. I have access to an ESP32, breadboard and a basic electronics starter kit.”

The AI will guide you through the project while getting you to work out the answers for yourself. This is more effective than just being directly given the answer.

Here’s a quick video showing what I mean:

Project 1: Basic LED Circuit

Goal: Create a circuit to turn on an LED. No GPIO pins, just a basic circuit.

Project 2: LED with Button

Goal: Create a circuit to turn on an LED when a button is pressed. No GPIO pins, just a basic circuit.

For the next project, try to look up the component online - search something like “potentiometer pinout” to figure out what role each pin plays. If there’s a name or identifier on your specific potentiometer try including that in the search query to get better results.

Project 3: LED with potentiometer

Goal: Create a dimmable LED using a potentiometer. No GPIO pins, just a basic circuit.

That’s the first three projects done! Consider doing them once more from memory to double check you understood them.

Now you are ready to start working on some projects involving programming. Instead of using the 3V3 pin, which outputs a constant voltage, you will use GPIO (general purpose input/output) pins that allow you to programmatically control the voltage, letting you control components like LEDs through code.

Coding Setup

We’ll use a programming language called C++ with a framework called PlatformIO to write code for the ESP32 (I tried alternatives, they all sucked). If you don’t know how to program, don’t panic, AI will help you bridge the gap.

1. Download and sign up for Cursor. Cursor is an AI code editor with a free two week trial. It’s the development environment that many professional software developers are switching to.

2. Install the PlatformIO extension

3. Wait for things to install

Here’s a short video showing the setup process:

And another video showing the workflow:

Project 4: LED Blinker

Goal: Use an ESP32 to blink an LED.

For the final two projects, I prepared some code you can copy to quickly add a web socket server to any ESP32 project. Web sockets allow you to communicate back and forth seamlessly between your laptop and your ESP32 through WiFi. Please see this video to understand how it works.

Here is the link to the web server template code.

Project 5: ESP32 WiFi LED controller

Goal: Use ESP32 to run a web socket server. Connect to the server from a web browser and click a button to turn an LED on and off.

Project 6: ESP32 WiFi LED Slider

Goal: Connect to the server from a web browser and slide a range slider to control the brightness of an LED.

That’s really all you need to know to be productive! You can probably already think of your own simple projects to extend these basic ones. The only additional skill you need is to lookup the data sheets / online guides for parts you buy which will explain how they should be wired up, power supply requirements etc.

Planning Projects and Sourcing Parts

Given an idea for something you want to build, the first thing you’ll want to do is to plan it out with GPT. Question all of the components and make sure you understand the purpose behind everything GPT tells you to buy.

When buying parts, remember that you can always screenshot the product page of a part you want to order and get GPT to double check whether it’s correct according to your plan. Sometimes it’s really hard to tell by yourself, especially for parts with inscrutable names.

And of course when wiring things together, you can get GPT to check the wiring is correct:

And it can suggest ways to check the circuit is correct with a multimeter before you turn it on:

In terms of places to buy from, Amazon is the most expensive but has the fastest shipping. You generally don’t need to worry about bundling everything into one order to hit a free shipping limit.

AliExpress (Chinese version of Amazon) has a good balance between price and shipping times. In the UK you only need to spend 8 pounds to get free shipping. Things seem to take 1-2 weeks to arrive. This is where I would aim to buy most of your kit to keep costs down.

Alibaba caters more towards businesses. I personally haven’t used it yet. The shipping times seem to be longer and there are minimum order quantities, but if you are buying in bulk it’s the cheapest option - 10-100x cheaper than Amazon depending on what you’re buying.

For components like resistors, diodes, MOSFETs and other generic circuit components I used Mouser. People also recommend DigiKey. Both of them have a 33 GBP minimum spend to hit the free shipping price so you should probably try to bundle as many orders as possible into one basket. I found Mouser’s UI to be confusing as hell. I heavily relied on AI to order these parts which worked fine - everything I ordered works perfectly so far.

Next Steps

If you enjoyed this, you should know that I started a little group in London to work on hardware and electronics projects like this. Read this, and send me a message if interested! Here’s what you can expect:

People

History’s most productive collaborative circles, third spaces and decentralised online networks have demonstrated the outsized impact creative groups can have on human progress.

Like the Ancient Greek agora - a small, densely concentrated marketplace for free exchange of ideas whose intellectual achievements have shaped the world for millennia.

Isn’t it ironic that these social scenes are so rare when the ingredients are so simple: a group of motivated people with a space to regularly meet?

Problems

Over the past two centuries the world has improved dramatically, but there are countless problems left to solve - the world can be much better.

How should you decide what to work on? The best approach might not be through existing philosophies and frameworks, but a peer group with a space to argue, debate, and refine your sense of what truly matters.

The best we can hope for is to fall in love with a problem that brings life meaning and find others who share that passion.

Agency

Progress is not inevitable. You cannot rely on others to solve the problems you think are most important.

Fortunately, you live in the best time in history to tackle them. You have a PhD in your pocket. Soon you’ll have small armies of intelligent agents at your disposal.

Increasingly the limiting factor to progress is not rare, exceptional talent, but the agency, will and determination to try.

Experimental Labs is a community for motivated individuals looking for a group to work on problems that matter. If that sounds like you, join us at our next meeting! Here’s what to expect:

Send me a message on Twitter for details.

I stayed at my sister’s place in London a few times over the past couple of weeks to meet people, go to some meet-ups and search for a place to move. There happened to be a couple of biology PhDs staying at their place while I was there and we randomly got chatting about wet lab work. They said a few things that surprised me:

• Bio experiments involve a ton of manual labor and often fail for unknown reasons

• Daily work involves manually running experiments over and over

• Extremely expensive equipment even for simple things like hotplates

• It’s hard to replicate anything - everyone uses different equipment

• Experiment protocols involve a lot of inexplicit tacit knowledge

See this bio experiment POV for example:

The workflows are surprisingly primitive. An army of PhDs just shuffling test tubes between different stations, constant exposure of reagents and lab equipment to contaminants in the environment and a lot of random heuristics in the protocols.

21st Century DIY Bio Lab

Since I’ve been searching for problems recently, I started to think about how we might redesign bio labs for the 21st Century. The most obvious approach would be to use robots instead of humans that could be programmed to march around and perform the experiments exactly the same way each time. Eg. imagine a robotic arm on a track which could slide around and visit different stations carrying test tubes back and forth.

Performing an experiment would be essentially like executing a program (a sequence of robotic operations) over the same hardware (bio lab equipment). The PhDs would be freed from their manual labor and could focus on more interesting things like proposing new ideas for experiments.

But the cost barrier to entry for this is pretty damn steep. I can hardly afford a single PCR machine, let alone a robotic arm with the level of precision required to operate lab equipment designed for humans!

So I came up with a much simpler/stupider idea - why can’t you just flow stuff through networks of pipes controlled by programmatic syringe pumps and valves? This is already a thing in chemistry (flow chemistry) but it doesn’t seem to really be used in biology. Or at least not in a typical lab.

There are many good things about this approach - it’s more automatable, contamination risk is much lower and everything can be programatically operated with cheap microcontrollers.

You would have to build a bunch of DIY hardware from scratch around this approach. But if you are a hobbyist like me, you are basically forced to do this anyway because traditional equipment is so extortionately expensive - why not make it modern and automatable in the process?

Here’s some more justificatory storytelling about why this is a good idea (take it with an enormous grain of salt):

Cheap Equipment

Improvements in 3d printing (higher precision, ability to print scientific/medical quality materials like PEEK) mean that bio equipment can (in principle) be manufactured extremely cheaply. There are already 3D-printable designs for things like syringe pumps that cost < $40 per pump compared to $2k+ commercial models. If you search DIY <name of biology equipment> you can generally find some attempt online to create a cheap version of the real thing from affordable parts. The main thing that’s missing is to design around automation - which I argue is made significantly easier and cheaper by flowing reactants through tubes - and to write some decent software to glue everything together.

Open Source

Everything could be open source - software could be written to make it literally effortless to set up the equipment even without electrical engineering or programming skills (LLMs mean we are already 80% of the way there). Hardware sheets, automation software and experiment circuit diagrams could be shared and collectively improved by the community.

This would create a huge amount of social value and would be an interesting stepping stone towards new ways of working in bio. It might make bio accessible to more people, especially to bored software engineers looking for a new hobby (like me). Maybe we’ll see more independent bio researchers and startups who can set up their own automated labs outside of academia for < $500 and an increase in the number of undirected tinkerers who can take risks with less capital? Think about how much progress in CS and tech came from having low-cost access to computers.

Why Bother?

Well mainly I am just curious - I want to know whether it’s possible IN PRINCIPLE to create cheap, automated bio lab equipment so I can run some experiments. But suppose it’s possible - it’s not crazy to say that costs could be cut 100x and automated equipment could eliminate a lot of the tacit knowledge required to run experiments or perform bio manufacturing.

This is the exciting part to me - a lot of people care about lab automation and AI in bio, but I think the real value is just in decreasing the capital costs of lab work by 100x and expanding the pool of people who can get involved by 100x.

Next Steps

There are probably a ton of reasons why this is a bad idea. But I already ordered $300 worth of tubes, ESP32 microcontrollers, heat plates, soldering equipment etc. Really I am just writing this as pure copium to console myself that this was not an incredibly stupid waste of money while I wait 10 days for everything to arrive from China.

In the next article I will share my experience wiring things up. My main goal is to be able to run PCR (polymerase chain reaction) in a completely automated way with modular DIY equipment with the ability to optimise over experiment parameters, like the number of PCR cycles or the temperature. If you live in / near London and want to help with the construction, please let me know on Twitter!

And if you don’t hear from me again it’s probably because I electrocuted myself wiring cheap AliExpress hotplates to the mains.

Human creativity and imagination, augmented by the raw search power of AI. The computer - the workhorse - automates away the predictable, zero-entropy decisions, freeing the human - the helmsman - to steer the ship through idea space.

The Centaur Era is well underway. We do not know how long it will last or whether it has an expiration date. But standing here today, you are a thousand times more capable than any human at any previous point in history, with a trillion more opportunities available. How will you decide what to do?

What’s possible?

Here’s a nice little appetiser for you to start sparking some ideas:

How about building a nuclear fusor in your bedroom using Claude 3.5 Sonnet and Claude projects?

I primarily relied on a giant Claude project filled with documentation from forums, call transcripts, mail threads and more…

I also used o1 pro for helping me with very complicated assembly or electrical wiring stuff. Found it much more useful for these cases. - Twitter thread, video explanation, podcast, blog post

Or perhaps you’d enjoy building a GPU from scratch:

I've spent the past ~2 weeks building a GPU from scratch with no prior experience.

My most important resources were Claude Opus, GPT-4, and these 2 repos…

More: decoding images from a portable brain scanner, home chip fab

Those are all tech and hardware focused, but consider this list of academic disciplines. For every academic discipline X there will be a computational variant of that field.1 Here’s an imaginary project I made up for “Computational History”:

I extended the work of historian Penelope J. Corfield, who studied optimistic and pessimistic attitudes in Georgian Britain. Using AI search tools, I located thousands of 18th-century novels, plays, poems, letters, diaries, guidebooks, journalism, sermons, songs, and sayings. I then designed an AI-assisted method to analyse national mood shifts over time.

The possibilities are endless.

Problem Search

Everyone’s capabilities have been amplified a thousandfold. We walk around with PhDs in our pockets. Soon, we may each have small armies of intelligent agents at our disposal.

As the cost to build drops, the value of people, ideas, interesting problems and intelligent starting assumptions increases. Problem valuation > implementation. The most urgent questions are philosophical not technical.

Finding a problem is our first problem. Here are some things to try:

• Debate valuation frameworks (one, two, three, four) and collect lists of ideas

• Survey many big open problems or deep-dive into a general topic like energy

• Get inspired by art eg. mind uploading from Pantheon (TV Series)

• Try lots of mini problems and challenges

The best measure of success - falling in love with a problem that brings life meaning.

(The search continues…)

Judges and Juries

But given a particular problem, how can you decide whether it’s worth pursuing? Sometimes it’s good to go in guns blazing because when under extreme uncertainty over-rationalisation is often a poor substitute for action.

But over the past 6 months I think I personally would have benefitted from discussing my ideas more with others. There’s actually an interesting evolutionary argument in favour of this - the argumentative theory of reason - which claims that reasoning evolved primarily for evaluating others’ arguments, not constructing our own.2

For example, research shows that groups outperform individuals on tasks like the Wason selection test.3 Of course, group decision-making is vulnerable to biases like groupthink, so the challenge is to create an environment that enhances rather than diminishes reasoning, which is something I’m actively thinking about:

Just like with AI, human decision-making benefits from reasoning, criticism, best-of-n and other test-time search methods. Here are some things to try:

• Get feedback on your ideas from ensembles of human and AI “judges”

• Write and discuss recursive “why?” documents4

• Write socratic dialogues arguing against yourself (I’ll share mine soon)

Learning

My sense is that in the Centaur Era, pre-requisites are largely a myth, and just-in-time learning is the modern way. The perfect AI-powered learning environment doesn’t exist yet, but it’s becoming easier and easier to learn while working in an authentic context, like an IDE, on a project you care about.

For example, I used the socratic tutor prompt (see the video below) to learn about hashing algorithms SHA3 and BLAKE3 while working on these bounties. I also used it inside Cursor to understand the DreamCoder paper and codebase.

There’s a GPT available here

Something more speculative that’s on my mind is what meta skills will be valuable to learn. I think one of the main ones will be task delegation. There was promising thread titled If I wanted to spend WAY more on AI, what would I spend it on?, but the lack of good answers from commenters suggests that people find it difficult to even think of what things they could be delegating to AI.

Maybe the skill of task delegation, or capital/compute allocation will become as important as the domain-specific knowledge of whatever problem you are working on?

Thanks for reading and get in touch on Twitter or Discord if these ideas are interesting to you!

Out of the blue, a handful of people messaged me regarding an obscure blog post I wrote a year ago about spaced repetition and recommendation algorithms. Surprisingly all of them found me through Exa, an AI search engine that seems to do a much better job of matchmaking weirdos than Google, which (justifiably) buries my blog somewhere in the abyss beyond the tenth page of search results.

The experience got me thinking: as the cost to build drops, the value of ideas, interesting problems and intelligent starting assumptions increases. A huge portion of the world’s knowledge is private - locked in people’s minds, private conversations, and informal networks. Since people are still the best idea generators in the universe, this means that the value of personal connections with interesting people will only go up over time.

Deferred Filtration

This led me to another thought: there aren’t many low-effort ways to meet interesting people to build things with in the UK. Everything seems to require vaulting into some kind of walled garden like a startup accelerator, or invite-only event, or a black-tie dinner etc.

In How might we heal the sick man of Europe? I argued that since Britain has plenty of talented young people, stable institutions and funding opportunities, the real problem is galvanisation. Like pre-industrial China or Russia, it’s an issue of mobilising the peasantry (fellow directionless youths) under a shared vision (national rejuvenation/improving the lives of others/solving hard problems) and getting them in the factories building (working together on businesses/other ventures).

I think that this view aligns with advances in AI and automation - if everyone’s capabilities have been raised a thousandfold and we’re all walking around with PhDs in our pockets, then the limiting factor to progress is not exceptional, rare talent that you need to meticulously filter for, it’s simply the will and determination to try!

Could a more egalitarian, open-source, third place model work? With the goal of maximising resource utilisation of the young and ambitious, not having a bunch of them lazing around between application deadlines until they finally meet people they can build with?

Historically, the most successful knowledge-creating cultures followed the principle of deferred filtration. They allowed for a large amount of random exploration and open-ended interaction - they didn’t start filtering too early. Like the Ancient Greek Agora - a small, densely concentrated marketplace for free exchange of ideas, with the stoa providing space for smaller collaborative circles to branch off, meet, and conspire.

Minimum Viable Community

This is one thing the Effective Altruists do well - their focus is on widening the number of people involved rather than prematurely filtering for an elite few. Having attended a few of their events here in Oxford, I got the sense that they were genuinely excited to have anyone interested in debating them about existential risk, AI doom, or morality.

They’ve built a smooth on-ramp - from the rationality community and Less Wrong to meet-ups, to shared houses where people can co-work and host events.1

Imagine something like this but centred around working on interesting problems. Sounds enticing to me. I feel like a minimum viable community is achievable for trivial money. What do you really need other than people, a building with WiFi and some chairs?

1. Tell a good story

2. Gather people

3. Brainstorm ideas

4. ???

5. Build Europe’s first trillion dollar startup

The plan is simple: People, ideas, machines—in that order!

“The UK is a miserable country and a bad country for young people to live… There is this culture of misery and pessimism that is quite contagious and it has a really negative effect on the collective psyche… There’s this apathy that comes with it at the same time that actually stops any progress”1

It’s not easy to be a Brit online these days. The discourse is flooded with laments about our economic stagnation and declining prospects.

Yet, there is a paradox at the heart of Britain’s economic woes. We have plenty of talented young people, a stable government, a strong legal system, and well-established institutions. And if domestic funding falls short, foreign VCs are eager to invest in ambitious British ventures.

But despite this we don’t produce nearly as many startups as our American cousins, and our economic growth has been non-existent for more than a decade.

How seriously should we take the idea that our “collective psyche” is what’s holding us back and how might we build a compelling aesthetic of British optimism for the 21st Century?

Pessimistic Pommies

A common refrain is that America thrives on a deeply ingrained cultural optimism - the American Dream, while we Britons are naturally more cynical, looking down our noses and scoffing at such naïveté. But optimism and pessimism aren’t fixed national traits, but moods that rise and fall in response to historical events and the prevailing ideas of the time.

Take Georgian Britain for example. Historian Penelope Corfield spent years collecting eighteenth-century reflections on “the age,” “the times,” or “the century.” Even in an age we now associate with boundless optimism, driven by industrial progress and scientific discovery, prophecies of doom were ever-present. Consider this dramatic warning from a Nottingham minister in 1778:

“Our country bleeds at its heart; our vices have risen to their crisis…and Britain, the envied among nations, the seat of glorious liberty, and science, and law, the refuge of the oppressed, the friend of mankind, is sinking into ruin.”2

Despite such dire proclamations, Corfield observed that that optimism gained cultural dominance during this era. In 1776, the philosopher Jeremy Bentham described his time as “an age in which knowledge is rapidly approaching towards perfection”. A popular song from 1830 captured a similar spirit of excitement about technological progress:

Open your eyes, and gaze with surprise
On the wonders, the wonders to come!3

Britain’s mood, then as now, was never wholly optimistic or pessimistic - it was a contest of competing narratives. If pessimism dominates today, it’s for two main reasons:

First, misleading claims like that global poverty is rising or that the world is getting worse have proven surprisingly persuasive and have led people to distrust the scientific, technological and economic transformations that have raised living standards across the world.

Second, Britain faces genuine challenges like unaffordable housing, the cost of living crisis, high energy prices and a rising crime rate, and young people see no clear path to solving them, creating a sense of despair.4

Manifesting Growth

Optimism and pessimism are not passive reactions to economic cycles or historical circumstances. They are actively shaped, often by great founders - whether religious leaders, philosophers, statesmen like Churchill, or the visionary US entrepreneurs from the 1980s.

If we were to craft an aesthetic - a national vision - to tell a compelling story of British growth in the 21st century, one that aligns with the opportunities of our time, what would it look like?

The Anglofuturist movement and podcast have already done great work here, broadcasting conversations with ambitious guests working on geothermal energy, artificial wombs and new islands in the North Sea - all from their orbital thatched pub, the King Charles III Space Station.

To return to the paradox we began with: Britain has all the ingredients for success. Talented young people, a stable government, a strong legal system, and well-established institutions. Is the missing piece simply the ability to galvanise them under a shared vision?

Looking to history, what can we learn from past efforts to mobilise societies toward ambitious goals? How did pre-industrial Russia and China move millions of peasant farmers into industrial production? How did wartime governments inspire women to take up factory work to support the war effort?

Take a look at these propaganda posters:

They created an attractive aesthetic of industrial scale, builder-led optimism…

They placed people at the focal point, as the agents of change…

They wove individual effort into a meaningful cause…

These campaigns crafted a clear, compelling narrative for the population, showing them their role in building the future.

And this approach maps surprisingly well to our current moment:

With AI and automation, everyone’s capabilities have increased a thousandfold. What’s scarce is not rare talent - but the will and courage to use these new tools to work on the world’s hardest problems.

Let’s build

As a first step towards that, here’s my humble pitch as a Brit sick of the pessimism and misery online:

I’m an ex-SWE from Oxford. I quit my job 5 months ago to find something more interesting to work on. Here’s what I’ve tried so far:

• A (bad) attempt at ARC

• (Slow) HVM3 optimisation bounty

• (Slow, buggy) parallel hashing algorithms for TinyGrad

• Fixed a bug in GPUOcelot’s PTX ⇒ LLVM compiler

• A (failed) refactor bounty for HVM3

• Other misc experiments: code agents, drum synth, writing, podcasts

• Now: DSPy-inspired library for text-based gradient descent (WIP)

I’d love to meet people interested in meeting up London to work together on a two-week project to build something interesting. Such as:

• A GPU from scratch

• New Brain-Computer Interface

• A Nuclear Fusor

Interested? Please message me on Twitter @experilearning or Discord.

Thanks for reading!

It’s been 5 months since I quit my job, here’s a summary and reflection on what I’ve been up to so far.

Overall I would say that there have not been many “wins”. I realise this sounds quite negative. Everyone wants to feel some form of positive feedback from the universe that they are not just wasting their time, and if you quit your job, you are going to feel additional pressure to make sure the opportunity cost of not working and earning money is worth it. How should you respond when it feels like you aren’t getting anywhere? Righteous indignation? Blame conspiracy or fate for condemning you to fail?

Unfortunately you can probably rule those out. Absent manipulation, markets (for attention, ideas, labor, etc) give honest feedback and aren’t designed to torture you in for no reason. In other words, if you aren’t getting the results you were hoping for, you know who to blame.

Again, this is overly negative and by the end of this essay I hope to persuade myself that it’s actually a cause for optimism.

Summary

• Learning

  • Math, textbooks, MathAcademy (~1.5 months)

    • Quickly began to feel like fake pre-requisites I’d invented for no reason

    • Why not work on real problems and use AI to backfill knowledge?

• Bounties and Competitions

  • A (bad) attempt at ARC (~3 weeks)

    • Learnt a ton - training NNs, implementing papers, running experiments…

    • Justified the idea that working on hard, interesting problems is The Way™

  • (Slow) HVM3 optimisation (~1 day)

    • Managed to get a +10% speedup but the bounty was for at least +50%

    • Again, justified working on hard, interesting problems > textbooks

  • (Slow, buggy) parallel hashing algorithms for TinyGrad (~2 months)

    • Tried to implement SHA3, showed that it can’t be parallelised

    • Implemented BLAKE3, wrestled TinyGrad’s CI, fixed GPUOcelot bug

    • Spent too long, stick to 2 weeks max for experiments/bounties

  • Found and fixed a bug in GPUOcelot’s PTX ⇒ LLVM compiler (~1 week)

    • This was actually a win IMO

    • Starting from ~zero knowledge of assembly, compilers, PTX, C++…

  • A (failed) refactor bounty for HVM3 (2 weeks)

    • Built a code context parsing tool for Haskell and C using tree-sitter

    • Used DSPy, active learning and LLM judges to create optimised prompts

    • Couldn’t reach accuracy/precision requirements

• Other Misc. Explorations

  • Autonomous Code agents, Drum synthesizer, Writing, Podcasts

  • Now: DSPy-inspired library for text-based gradient descent

Math

The thing I was most interested in when I quit my job was going deeper into learning math. I had already been spending all of my evenings and weekends on this before I decided to leave my job and I’d made solid progress through a number of undergrad textbooks. I even tried to make my own mini logic and proofs course using the Lean theorem prover.

So from the beginning I spent a lot of time going through math textbooks. I was also lucky to learn about MathAcademy pretty much immediately after quitting and interviewed one of the cofounders multiple times.

But after a couple of months I started to lose faith in studying math for math’s sake. Learning aimlessly divorced from a concrete problem lost its appeal. I didn’t create any interesting artefacts or “proof of learning” during that period (apart from flashcards) which also weighed on me. It felt like I was just fulfilling a sort of made up prerequisite. Why wasn’t I trying to be creative or solve real problems instead?

Was it short-termist to abandon learning in favour of problem solving? I felt (and still feel) this persistent stress caused by the idea that I’m on borrowed time, combined with the pressure to act that probably all of us are feeling due to advancements in AI.

I do genuinely think AI has changed the optimal strategy here. 1) It’s getting so much easier to backfill knowledge as you work on something. 2) AI updates so often that what you learn today may not be relevant in 6 months. 3) I think the main bottleneck is problem discovery and valuation which is a function of knowledge/learning, but probably not textbook/course learning.

HVM3 Optimisation Attempt

I looked over my list of potential projects and ideas and picked something I stumbled across on Twitter - a bounty to optimise a part of the HVM3 codebase. One of the things I was very keen on gaining from this period was deeper technical abilities and this seemed like a good way to work towards that.

I felt like I was able to quickly go from knowing nothing about the problem to actually making a reasonable attempt - I managed to speed it up ~10% or so and tried some other plausible things that didn’t work out. I looked at solution attempts from more experienced people on their Discord which showed me my ideas were reasonable.

To me it validated the idea that you can make much more progress than you would expect by just picking something hard and grinding at it.

ARC

Next I saw that the ARC competition was coming to a close but there might still be time to make a rushed submission. I worked on a ViT using slot attention in an attempt to create a model that could learn to partition the input grids into meaningful objects. I did a complete write up here.

In just 3 weeks I managed to learn a ton including a lot of tacit knowledge I couldn’t have gotten from textbooks or courses. It felt like a much better investment than reading a textbook, even though the end result was obviously no where near a viable solution to ARC. I reused a lot of the ideas I learned here later on to fix the GPUOcelot bug and when working with DSPy.

TinyGrad Parallel Hashing Algorithms

Then I remembered that TinyGrad has a public list of bounties. I’d been following the project casually because I used to watch George’s streams when I was starting out learning programming. I picked one of the $200 bounties which was to implement a parallel SHA3 algorithm in TinyGrad for hashing machine learning models using GPUs.

Unfortunately after 2 weeks it turned out that this is just impossible because SHA3 is just inherently sequential.

George updated the bounty, so I decided to give the BLAKE3 algorithm a shot.

This was more involved. I spent ages getting CI tests to pass on all of the different devices (Metal, CPU, Nvidia, AMD, …) TinyGrad supports. I ran their GitHub actions workflow locally and found a bug in one of the libraries they use to test the Nvidia device backend in CI. So as a side quest I spent around a week fixing that and managed to get it merged which was pretty cool given that I was starting from ~zero knowledge of assembly, compilers, PTX, C++ etc.

After that I spent ages trying to fix the final issues with BLAKE3 but in the end it started to feel a bit futile.

This was a recurring challenge I found while working on bounties. How do you know when to give up vs stubbornly persisting? One part of your brain will criticise you for being weak if you give up. The other half will remind you that every additional week you spend without making progress is a week you could have spent doing something more promising. I think this is something you can only really calibrate by trying and failing (there’s no way you can learn this from a book or course). After writing this I’ve settled on a heuristic of 2-3 weeks max for uncertain experiments/bounties.

HVM3 Refactor Bounty

Another HVM3 bounty. This time it was to develop an AI tool that given a snapshot of HVM3's codebase and a refactor request could correctly determine which chunks of code must be edited.

I wanted to see how well a small, cheap model could do at this task. I wrote a context parser class that used tree-sitter to parse the C and Haskell code and provide accurate context and used DSPy to write and optimise the prompts with an active learning-ish approach to mining synthetic data.

This was pretty fun, it used a lot of pretty cutting-edge ideas for building LLM programs like generating synthetic data, active learning, automated prompt engineering etc. It’s given me a solid understanding of how these general techniques can be used to solve problems and what their current limitations are. I’m very confident I will return to these ideas in the future. But again the accuracy and precision just weren’t good enough to reach the bounty’s target.

Final Reflection

Had I persisted with the math courses would I feel better off now? Certainly I would have gotten some grades or some other form of tangible feedback from the courses or textbooks to tell me that I was making progress. But progress towards what? I want to be able to solve difficult problems, create useful ideas, become more technical, build a company, do something interesting. I think creating unnecessary indirection in the way of that is just bad idea.

Ultimately the feedback you get from really tricky, uncertain problems like public bounties and competitions is going to be ruthlessly honest with no sugar-coating. If you suck, it’s because you actually suck. But that’s cause for optimism because it means that in terms of progress there is plenty of low hanging fruit yet to be picked. Looking at the quality of the work I did at the beginning compared to recently, there is clear improvement, I am capable of things I was not 5 months ago. I am confident that if I keep going I will just get better and better.

Next Steps

That’s all of the main stuff I worked on over the past few months. So what’s next?

“People, ideas, machines — in that order!”

In terms of projects I’m working on and off on a DSPy-inspired library for text-based gradient descent, but my main priority is fixing my environment. I’ve learned it’s just extremely mentally taxing to something like this by yourself. Starting a small Discord community with friends I met from Twitter and other corners of the internet certainly helped, but I think I need to find similar people IRL, ideally in London where I intend to move very soon. Definitely let me know if you want to meet up there.

Otherwise, cya in another few months, fingers crossed for some concrete WINS.

The human race will have a new kind of tool, a tool that will increase the power of the mind much more than optical lenses helped our eyes, a tool that will be far superior to microscopes or telescopes... What other consequences will follow from this tool are in the hands of the fates, but they can only be great and good. For although people can be made worse off by all other gifts, correct reasoning alone can only be for the good. - Leibniz in Preface to a Universal Characteristic

While I might be an optimist in the spirit of Leibniz, if reason is what separates man from beast, what’s left to separate man from machine? I want to be able to create and share ideas that others find useful and interesting, but is this a sensible ambition to have in an age where AI is advancing to the point where it will be competitive with or supersede humans across every intellectual domain? Will machine reasoners relegate human intelligence to a minor footnote in history, or empower us to achieve more than we ever thought possible?

Comparative Intelligence

The first thing to determine is how far to go extrapolating AI capabilities, because if there is a comparative advantage to human intelligence then I think it makes sense to lean into that instead of trying to compete directly with AI.

If our predictions are too conservative then we risk investing in skills that AI is better suited to, like spending six months learning mental arithmetic before the invention of the calculator. But if we don’t constrain our imagination at all then there's no limit to the capabilities we can invent, and we end up prophesying the arrival of artificial omnipotent Gods.

Hyperscaling

With that in mind, one of the main lessons I drew from my research is not to underestimate the speed of datacenter scale out and the eagerness of the so-called "hyperscalers" to pour eye-watering sums into training runs, test-time compute and AI infrastructure.

Maintaining the historical yearly increase in training compute will become challenging as companies run into nation-level constraints like the amount of electricity the energy grid can supply. But companies are already investing in huge infrastructure projects to overcome this.

Take Amazon for example, who just bought a nuclear-powered data centre1, or Microsoft who is re-opening the Three-Mile Island nuclear power plant in Pennsylvania2. Meanwhile Google is figuring out how to link together geographically distributed training clusters to tap into multiple regions' energy infrastructure3 and OpenAI is plotting a $500 billion AI supercomputer called Stargate.4

The justification behind this can be explained by the blue bars in the following graph of o3’s performance at competition math problems. Full-compute o3 achieved an accuracy 40% higher than o3-mini with low compute. What that means is that paying more in inference-time compute allows o3 to exhaust more of the solution space giving it a better chance of finding a solution to a problem. What if we were to train even better models and apply even more compute to important outstanding problems like the creation of new life-saving drugs, or a solution to the Riemann Hypothesis? There is a clear incentive for the world’s wealthiest companies to put everything on the line in pursuit of systems capable of doing this.

So be really wary of hiding your AI capabilities estimates behind compute or scale bottlenecks. If there's a large, but finite search space between current-day AI and the solution to a problem, it's best to assume that those bottlenecks will get vaporised through trillions of dollars and sheer force of will.

Expert Human Approximators

It's hard to say what will emerge at the end of a frontier model training run but I think it’s sensible to plan with the expectation that at the very least the hyperscaling era will stretch the o1 paradigm to its limits, and that AI language and visual reasoning will increase to a similar level relative to humans as the performance of AlphaGo Lee relative to Go players.

See the video below for a recap of the relationship between o1 and AlphaGo.

In other words, the lower bound for o1-style large language models is expert human approximators in any task where data is easy to collect, performance is easy to measure and where it’s easy to access or generate problems to learn from - think solving competition math problems, fixing well-specified GitHub issues, navigating the web, operating factory equipment and so on. “Expert human approximators” actually feels quite conservative - there doesn’t seem to be a barrier to superhuman performance in these areas.

If this is the case, then what will become of humans? Will there be any meaningful intellectual work left for us to do?

Imagination Games

There is something conspicuously missing from the list of recent AI achievements - “if even a moderately intelligent person had this much stuff memorised, they would notice - Oh, this thing causes this symptom. This other thing also causes this symptom. There's a medical cure right here.”

Why aren’t we seeing AI autonomously making creative connections leading to new discoveries? From Dwarkesh’s Twitter thread, commenters advanced a number of interesting hypotheses:

• “Models are capable of outputting many groundbreaking ideas, they don't yet have the capacity to test them, so they can't make discoveries”5

• “Superhuman research taste will remain out of reach for LLMs until we find and scale a new post-training paradigm analogous to the post-training that enabled high-quality CoT reasoning in math and coding”6

• “There aren’t any discoveries to be made just by reading the literature”7

Regardless of the reason, at the beginning of this essay I wrote that if human intelligence has any advantage whatsoever, then you should go all in on that instead of trying to compete with AI, and idea generation certainly seems like it is our biggest intellectual advantage.

As further evidence, consider the fact that even the least learned humans - children - are capable of surprising levels of creativity relative to modern AI. For example, children who haven’t read a single book are capable of spontaneously inventing new words that actually make sense.

When children play games, they invent fantasy worlds and scenarios they haven’t seen or experienced before, and are capable of resolving disputes by inventing (or re-discovering) moral rules.

Children frequently clash over the “finders-keepers” rule which grants individuals monopolies over desirable objects. It’s common for them to reinvent the principle of “sharing through turn-taking” to resolve such conflicts. - Imagination Games

Whether or not this is a “defensible moat” over the long term is obviously extremely uncertain, but the fact that children are capable of independent discovery and the world’s most intelligent AI systems are not gives me the impression that hyperscaling existing systems further along the same axes will not magically imbue them with this capability.

So where does this leave us, and what was this lengthy exposition a prelude to?

Centaur Era Starts NOW

We are entering the CENTAUR ERA where human creativity and imagination is augmented with the raw reasoning power of AI. If you care about progress, about knowledge creation and you want to play some role in building the future, abandon anything that the machine can perform on your behalf, automate away the predictable, zero-entropy bits and free your mind to focus on the questions of what is valuable, what problems are worth solving, what is beautiful, and what it means to make progress on a civilisational scale.

Embrace your unique intellectual endowment, get weird, come up with genuinely creative insights and create companies, organisations, partnerships to see them through. Help drive the next golden age of human flourishing and advancement.

The arrival of machine reasoners have only made it more obvious what you should have been doing all along, and they have opened infinitely many more doors than they have closed. All you need is the courage to abandon your outdated plans and pick something big to work on…

When I look out of the window behind my desk, why is the landscape so perfectly partitioned into different objects? Why do I see trees, cars, flowers and houses instead of just a swirling blur of particles or a chaotic kaleidoscope of colours? Human beings seem to have an effortless ability to turn raw visual data into meaningful objects in an integrated scene.

What criteria do we use to decompose raw visual data into objects? One obvious constraint is that the set of objects should faithfully represent what is really there. Our ability to survive in dangerous environments would be greatly impaired if relevant objects were omitted or represented in a misleading way. A faithful object decomposition should create a reliable foundation for reasoning and problem solving.

But for any scene, there are many possible faithful object decompositions. Take these stacks of chairs for example.

When moving a stack of chairs to another room it is useful to group information about the individual chairs together as a single object. On the other hand, when the goal is to count each of the individual chairs, a more fine-grained decomposition is preferred (and perhaps when repairing a chair an even more fine-grained decomposition is needed). - On the Binding Problem in Artificial Neural Networks

The best choice of objects and the appropriate granularity of decomposition depends on what is most useful for the particular problem at hand. For challenging problems, the large number of decompositions necessitates visual search over the possibilities to find the one that fits best, and with ambiguous scenes like the illusion below, your visual system may just flick restlessly back and forth without settling on a winner.

ARC

The reason why I’ve been thinking about vision and perception is because I got tired of just grinding math and decided to work on a concrete project for a bit, so for the past month I’ve been working on a machine learning model to solve ARC tasks.

If you aren’t familiar with ARC, it stands for the “abstraction and reasoning corpus” which is a dataset of grid-based reasoning tasks designed to assess an AI's capacity to understand and apply abstract concepts without relying on extensive prior knowledge. Solving these tasks using AI has been turned into a competition hosted by François Chollet and Mike Knoop with a $600,000 prize available for whoever is able to create a model that can solve 85% of the tasks.

Each ARC task is like a visual IQ test question. The input is a set of 3-5 pairs of example grids followed by a test grid. Your model should infer the common pattern between the example input-output pairs and apply it to the test input to predict the corresponding output grid.

Based on this context, the model should infer the following output grid:

There’s a lot of variety in the kinds of tasks and some of them are quite challenging even for humans. You can check out the tasks using this online viewer and try to solve a couple yourself.

Now one month wasn’t nearly enough time for me (a noob) to build a machine learning model good enough to solve full ARC challenges, so I won’t be getting the $600,000 prize any time soon. But I made some progress and I’d like to share my current approach and what I’m going to try next.

Perception

Solving an ARC task can roughly be broken down into two steps: perception and reasoning, so the first challenge is to create a model that can partition a grid into meaningful objects.

One thing you could do is feed the model human-annotated grid partitions and reward it for learning human object decompositions. But not only would that require a lot of boring manual effort to create the data, it would also result in a much less interesting model. I want to know whether a model can learn useful object representations by itself, not just copy mine.

But starting from scratch is quite daunting. Why? Well consider how many possible object decompositions there are for a simple 4x4 grid.

Since you can think of the grid as a set of 16 cells, the number of object decompositions is equivalent to the number of possible partitions of a 16 element set. Asking about the size of the set of all partitions is like asking how many unique ways you could smash a plate such that all the pieces could still be glued back together into the original shape.

The number of possible partitions of a set of elements is described by the Bell number B(16), which is over 10 billion! Since ARC grids can contain up to 900 cells, brute force search over all possible partitions would quickly become intractable.

But when humans look at a scene we generally only consider partitions into between 2 and 7 objects, a range that is likely related to our working memory capacity. If humans can solve ARC challenges under these constraints, then we should assume our model could as well.

We can count partitions of a certain size using the so-called Stirling numbers of the second kind. The number S(16, 3) is the number of ways 16 cells can be partitioned into 3 objects. If we want to find the number of possible partitions between 2 and 4 objects, we can simply sum S(16, 2) + S(16, 3) + S(16, 4) which gives a number 100x smaller than the Bell number B(16).1

Still, 100 million partitions is a lot to check, and the vast majority of partitions will be useless and uninteresting. For instance, this is one of the elements in S(16, 3).

To our human eyes, it just doesn’t feel meaningful. It ignores important information like the shape and colour information in the grid.

But this partition on the other hand feels much more useful because it segments the grid into two similar objects and the background.

So the problem of perception is how to create a model that can learn to pick out the few meaningful object partitions from the hundreds of millions (or billions) of useless ones.

Slot Attention

I experimented with an object-centric approach. My model starts by extracting features from the grids using a vision transformer (ViT). I was somewhat familiar with transformers already because I knew they were used for large language models, but they can also be used for non-textual inputs with structure like images. You can encode ARC-style grids of colours by flattening them into sequences of tokens and adding 2D coordinate position information to each token.

The tokens get passed into the transformer’s self-attention mechanism, followed by the slot attention module. The slot attention module is the key part of the model that enables the model to learn to recognise objects. You can think of the slots in slot attention as empty registers in working memory that can focus on different features of the input grid.

The slots compete with each other to focus on various features from the input like colour, shape and spatial position, which has the effect of dividing the input grid between the slots. At the beginning of training, it looks like nothing is happening - the slots look quite homogenous and there isn’t any differentiation between the objects at each slot position.

But gradually over training they learn to divide up the input grid between themselves. Because they see many thousands of grids during training and the slot attention module forces them to partition each input grid into a small number of slots, the model learns to pick up on common grid patterns that help it reconstruct the input grid.

By the end of training, you start to see some interesting partitions. For example, this one looks fairly logical. It segments the grid by colour to produce three meaningful objects.

Likewise with this grid, there are multiple ways you could logically partition it, but this one which divides the grid horizontally into a top and bottom piece feels pretty reasonable.

But there were plenty of unusual examples too.

Perception and Reasoning

The partitions above were only evaluated based on how well they were able to successfully reconstruct the input grid. But being able to reconstruct the input grid from a partition is a necessary but insufficient pre-requisite to having a good model. There’s actually a hierarchy of partition usefulness summarised in the image below.

The largest subset of partitions and the one that is easiest to train a model for is the what we’ve already discussed above - the ability to partition and reconstruct a single input grid. But ARC tasks are composed of multiple example input-output pairs. So a further constraint is that we only care about grid partitions that are in some way consistent or analogous across each input grid. It would be useful to be able to extract and match analogous objects that play the same “role” between the input grids and output grids and use that information for downstream reasoning.

I did some preliminary experiments here by tweaking the slot attention module’s initialisation strategy. I initialised the slots by extracting and combining features from all of the input grids using cross-attention. Then I ran the model independently over each grid with the shared slot initialisations. This meant that the model could semi-reliably extract similar objects at each slot index. Here are a couple of examples.

Overall I’m quite skeptical of this shared initialisation approach working beyond simple grids. It seems to heavily rely on spatial similarity as opposed to more abstract similarities. It’s likely that matching analogous objects for more complex ARC tasks would require a more sophisticated approach.

Returning to the hierarchy, the gold standard determinant of partition quality is whether the object partition the model finds actually helps you perform the downstream reasoning task. I ran out of time to perform any concrete experiments, but here are some vague ideas.

One of the benefits of object decomposition is that it allows us to perform compositional reasoning, meaning we can divide up a scene into objects, perform mental or physical operations on them independently and combine them into a solution. This divide-and-conquer approach makes the problem solving process simpler because each step requires less cognitive horsepower compared to trying to perform a single mega-operation over a complex object.

What this might look like practically is using an object-centric model to partition the input and output grids, matching the corresponding objects between the inputs and outputs, then having an LLM or symbolic reasoning engine write a simple program to perform the transformations over each set of analogous objects, checking that they succeed on the example input-output pairs before finally applying them to the test input grid. More experimentation here is needed!

My plans are to take a break from working on ARC to learn more about approaches to reasoning like discrete program search, LLM-based program sampling and DreamCoder. Excited to share more about that soon!

Here’s a three minute summary of the article if you want a quick overview:

The Drosophila of AI Research

Once upon a time, chess was the frontier of AI research. With its static set of explicit facts and rules, it provided the ideal laboratory environment in which to study problem solving and creativity.

Although perhaps of no practical importance, the question is of theoretical interest, and it is hoped that a satisfactory solution of this problem will act as a wedge in attacking other problems of a similar nature and of greater significance. - Programming a Computer for Playing Chess by Claude Shannon (1949)

Chess is one of the simplest games where in order to pick a good move, players must use strategies more sophisticated than a simple comparison of all possible moves. To understand why, start by thinking about the simpler game of noughts and crosses. With a 3x3 grid, you have a choice of 9 initial moves, and each turn the number of possible legal moves decreases as more of the grid is filled up.

To code an AI noughts and crosses player that performs optimally, it's sufficient to use an exhaustive brute-force search. Each move, recursively simulate playing every possible move and your opponent's responses until you find a line of play that will either lead to a win, or if that’s not possible, a draw. This is feasible because there are at most 362,880 total possible noughts and crosses games.1 While this might seem like a lot to a human, any modern computer would be easily capable of calculating this to see to the end of all lines of play and find the next best move in any grid state.

But problems arise when we try do the same thing in chess - each turn you have a choice of around 38 moves on average. Then for every one of those moves, your opponent will have on average 38 responses to choose from. This results in a game tree with a branching factor of 38 and creates a combinatorial explosion in the number of possible games.

A typical chess game takes about 42 moves, and because there are two players, this is multiplied by two, giving roughly 38^84 (38 to the power of 84) possible choices for a single game. Even after illegal moves are excluded you are still left with roughly 10^40 possible moves, which is more than the number of stars in the observable universe, far too many game states for a computer to reasonably compute.

Therefore the branching factor is a significant contributor to a game's complexity and shares an important relationship to the level of intelligence a game requires. As the complexity rises, players are forced to develop creative problem-solving techniques to help them find good moves without just comparing all possible moves.

The game of chess is therefore an endless open-ended search to find ever-more creative ways to overcome the computational limits of the brain, which makes it extremely addictive for intelligent young minds.

The passion for playing chess is one of the most unaccountable in the world. It slaps the theory of natural selection in the face. It is the most absorbing of occupations. The least satisfying of desires. A nameless excrescence upon life. It annihilates a man. You have, let us say, a promising politician, a rising artist that you wish to destroy. Dagger or bomb are archaic and unreliable - but teach him, inoculate him with chess. - H.G. Wells in Concerning Chess

Players express their creativity through the reasoning shortcuts - intuitions, sixth senses, so-called "heuristics" they hone over years of play. Heuristics are what the psychologists Khaneman and Tversky called "fast thinking", in contrast to manual search which can be thought of as "slow thinking".

So the first step for AI researchers was to turn these human-discovered heuristics into code, then have the computer assign values to potential moves and prune the branches of the game tree down to the most promising options.

As an example of how this was implemented, IBM's chess engine DeepBlue had access to a human-curated database of moves for its "opening-book" - the moves used in the initial stages of a game. It also contained a grandmaster game database of 700,000 games to help it evaluate potential moves. Since chess games are often annotated with “!” to mark strong moves and "?" to mark weak moves, DeepBlue was also able to assign bonuses (or penalties) to such moves in its evaluation function.2

But it's important to note that knowledge alone is not enough to perform well at a game like chess. Since the number of possible chess games is so vast, it's very unlikely that a player will ever see the same game twice, so it's impossible to be completely prepared in advance. Some amount of search will always be useful in allowing you to apply your general knowledge to board states you’ve never seen before.

In fact, DeepBlue actually heavily favoured search over knowledge. Remember that IBM was first and foremost a hardware company, so they built complicated custom parallel processors to power DeepBlue's search system. During its match with the world champion Garry Kasparov, the system was able to evaluate an average of 126 million chess positions per second! Surprisingly, even without revolutionary improvements in its heuristic knowledge, increases in the processing speeds of computers were enough to increase DeepBlue’s performance to the point where it was able to beat Garry Kasparov in 1997.

That’s an extremely important lesson worth highlighting. While relying on search might seem like a less intelligent approach than building up DeepBlue’s store of human knowledge, time and time again through the history of AI “breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.”

In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that “brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not. - The Bitter Lesson by Rich Sutton

Intelligence from Thin Air

After the defeat of Kasparov, researchers turned their attention to a new game called Go. Go is a further step up in complexity, with a board size of 19x19 and a branching factor of 361 (compared to 38 for chess). So while chess has more board positions than stars in the observable universe, Go has more positions than atoms in the observable universe!

Go masters are therefore forced to rely even more heavily on fuzzy heuristics and intuitions as opposed to raw calculation in evaluating potential moves.

Because Go masters' intuitions were so hard to formalise into a hand-crafted evaluation function in the style of DeepBlue, it took until the 2010s with the "deep learning" revolution for AI to finally have a shot at competing with the top Go players.

Deep learning is what allowed machines to capture these fuzzy, inexplicit heuristics into a densely connected web of knowledge called a neural net. By asking the network to predict the next best moves in the positions stored in a big database of expert Go games, the network was slowly tuned over time to improve its predictions and approximate the intuitions of an expert human player.

The essence of intuition is the ability to abstract away unimportant details and see the similarities between different scenarios. In noughts and crosses, a simple way we do this is by recognising that rotations and reflections of positions make them effectively the same. So while the total number of possible positions in noughts and crosses is around 5,478, there are only really 765 positions that are different after taking board state symmetries into account.

In Go, learning this kind of fuzzy pattern recognition enables the computer to create its own understanding of high-level concepts Go players use to describe the game, like "life-and-death", "influence" and "territory". Capturing these notions into neural nets allows the AI to see through the fog of war created by the combinatorial complexity of the game tree. Combined with search, the AI can produce reasonable moves in board positions it could never have experienced before.

DeepMind's first attempt at an AI Go player, AlphaGo, was similar to IBM's DeepBlue in that it was bootstrapped with human knowledge through a large database of expert Go games. But after this, the system gathered additional "experience" using a technique called self-play, a kind of reinforcement learning where the model played against itself and used the result of each game (win or loss) to determine which sequences of moves led to winning outcomes. The fact that it was able to train itself autonomously through this process was a very significant departure from the chess and Go engines of the past. It meant that the network was able to go beyond human-discovered knowledge and create completely new ideas of its own.

This was demonstrated in dramatic style in 2016, when AlphaGo defeated the Go world champion Lee Sedol. Both Lee as well as the Go community were shocked at the creativity displayed by AlphaGo, especially in move 37 of game 2 of the match. This move, which commentators originally thought to be a mistake, not only led to AlphaGo winning the match, but has since transformed the way human players see the game.

But the defeat of Lee Sedol was just the start of what makes AlphaGo so important in the history of machine reasoning. Like the defeat of Kasparov in 1997, it was yet more proof that machines could compete with human reasoning, but AlphaGo was still dependent in the early stages on a human-curated database of games. The next logical step was to remove this dependency and have the AI learn completely through self-play. And so AlphaGo Zero was born, zero meaning “zero human knowledge”.

During training AlphaGo Zero used randomised tree searches to explore potential moves. The fact that it was random is very important - firstly, it meant that it abandoned human prior knowledge, relying purely on trial and error instead. Secondly, as you might expect, random search involves much more failure than success - there are only a tiny percentage of good moves in a given position compared to bad ones, so these random searches were computationally expensive. But since there is a very clear feedback signal in the form of wins and losses, the network is able to get better and better at predicting good moves over time. It bakes the knowledge about the good moves it finds back into the network allowing it to directly guess the output of the tree search in the future without paying the computational cost.

It seems so counterintuitive, but this evolutionary approach of abandoning any pre-conceived good ideas about how to play Go and using random search as a teacher instead turns out to produce a superhuman Go player!

AlphaGo Zero is easily able to beat the original version of AlphaGo that defeated Lee Sedol in just three days of training and achieves superhuman levels of performance after training for 21 days.

Just like with chess, it's important to stress that knowledge alone is not enough to reach full performance. The following graph shows the Go ELO rating of “Full AlphaGo Zero”, which includes some randomised tree searching at play-time, against the "Raw Network" which simply predicts moves using the neural network. Without search, the network performs at roughly the level of a 3000 ELO rated player. But simply adding search to the raw network allows it to reach well above the performance of the best human players.

Once again the “Bitter Lesson” was vindicated: abandoning human-discovered knowledge, allowing the computer to learn by itself from scratch and combining this with play-time search proved to be the most effective way to scale machine intelligence to a superhuman level.

Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale… Learning by self play, and learning in general, is like search in that it enables massive computation to be brought to bear. Search and learning are the two most important classes of techniques for utilising massive amounts of computation in AI research. In computer Go, as in computer chess, researchers' initial effort was directed towards utilising human understanding (so that less search was needed) and only much later was much greater success had by embracing search and learning. - The Bitter Lesson by Rich Sutton

The Self-Taught Reasoner

Following in the footsteps of AlphaGo, the recent release of o1 from OpenAI represents yet another leap in AI reasoning capabilities, massively improving its ability to solve problems in mathematics, programming and science.

...o1 uses a chain of thought when attempting to solve a problem. Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognise and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn’t working. - Learning to Reason with LLMs

While the exact details of how o1 was trained are not publicly available, it is pretty clear from clues released by OpenAI that it must be using a similar approach to AlphaGo. o1 is basically the application of AlphaGo-style reinforcement learning to language-based reasoning.

Reinforcement learning is actually already used to fine tune chat models’ outputs in an approach called RLHF (reinforcement learning with human feedback). In RLHF, humans rank the generated text outputs from a language model by comparing multiple answers it generated and picking the best. This data is then used to fine tune the model to steer it in the direction of producing similar answers to the ones humans prefer.

Collecting data using humans is slow and expensive, but once you have enough data, you can use it to train a reward model on these human-assigned ratings and get it to predict what grade a human annotator would assign to a new output without human assistance.

Once the reward model is trained, the process becomes much faster. You can automate the feedback loop by having the LLM generate multiple outputs to the same question, grade the outputs with the reward model (no human required!) and then fine tune the language model only on the best answer.

This process is called outcome supervision because the reward model assigns a score only to the full output of the language model. But this approach has its limitations. Sometimes the model could produce a correct answer to a question using incorrect reasoning steps. The big innovation made by o1 is in applying reinforcement learning at a more granular level by breaking down the responses to long problems into “chains-of-thought” and grading every individual step towards a problem's solution. This approach is called process supervision.

Just like with RLHF, OpenAI started by collecting data from human annotators.

But once they collected a certain amount of data, it became possible to switch to the automated approach of training a reward model and using it in a feedback loop to grade the model’s outputs without human supervision.

When we think about training a model for reasoning, one thing that immediately jumps to mind is you could have humans write out their thought process and train on that. The aha moment for me was like when we saw that if you train the model using RL to generate and own its own chain of thoughts it can do even better than having humans write chains of thought for it. That was an aha moment that you could really scale this. - Building OpenAI o1

Similar to the self-play training in AlphaGo, o1 generated thousands of chain-of-thought style answers to problems. Like AlphaGo’s random tree searches, o1 used a high temperature to increase the variance in the kinds of answers it attempted. While the model may only get the answer correct one out of a thousand times, when it succeeds, it can easily be verified using the reward model and fine-tuned back into o1 to improve its reasoning process in the future.

The final graph I will leave you with is this one from the Let’s Verify Step by Step paper by OpenAI which shows the results of best-of-N search with three different approaches. The orange line represents the performance of the process-supervised method used by o1 as compared with the outcome supervised approach, or majority voting (picking the most commonly generated output). Notice that while the other two approaches plateau after sampling 1000 solutions from the language model, the process-supervised approach seems like it has more room to scale even further. What if o1 was allowed to sample 100,000, a million, or trillions of solutions per problem? What sort of problems might it be able to solve as a result?

This graph explains the sudden focus on scaling energy production to power AI. As the Bitter Lesson has shown us, even if search seems inefficient or unintelligent, even if it requires re-opening nuclear power plants to pour energy and compute into these models, companies will be willing to put everything on the line if they believe there’s a chance it might help us solve some of humanity’s most challenging problems.

What inference cost are we willing to pay for a proof of the Riemann Hypothesis?What inference cost are we willing to pay for new life-saving drugs?

- Noam Brown in Parables on the Power of Planning in AI

I hope you enjoyed this article. If you are wondering what the implications might be of scalable machine intelligence, what the limits of o1’s approach are and how to plan your next move based on these developments, you are not alone! My next article Centaur Era Starts Now will explore those questions. Consider subscribing here or follow me on Twitter to get notified when it releases.

Thanks to algo-baker, Niccolò Zanichelli and Rammohan Sharma for their feedback on drafts of this article!

The economist Albert Hirschman once said of developing economies that "all growth is unbalanced growth" and that they should concentrate their limited resources into developing a few key sectors at the expense of others. By doing so, he argued that the short term chaos and exacerbation of inequalities would eventually smooth itself out across the economy through positive spillover effects where growth in the economy's key sectors would stimulate growth in others.

It struck me recently that a young man is literally and figuratively a developing economy: at any point in time, one or more sectors of your life are in poverty and you need to decide how to allocate your limited resources to plug the holes in the sinking ship. Like Venezuela, your debt levels are soaring from student loans. Like Greece in 2008 you have no savings, making you vulnerable to downturns in the global economy. And environmental shocks or Black Swan events threaten to send you into depressions that destroy your productive capacity.

While Hirschman's advice may work for developing economies, I don't think it applies to a young man's life. The trap I fell into repeatedly during my early 20s was abandoning every other interest I had to focus on a single specific thing. In secondary school, it was going to the gym at the expense of grades and social life. In university I spent almost the entire the time locked away inside my room studying instead of joining social clubs, partying or making friends. Then after university it was coding, while other areas of my life fell into disarray.

I've taken to calling that period of my life my “Great Leap Forward” in reference to Chairman Mao’s radical campaign in the '50s and '60s to revolutionise the Chinese economy. Just as Mao's blind idealism drove up China's industrial output at the cost of disastrous nationwide famines, so did my ascetic daily routines help me find a career I'm passionate about, but at the expense of having no deep or meaningful friendships with people in real life.

This is by far my biggest mistake and regret in life - chasing something I wanted by withdrawing from social life to spend as much time as possible working alone. And I suspect that many of the men reading this can relate. The idea of isolating yourself from the world for six months to undergo some sort of self-imposed rite of passage appeals to that Mao-like revolutionary fervor in our genes to achieve our goals through sheer force of will and self-destruct in the process.

But isn't the expression of talent, even reasoning itself fundamentally social?1 Look back through history at how many of humanity's greatest intellectual and creative achievements were the products of so-called collaborative circles like the Inklings, the Lunar Society or the Vienna Circle.2 Isolating yourself and expecting to reach your potential is like trying to lift yourself off the ground by pulling on your own hair.3

I feel blessed to have finally found people online scattered across the globe who love me for my authentic self rather than the character I play in real life, brothers with whom I’m comfortable sharing my wildest, most tentatively held ideas, friends who've funded my speculative projects despite having never met me and who lift me back up when I fall flat on my face.

The unpayable debt I owe them is for their sheer encouragement.4 Because while I might be able to regurgitate some reasons why optimism is rational when I feel hopeless or rattle off a few motivational quotes when I’m feeling depressed, it’s only when I’m with close friends that I can achieve truly delusional levels of self-belief and experience those spirals of escalating mania as we talk away into the night about our niche obsessions.

To anyone struggling to find their people, I hope you find your way.

We Walk On the Great Road (1962)5

An excerpt from a Chinese patriotic song published in 1962, the end of Mao’s Great Leap Forward. The “Great Road” symbolises China’s future road to development. Cast in their historical context, the lyrics are bittersweet; the dreams of China’s “infinitely brilliant” future stood in stark contrast to the horrors endured during the deadliest famine in human history.

Our friends are all over the world,
Our songs are heard in all directions.
The revolution’s storm sweeps far and wide,
Sending ghosts and demons to hide.

March on! March on!
The revolution’s force cannot be stalled,
March on! March on!
To victory’s call we’re all enthralled.

Our road is endlessly broad,
Our future is infinitely brilliant.
We give ourselves to this grand cause,
With boundless joy and noble applause.

March on! March on!
The revolution’s force cannot be stalled,
March on! March on!
To victory’s call we’re all enthralled.

Thanks to Alexey for his feedback.

I want to introduce you to a new concept I’ve thinking about recently - the concept of an imagination game. It was initially inspired by the mathematics courses I’ve been taking, especially the concept of hypothetical reasoning in logic, but it feels like it could become a generally applicable mental model for analysing many different open-ended creative processes.

Thinking through the lens of imagination games helped me see the connection between how mathematical and scientific knowledge grows, and how many creative domains share similarities with childlike play. It has also helped me gain an understanding of things I couldn’t explain before, like why certain mathematical theories are the way they are, and how they could have evolved differently.

In this essay I will argue that the kind of reasoning children use to dream up fanciful worlds of princesses, trolls and heroes to play in with their friends is fundamentally the same kind mathematicians and scientists use to formulate their abstract theories and models of the universe.

Imaginative Reasoning

There’s a common pattern of reasoning that goes by different names in different disciplines: in natural deduction, the most popular proof system in mathematics, it’s called hypothetical reasoning. In science it’s called hypothesis generation or conjecture and in philosophy it’s the kind of reasoning associated with thought experiments. It exists all across the social sciences too - historians often use counterfactual reasoning when analysing past events, and in finance, speculating about what could happen in the market captures the essence of this kind of reasoning with a pejorative slant.

But it’s not just professionals or academics who use this style of reasoning - take the cognitive distortion catastrophising1 for example, or daydreaming about what you would do if you won the lottery - both of these are universal thought patterns that use the same underlying style of reasoning as the more academic examples above. In fact any time you’ve used conditional or speculative sentence constructions you’ve used this style of reasoning:

• Imagine if I won the lottery, then I could travel the world.

• Suppose Napoleon had not invaded Russia in 1812, then Paris would not have fallen in 1814.

• Assume for the sake of contradiction that to occupy a given position in space is to be at rest, then a flying arrow is at rest.2

I group all of these terms and sentence schemas under the general category of imaginative reasoning3. In each case imaginative reasoning boils down to making one or more assumptions (without pressure to explain or justify them in detail) and exploring the consequences. In the sentences above, the words “suppose”, “assume”, and “imagine if” introduce assumptions while the second clause (after the word “then”) derives a consequence.

Imaginative reasoning by itself is quite a granular and “zoomed-in” view of reasoning because it’s defined at the level of sentences and sub-clauses. But when we zoom out from, say, how imaginative reasoning is used in the context of a single line of a single proof in propositional logic, and instead think about the role of imaginative reasoning in the creation of an entire mathematical theory, then there’s a whole new class of interesting questions to explore.

We can wonder about things like where the assumptions at the very bottom of the theory, called axioms, came from in the first place? Are they self-evident truths or something else? Can they be connected to imaginative reasoning somehow? How is it that starting from a blank page mathematical theories evolve to so perfectly capture the relationships and behaviours we wish to model while excluding those we want to forbid?

These are the questions that motivated my interest in imagination games. But before extending the idea to the highest intellectual realms of human endeavour like mathematics or science, let’s being by examining its humble roots in the minds of young children.

Make-Believe

When a 2- to 3-year-old boy plays airplane with a wooden stick, … he is at the same time spontaneously creating his own imaginary reality. Who is to say that this reality is not more "real" to the boy than the adult reality? - Desire for society: Children’s knowledge as social imagination

Do you remember when you were a young child hanging out with your friends and how you would often spontaneously conjure up elaborate fantasy worlds of princesses, trolls, and heroes to play in? I realised that I have gone my whole life taking for granted what an incredible feat of intelligence this is. The invention of a pretend world involves a great deal of imagination - to think of an interesting setting, to come up with roles, names and titles and use symbolic substitution to turn the plain, meaningless objects in the environment like sticks and stones into wands, guns and currency.4

Before they can start playing, children must negotiate a starting set of facts and rules governing how they are expected to behave during the game. Some of these will be ad-hoc rules specific to the game, while others appear to be universal, like the “finders-keepers” rule for role-playing props which is consistently reinvented by children across different cultures. Each player also brings their own set of inexplicit background rules and facts, like moral rules, knowledge of generations and gender, cultural assumptions and more.

The set of rules created at the start of make-believe games is never rigorous and fixed, but fuzzy and malleable. Once children start playing, power struggles over possessions or the relative status of their roles, and arguments about the game being boring or unfair result in re-negotiations and modifications of the rules. For example, children frequently clash over the “finders-keepers” rule which grants individuals monopolies over desirable objects. It’s common for them to reinvent the principle of “sharing through turn-taking” to resolve conflicts. In this way the game and its ruleset evolve progressively over time as problems are encountered and resolved during play.

Most make-believe games die when they are abandoned and forgotten by the small number of players who created them, but every now and then a game captures the attention of its players in a way that defies intuition - its small, finite list of facts and rules produces games with seemingly infinite depth and complexity. It feels like you could play them every day for the rest of your life and find endless novelty and enjoyment in them.

When we find games with these properties, we are eager to copy them and share them with others so they can explore them with us. Such games are replicated across different social groups and are passed down from generation to generation. Piece-by-piece, the ad-hoc conjectured rule sets existing purely in the minds of players are transcribed into explicit lists of rules and regulations, like those governing modern sports and chess. Formalisation and sharing allows others to play the game without having to go through the slow trial and error process of inventing it for themselves. Over time, these formal facts and rules become precious cultural artefacts that encapsulate a vast amount of knowledge about how to solve the problems encountered during the game’s evolution, making them harder and harder to improve upon without causing regressions.

Imagination Games

There are a few properties of make-believe play which I believe can be found across all imagination games.

Imagination

An imagination game always starts with an act of creativity to conjure up an initial set of facts, rules, or assumptions that define the game and guide the players’ behaviour. Imagination is also used throughout the game to conjecture refinements to the rules and facts in response to problems deliberately caused or stumbled across by the players that were not possible to predict in advance.

Facts and Rules

Players establish a set of facts declared to be true for the purpose of the game, and some rules governing the behaviour of the players. Don’t mistake “facts” for “true statements”, they are best thought of as “imaginary assumptions”, and could be fantastical, improbable or downright absurd. Together, they are like the chessboard and pieces in chess: they delimit the game’s boundaries and permissible actions, but unlike a rigid chess game, they can exist on a spectrum from formal and explicit to fuzzy and implicit. Whether the facts and rules of the game are constrained to what is deemed “reasonable” or “rational” depends on the game, so the subject matter of imagination games is not limited to what’s real, or even plausible but merely what’s logically possible.

Play

Playing the game constitutes exploring the make-believe world. It’s a cyclical process in which players explore and experiment with the consequences of the facts and rules. Inevitably, rules are broken and facts challenged. Through argument, debate, consensus or force, the facts and rules are adjusted to solve problems and the game is continually re-imagined in the minds of the players. While the facts and rules of the game become harder to vary and improve on over time, they never reach a point of perfection beyond which they cannot be further improved.

Understanding Mathematical Creation

I want to show you that the kind of reasoning children use to dream up their fanciful worlds of princesses, trolls and heroes is fundamentally the same kind mathematicians use to formulate their abstract theories. To that end, I’ve written a simplified piece of discovery fiction about how you might have gone about formalising the basic rules of arithmetic, to help you understand how mathematical creation works.

Your name is Giuseppe Peano, the year is 1889 and you are polishing your latest manuscript, Arithmetices principia, nova methodo exposita, “The Principles of Arithmetic, Presented by a New Method”. Influenced by the axiomatic approach of Euclid’s Elements, the goal of your work is to formalise the rules underlying the elementary mathematical operations of arithmetic like addition and multiplication. Your dream is to find a minimal set of simple facts and rules, called axioms, to derive the full theory of the counting numbers.

To an outsider it might seem silly for you to direct your genius intellect towards something that seems so inconsequential, after all, can’t we just take it for granted that 1 + 1 = 2? But to you, mathematics is the study of formality and rigour, even at the expense of frustrating your colleagues when you point out their errors in public. Even amongst mathematicians you are considered eccentric in terms of your meticulous attention to detail, going so far as to develop your own simplified dialect of Latin to communicate your ideas more precisely.5

Through a great deal of trial and error, you’ve managed to create an ingeniously simple list of axioms which you believe are sufficient to model the counting numbers.

Since it’s almost time to publish your manuscript, you scan through some of the axioms one last time, recalling how and why each became a necessary component of your theory.

Let’s start with axiom number one. It’s the simplest of the axioms which asserts that zero exists. In contrast to the other axioms, this one was pretty obvious - without it, the number line would lack a beginning, a starting point from which all other numbers could follow. Zero provided the single raw ingredient required to create all of the other numbers.

The second axiom was quite the invention, even if you say so yourself. The challenge of the theory is to model the positive side of the number line which is an infinite set of numbers. You tried listing numbers as axioms like you did for zero:

• one is a number

• two is a number

• three is a number

• …

But this wouldn’t work - even if you spent an eternity writing additional axioms for each number, you’d never reach infinity. And even if you did somehow write axioms for all numbers to infinity, you wouldn’t have specified the relationship between each number - they would be a jumbled bag of disconnected dots instead of an organised line. These problems motivated you to formalise the concept of a successor, which says that every number leads to the next number called its successor.

So with axiom one you established that zero exists, and according to axiom two, if zero exists, then so does the successor of zero, called one. And if one exists, then so does its successor, two, and so on…

• zero exists

• one is the successor of zero

• two is the successor of one

• …

In one fell swoop you took care of creating an infinite set of numbers and ordering them in a line through the successor relationship. Bravo!

Now look at the third axiom in the list. It states that given any number, that number plus one cannot be equal to zero. You realised it was necessary because without it, you ran into situations where nothing could prevent 7 + 1 being equal to 0. This produced weird effects, like causing “wrap around” issues where instead of being able to count to higher and higher numbers, you got trapped for an afternoon in an infinite loop cycling endlessly through the first 8 natural numbers with no escape.

The fourth axiom establishes a unique link between a successor and the number that comes before it, for example between the number 5 and the number 4. Only 4 + 1 can equal 5 and nothing else. Again this rule was necessary to exclude. When you spent the morning testing your theory when you didn’t have this rule, you realised that 3 + 1 and 4 + 1 could be equal to 5, and both 6 + 1 and 9 + 1 could be equal to 10! This turned the number line into something that more closely resembled a branch of a tree with twigs protruding from it.

Now back to reality. From reading textbooks I got this very misleading impression that Peano somehow sat down one day and wrote out the axioms of the natural numbers in one shot. I had this misconception that axioms were unquestionable “self-evident truths”.

But despite being a simplified caricature, don’t you think the fictional account of Peano’s formalisation of arithmetic seems like a more realistic description of mathematical creation? While Peano was uniquely suited to the challenge of formalising arithmetic because of his penchant for rigour and minimalism, it can’t have been a simple task. Since he was working at the level of axioms, there wasn’t any “raw material” to use - the axioms he came up with couldn’t be mechanically derived from other mathematicians’ work. They had to be imagined into existence.

Would it therefore be fair to say that mathematics is an imagination game? At the very least it seems reasonable to say that imaginative reasoning is heavily involved in the creation of mathematical theories, to invent axioms out of thin air and reason under the assumption they are true. But to show that it’s an imagination game we need to challenge a widely held misconception that once a mathematical theory is created, it is frozen in time as an eternal truth. Because this would mean that the growth of mathematical knowledge proceeds in a linear fashion where once it is created, it is simply added to the bedrock of established truths. We need to show that like the set of rules defining a child’s make-believe game, mathematical theories can never be considered final truths, they are always fallible and always subject to change in response to problems encountered by players.

There’s a wonderful book called Proofs and Refutations by the Hungarian mathematician Imre Lakatos which makes a strong argument in favour of this view. Lakatos’ book is structured as a socratic dialogue between students and a teacher that reconstructs the history of mathematicians’ attempts to prove Euler’s polyhedron formula, which states that the number of faces plus the number of vertices minus the number of edges is equal to 2, or concisely, that V - E + F = 2.

The students produce counterexample after counterexample to the teacher’s statements by playing with the conception of what it means to be a polyhedron. They repeatedly come up with “deviant” polyhedra that satisfy the teacher’s definitions strictly, but aren’t what you’d normally think of as polyhedra. In the picture below you can see the kinds of shapes that the students came up with to break the formula.

The teacher responds to these counterexamples, or “monsters” as he calls them, by refining his definitions to exclude these undesirable cases through a process he terms “monster-barring”. The back-and-forth dialogue shows that mathematical creation is a much messier affair than the clean, linear way it’s presented in math textbooks, instead the repeated cycles of conjectures put forward by the teacher and refutations by the students make mathematics look much more like a science.

So just like make-believe games, mathematical theories are brought into existence by an act of imagination. Just as fantasy creatures don’t need to exist for children to play and behave as if they did, nor do the axioms of mathematics need to describe physical reality for mathematicians to explore their consequences. And as Lakatos showed us, play in both scenarios is a cyclical process of trial and error: children repeatedly re-negotiate the facts and rules of their games in response to problems, while mathematicians make continual adjustments to their axioms to banish the “monstrous” inconsistencies and counterexamples from their fantasy worlds.

Reverse Engineering Science and Technology

For, as [Popper] saw it, all advances of scientific understanding begin with a speculative adventure, an imaginative preconception of what might be true... It constituted an invention of a possible world, or a fraction of that world… - Sir Peter Medawar: science, creativity and the popularisation of Karl Popper

If you accept the idea presented above that mathematical creation is an imagination game analogous to the make-believe games played by children, then the idea that science could be too should feel like less of a stretch.

Just like mathematics, the creation of a new scientific theory involves the imagination of a new possible world that is not directly derivable from past theories because if the current best theory directly implied the challenger theory, then the challenger theory wouldn’t really contain new knowledge.6 There must always be a discontinuity between the old and the new bridged by a leap of imagination, and it must always be a speculative leap into a sort of fantasy world, because by definition a new theory will not be considered “reasonable” or “rational” by the standards of the current best theory. Take Einstein’s theories of relativity published in 1905 and 1915 for example. It took at least until 1919 with Eddington’s famous experiment for his theory to become accepted as superior to Newtonian gravity, because it directly contradicted centuries of Newtonian mechanics.

Now instead of walking you through a story similar to the formalisation of arithmetic example above, I want to show you how the concept of an imagination game can be used as a tool. Whenever you recognise something that seems like an instance of an imagination game, you can often reverse engineer why it evolved into its current state. In other words, imagination games are interpretable.

Implicit in the current state of the game is a long series of decisions made by players about how the facts and rules should evolve to solve problems encountered while playing. For example, when children playing make-believe tell you that they invented the rule of sharing role-play props through turn-taking, we can come up with reasons as to why they invented this rule by imagining what problems would have arisen in its absence, like arguments about the fairness of the game. Of course, how the children came up with the rule is difficult for us to say, it could have been suggested by an adult or an original creative insight by one of the children, but understanding why the rule was retained is simpler. It should follow some sort of interpretable logic.

To perform this kind of backward reasoning, all you need to do is ask “let’s assume this did not exist, what would break?”. Recall that we already did exactly that during the fictional account of the formalisation of arithmetic.

Let’s assume the third axiom did not exist, what would happen? Ah, then I could cause ‘wrap around’ issues where I could count from 0 to 7 and wrap back around to 0.

This kind of reasoning is actually a type of imaginative reasoning known since Ancient Greece, called “reductio ad absurdum”, or “reduction to the impossible”. The Ancient Greeks relied on it heavily as a tool for their philosophical and scientific endeavours because they could use it to pinpoint misconceptions in people’s arguments. Think about the Socratic dialogues or Zeno’s paradoxes, they are both great examples of this style of reasoning.

“Assume that to occupy a given position in space is to be at rest, then a flying arrow is at rest, which is a contradiction.” - Paraphrasing Zeno. He used this as a counterargument to the Pythagorean theory of motion, showing how it leads to an absurd result.

In the context of imagination games, especially in science and technology, this technique is valuable for three reasons.

Firstly, you might discover after challenging every rule and fact of the game that each seems to play an essential role. Try as you might, it doesn’t seem possible to make an alteration and improve or simplify the game. This is exactly what happened in the formalisation of arithmetic example above. We tried to delete axioms one-by-one and found that it caused catastrophic errors. We can describe Peano’s axioms as “hard-to-vary”.7

Another possibility is that you find that the theory exists somewhere on the spectrum between hard and easy to vary. Michael Nielsen describes this in this essay where he talks about the insights he got from writing a piece of discovery fiction about Bitcoin. .

To deepen my understanding I wrote a discovery fiction essay that begins with the question: how could we design a digital currency? And then describes an extremely simple, obvious idea for such a currency – an obviously flawed back-of-the-napkin idea, the kind of thing anyone familiar with computers would come up with if they took the question seriously. Then the essay points out an obvious problem with the idea. And proposes a simple, obvious fix. Then points out another obvious problem. And another simple fix. And so on, over and over, until eventually we "discover" Bitcoin! This approach shows very concretely how and why each element of Bitcoin is introduced, and what purpose it serves; it also makes clear that some features of Bitcoin are a little arbitrary, and how they could have been different. - Discovery Fiction

By finding a mix between essential and non-essential properties you understand there was some randomness in the evolution of the technology, that if you were to re-run the imagination game from scratch you’d likely end up with a different result

But there is also a third possibility - that you uncover a hidden problem no one spotted yet.

You may, for example, find that there are difficulties in a certain presentation, something that you don’t quite understand. Others may not even notice these difficulties… There may, for example, be gaps in an explanation. Or there may be actual inconsistencies—perhaps in what a theory says, or perhaps between what a theory says and what you observe. - Lecture 3: Problems by Karl Popper

And this is the most exciting outcome. Because every new scientific theory or piece of technology starts with a problem, the seed for a new imagination game.

—

Many thanks to Niko, Alex, Paul and Srdjan who gave very helpful feedback on drafts of this essay! And feel free to send me a message on Twitter if you have any thoughts.

Part of the reason I’ve been thinking about this is that I’ve been finding it tough to find people in real life who are interested in chatting with me about things like spaced repetition systems or doing mini projects together. It’s rare for me to meet people who use flashcards for things other than school, and I can’t exactly walk down the street and expect to run into geeks like Zander who’ll spend months deliberating the intricacies of flashcard formulation with me. Online it’s different - no matter how niche your hobby, it’s pretty much effortless to find others interested in the same thing.

So something I’ve been trying recently is reaching out to people and inviting them to do synchronous learning sessions together, inspired by a blog post called “Synchronous Reading with Friends”.

Reading synchronously means reading the same thing together at the same time. Read out of the same physical book or share screen on Zoom. Only flip to the next page when both of you have finished. Read silently, but interrupt each other if you don’t understand something or have a comment. - Synchronous Reading with Friends

This was something I used to do quite often during the COVID years with people I met in the SuperMemo Discord. We would take turns going through our flashcard queues, reading snippets of papers, textbooks and articles together. This turns out to be way more fun than you might expect. Doing reading and flashcard reviews collaboratively introduces an extra creative element because there’s an additional brain helping you search for novel associations. The observer can also argue with you about why your flashcards are incorrect or poorly formulated. If one person knows more about a particular subject than another, they can break down the text into simple terms the other person can understand, or the person less familiar can watch more passively and interject from time to time with questions. It even works when one person is a complete novice and knows little about the content the other person is learning because they can still comment on aspects of the learning process.

Here’s a great example of synchronous learning in action courtesy of Andy Matuschak and Dwarkesh Patel:

Beyond being a great way to get to know someone and their current interests, synchronous learning also helps you uncover tacit knowledge and workflows that aren’t typically shared publicly. There are plenty of polished “My Workflow for <thing> in <note taking tool>” videos on YouTube, but there isn’t much raw footage of people just learning. It’s probably because it’s hard to make this kind of content engaging for the audience, especially if you are recording it solo because learning often requires long periods of concentrated silence. Videos like the above are much more watchable because the person observing can interrupt and get the learner to explain their process aloud.

There are some more examples of this kind of footage. For programming you can find lots of raw footage from George Hotz’s streams. But while the Twitch audience plays somewhat of a role, they are too easy for the person being recorded to ignore. It’s not as interactive as a synchronous learning session where the observer can directly interject and demand an explanation! Similarly, me and Soren have made a couple of videos about how to read a book and make flashcards in RemNote and how to use RemNote to learn math. But I think they would be improved massively by having an observer there during the video asking questions like Dwarkesh.

There was a cool project I stumbled across in a language learning server ages ago where they were encouraging members of the community to share their sentence mining workflows in a more organised way. They wanted to use the wisdom of the crowd to converge on an optimal workflow for sentence mining. I’d be curious to see something similar for learning workflows involving SRS outside of language learning.

One final thought I had about synchronous learning is that it’s a great replacement for reading groups. The big problem with things like reading groups and tutorials at university is that no one wants to schedule time to do the reading in advance. Synchronous learning is a similar solution to what Bezos implemented at Amazon for meetings.

In a typical meeting, we'll start with a six-page narratively structured memo and we do study hall for 30 minutes - we sit there silently together in the meeting and take notes in the margins and then we then we discuss… We could say that we would like everybody to read these memos in advance but the problem is people don't have time to do that and they end up coming to the meeting having only skimmed the memo or maybe having not read it at all… it's better just to carve out the time for people so now we're all on the same page we've all read the memo and now we can have a really elevated discussion.

Next Steps

Let’s do some synchronous learning! DM me on Twitter if you want to find a time that works :)

If you find this topic interesting, consider watching my video and interview with Paul Raymond-Robichaud. He's a mathematician working at the foundations of theoretical physics and all of these ideas about how to structure math flashcards originate with him, not me!

• Spaced Repetition for Mathematics: A Quick Summary - YouTube

• Learning Math with Spaced Repetition: A Discussion with Paul Raymond-Robichaud - YouTube

While this article stands alone, it's part of a series I wrote called Mathematics for the 21st Century: An Introduction to Logic and Proof which you might be interested in.

What's Wrong with Textbook Proofs?

17th Century math proofs and formulas were written in prose. They were difficult to read and it was easy to make mistakes when writing them. Here's an example from Isaac Newton's Principia side-by-side with a proof of the irrationality of the square root of two you might find in a typical modern textbook.

Notice that they are very similar in terms of style - both are written in prose interspersed with some variables and equations.

In his paper How to Write a 21st Century Proof computer scientist Leslie Lamport criticises this style of writing proofs.

1. They are hard to understand because when reading a sentence in a proof it often isn't immediately clear whether it is justifying a previous statement or introducing a new one. Sometimes you have to read the entire proof just to understand the first sentence!

2. They encourage sloppiness that leads to errors. Leslie gives an example from Spivak's Calculus which is intended to be an introductory calculus textbook, but contains mistakes and confuses the reader with ambiguous assertions. “Mathematicians think that the logic of the proofs they write is completely obvious, but our examination of Spivak’s proof shows that they are wrong. Students are expected to learn how to write logically correct proofs from examples that, when read literally, are illogical.”

3. There is a tradeoff between making proofs more detailed and increasing complexity. Reducing ambiguity and making sure all of your statements are justified requires making your proof longer and consequently harder to understand.

Common Proof Flashcard Mistakes

A common mistake people make is to copy and paste prose proofs like the ones above directly from their textbook into a flashcard and try to memorise it line by line using cloze deletions.

The problem is that textbook proofs are quite inconvenient to remember with SRS. In addition to the problems pointed out by Leslie Lamport:

• Prose is complex and hard to remember.

• Clozing sentences in prose paragraphs doesn't support proof reuse.

• Textbook proofs tend to be long and don't break things down into enough sub-theorems.

• Textbooks can't have the best proof for your specific prior knowledge because they are generally intended to be self-contained and targeted at a particular area. You might prefer more general proofs, or proofs which reuse knowledge from other mathematical areas you've learned.

How to Write a Memorable Proof

We want to write proofs for the 21st Century so we will write hierarchically structured formal proofs in RemNote instead.

If you follow these two principles your proofs will be:

• More rigorous and correct.

• Modular and reusable.

• Shorter and easier to memorise.

Formal

By formal I mean: concise, unambiguous, inspired by formal logic. Here's a simple example from set theory.

I like borrowing the names of so-called "proof tactics" (like by contra) from Lean because they are concise and keep the language I use across proofs consistent. I have some other random style choices I use too like using the word suppose instead of assume for negation introduction. It's just personal choice. As you learn more math you'll also have to contend with notation overlaps and choose your own notation for consistency.

Every step in the proof should be something proven previously or the application of a rule of inference you have already mastered. The goal is to create new knowledge by reasoning about the implications of facts you already know. If you are familiar with language learning theory, this is similar to the i+1 part of Stephen Krashen's Input hypothesis where i refers to your prior knowledge and the +1 part refers to the new knowledge you are creating or learning in the flashcard. The point is that you are only adding one unknown per flashcard.

You can refine your style over time. Once you get familiar with these kinds of proofs you can add shortcuts, eg. skipping over simple assumptions and omitting extra details. You might prefer the following if you really want to keep things concise:

The goal is to train ourselves to think in terms of logic and natural deduction rules, but express ideas to other mathematicians in plain English. This allows us to reason precisely without sacrificing our ability to express ideas using higher-level explanations. In my experience so far, if you can write a proof formally like this, expressing it in plain English to share with another mathematician is no trouble.

“The best way I know to eliminate errors is to imagine that there is a curious child sitting next to us. Every time we write an assertion, the child asks: Why?” - Leslie Lamport

Hierarchical structure

“Computer science offers a standard way to handle complexity: hierarchical structure.” - Leslie Lamport

Inspired by Leslie Lamport's emphasis on Hierarchical Structure, we will use RemNote's graph structure and references to break down proofs into chunks and structure them as trees of bullets.

Instead of trying to memorise extremely long and complex proofs with lots of duplication, we'll use a divide and conquer strategy - we'll split them up into theorems and sub-theorems which are reusable chunks we can use as reasoning shortcuts in other proofs. The key to this is the ability to abstract the sub-theorems so that they are reusable. Reusability is valuable to us because it reduces our review burden significantly - we only need one proof flashcard to memorise a sub-theorem, then we can just make references to it in other proofs without needing to spell out in detail every time why it's true.

Notice in the proof above how the second line is a reference to another proof from first order logic. Set theory builds directly on top of first order logic, so all of those proofs can be reused to justify the proofs in set theory. If every step of the proof is just a trivial application of something you have already learned, it gives your brain no choice but to understand it. New knowledge slots in effortlessly to what you already know.

It's exactly like the DRY ("Don't Repeat Yourself") principle in software development. In the same way that programmers try to write programs composed of reusable functions rather than an enormous list of statements, we want to write proofs that are divided into small self-contained sub-theorems that we can keep referring back to in our higher-level proofs.

More Involved Example

A proof of part of one of De Morgan's laws. The negation of a conjunction implies the disjunction of the negations.

Here is the first attempt. It will look strange and ugly if you are reading this without having gone through my Mathematics for the 21st Century: An Introduction to Logic and Proof series because the way I like to write my proofs is inspired by the interactive theorem proving language Lean (a programming language for math). In general this works well, but as soon as you start nesting assumptions the proof becomes a mess!

• 🚫 The steps feel like symbol-shuffling rather than meaningful

• 🚫 It looks like code to be read by a machine rather than a human

• 🚫 There is quite a bit of nesting which adds complexity

• 🚫 It's too long

Here is my revised proof. Immediately you can tell it's much shorter. Extra details are tucked away inside references. If I need more detail I can hover over the hyperlinks to inspect the details of those proofs. Otherwise they remain hidden and I just have a high level overview.

• ✅ It’s very short

• ✅ Each step feels meaningful making it easier to remember

• ✅ It reuses other proofs I have in my collection rather than duplicating

Proof Reuse

Line 1. contains a reference to my proof of the contrapositive theorem.

Line 3. contains a reference to my proof of:

See how that proof then references other more fundamental proofs. Because each subproof is its own independent flashcard, we assume that we can recall the details if necessary and just rely on the abstraction in higher level proofs to make them shorter and easier to remember.

Proof Reformulation

Your proofs should always be evolving in the light of new knowledge. The proof above is a perfect example - it wasn't immediately clear to me that I could use the Contrapositive theorem to create a better proof. It was only after reviewing my other cards for a while and chatting with Paul that I realised I could shorten it.

One thing I found very interesting about how Paul uses SRS is that 90% of his new cards are created by rewriting his old cards. An important part of this process for him is constantly asking the question: "How can I generalise this?". An example of generalising your math knowledge could be recognising that many of the proofs in propositional logic and set theory are effectively the same. You could generalize those proofs by expressing them in terms of boolean algebra instead. Now you have a set of proofs with a wider "reach" than your original proofs because they apply to any mathematical domain which satisfies the criteria to be a boolean algebra.

Community

A fully comprehensive text on mathematics and SRS is not really possible. I cannot cover and discuss all cases because I don't have much knowledge beyond the basics. You need to try on your own and become a part of a community of people interested in using SRS for math to bounce ideas off of.

I personally benefitted a lot from being able to interview Paul (Learning Math with Spaced Repetition: A Discussion with Paul Raymond-Robichaud - YouTube) and send him questions about flashcard formulation when I got stuck. If you are interested in using spaced repetition and flashcards for math and want to join a community, then I recommend checking out the math channel in the RemNote Discord server.

Next Steps

Send me ideas and questions in the comments or on Twitter!

Metrics seem like an antidote to this uncertainty. They give us a target to aim for, allow us to infer trends and accelerate the feedback loop between action and reward. But at what cost? Over the past 6 years, I’ve repeatedly fallen prey to the siren call of blindly optimising metrics in pursuit of a goal. Everyone knows that metrics can be abused - popular aphorisms like Goodhart's Law and cautionary tales of breeding cobras to sell to the British authorities in colonial Delhi have made it common knowledge. But since there’s no way to compare your current situation with the counterfactual universe where you didn’t use metrics as your compass, not all of the downsides are obvious.

Strongly influenced by Kenneth Stanley’s book, the Myth of the Objective, I’ve been thinking recently about metrics and optimisation in the context of my daily routine. I admit that I am somewhat of a “freak moron” (to quote Zander) when it comes to planning my day to the minute, and I suspect that I may have driven myself into a local optimum. How do I escape?

Tunnel Vision

Here’s me reading the first paragraph from a draft of an article I’m writing about my experience planning my day during a podcast.

The trap I repeatedly fell into during those periods was an excessive focus on one goal at the expense of everything else. In secondary school, it was going to the gym, at the expense of grades and social life. In university, I spent 99% of the time in my room studying instead of making friends. Then after university, it was coding, while other areas of my life fell into disarray.

So there’s an obvious downside to investing all of your time and energy into one area which is that life is multi-faceted and the areas you ignore will either stagnate or regress. But there’s also a hidden cost. The act of setting a specific goal or target kills unexpected opportunities.

Take this bee for example. If the bee sets one particular flower as its target, it will ignore other areas of the environment, even if they contain interesting things worth exploring. The bee optimises for one at the expense of many.

If the bee doesn’t have a goal or uses a less specific goal, its path becomes much more open-ended. Instead of saying “I must visit that flower in particular” it could use a more abstract, generic goal like “I will explore beautiful flowers”. It’s hard to say in advance where the bee will end up, only that it will eventually find something interesting.

The difficulty is that the more open-ended and vague your goal is, the harder it is to explain or justify in concrete terms. For example, I have benefitted in many ways from using Twitter. But Twitter is exactly the kind of thing that gets cut from a routine which over-optimises for productivity, because to defend it you have to use terms like “serendipity” and “stepping-stone collection” which sound like unquantifiable bullshit, even though they are real.

Deception

The underlying assumption behind metrics is that we can rely on them to tell us when we are getting closer or further away from our goals. But like a faulty compass metrics can often lead us astray.

Here’s another bee analogy. If the bee’s goal is to get to the golden nugget, the most obvious metric it could track is the distance away from the nugget. The bee expects that it can use the metric to guide it closer and closer towards its goal. However, it fails to account for the structure of the environment. As long as it sticks stubbornly to its metric it can never escape the dead end local optimum its metric has driven it into.

If the bee wants to escape and achieve its goal, things must get worse before they get better.

I experienced this during my third year of university. I decided I hated the course I was studying and took a year out to live at home with my parents. This was tough because while I didn’t enjoy university, at least it made me seem as if I had a purpose. Taking a year out without a plan was tough socially because it was difficult to explain to curious friends and family what I was doing.

But the abundance of free time I had with little pressure to justify what I was doing gave me the freedom and time required to find what I was really passionate about. In other words it allowed me to cross through the adaptive valley and find a higher peak in the adaptive landscape.

Surrogation

Surrogation describes the replacement of a meaningful end goal with a metric.

An everyday example of surrogation is a manager tasked with increasing customer satisfaction who begins to believe that the customer satisfaction survey score actually *is* customer satisfaction. - Wikipedia page on Surrogation

When the metric becomes the goal rather than a mere indicator, you adjust your behaviour to optimise purely for that metric. And when you optimise purely for metrics, the process becomes disconnected from reality.

I noticed this frequently in my spaced repetition usage. I love spaced repetition systems (SRS) and I think they have the potential to change the world, but there are plenty of metric-based traps you can fall into. Take the classic flashcard streak heatmap for example. When you don’t focus on it too much, then it’s a useful measure of consistency which is important in SRS because flashcards are scheduled to be reviewed at particular times and missing those times impacts your learning efficiency.

But if you let your heatmap and streak get into your head, you stop seeing meaningful goals as the purpose of your reviews, and instead your reviews become a grind to beat your high score at the flashcard slot machine without it ever culminating in a payoff in the real world.

I’ve certainly fallen into the trap of persisting with reviews of certain branches of my flashcard collection out of a mixture of the sunk cost fallacy as well as a sense of dread at breaking a long streak. But what’s the point of maintaining a streak for the sake of it? Should I feel good about myself if I dutifully persist through a boring 30 minutes reviewing cards I hate? There are a million other things I could be doing, why don’t I choose to do them instead?

So what’s the solution?

I don’t really know. If you have any good ideas, DM me on Twitter so we can discuss them. I simultaneously feel a strong urge to start religiously planning my day again, but I also don’t want to make the same mistakes I made before. Is it possible to strike a balance between metric following and open-ended exploration?