I want to discuss two programs, one the program that controls a vibrator, and one that controls a pacemaker.

First pseudocode for a program of a vibrator:

def fib(n):
  if n <= 1:
    return n

  return fib(n-1) + fib(n-2)

def pulse(ms, freq):
  start_vibration(freq)
  sleep(ms)
  stop_vibration()

i = 0
while True:
   pulse(2000, fib(i))
   i += 1
   sleep(1000)

This program will vibrate with increase frequency using the Fibonacci numbers 0, 1, 1, 2, 3, 5, 8, 13, 21, 34.. (there was a bug in the code, and by bug I mean it was completely fubar, spotted by tzs, makes the point of transparency even stronger)

Pacemakers are implant devices that have electrodes connected to chambers of the heart and can force it to contract and pump blood. Its goal is to create stable heart rate, of course the heart might pump without it, so it also needs to monitor what is going on and when it should send an electric pulse. This is how much we trust other people's programs, we allow them to control our heart.

Can you imagine? Just think of the simplest part of a pacemaker: "send a 2ms pulse every 810ms", ignoring the inhibited mode where it senses cardiac activity and does nothing if present, but focus on just sending regular pulse to maintain 74 beats per minute. A naive and absurdly reduced pacemaker program:

def pulse(ms):
    start()
    sleep(ms)
    stop()

while True:
    pulse(2)
    sleep(810)

Notice how similar it looks to the vibrator program: loop, pulse, sleep. On the surface the vibrator program even looks more complicated because of the Fibonacci sequence.

Oh, I forgot to say, the pacemaker's computer and battery are inside the human body.

If the battery runs out they have to cut it out.

If you read some pacemaker code you will see in the comments they put not only how many cycles an instruction takes but also how much current it will use, its very impressive. The real pacemaker code is ridiculously complex, tens of thousands of lines, true fault tolerant code is very difficult. It has taken many many years of talent and politics to make those kinds of programs possible, standards like ISO 14971, or IEC 62304:2006, IEC 60601, ISO 13485, DO-178C and hundreds more. Thousands of people have spent their life thinking: How do you write programs when a bug can kill their user?

Lets make an argument, who will write a better program, djb (Daniel J. Bernstein: a brilliant programmer) or FDA audited and ISO certified contractor company. Or maybe ask, which pacemaker would you buy? Or car? At some point we go down the critical list you will chose djb for sure, but when your life depends on it? Most people will prefer the FDA audited program.

Now, imagine in 1 year we have new AI models, and we are absolutely convinced that they can actually program. They go beyond tokens and patterns. Now, which code would you choose if you can use the all-seeing AI that can look at the code, each hardware component and their interconnection, the manufacturing process, the QA process? I will choose the "better" program, completely agnostic to who wrote it.

The same AI can of course write the program itself, now what would you choose? AI, djb, FDA? It should be possible that holistic AI writes better code than human specialists that communicate through abstractions and interface. Imagine the AI is djb in software, hardware, electrophysiology, in biology, biochemistry and physics.

At this very moment Opus 4 can absolutely code a vibrator together with its user, but what is the path to coding a pacemaker?

What about programs that are not individual, but affect all of us? What about people driving AI programmed cars? And I don't mean in comma.ai or Tesla way, I mean saying something like "hey the new v-jepa is out, can you retrain and condition on it, and lets hit road when you are ready", and then getting in their car with their fresh model and not crashing with other people's newly programmed cars. It will lead to complete chaos. Not even self driving, imagine if just the ABS is programmed by AI and the car's user.

How do we get to a point to `trust` it? I am not sure, I just know its not solving SWE-Bench. There is of course a whole spectrum of programs between the vibrators and pacemakers, and SWE-bench will do very well for most, but when push comes to shove, programs leak outside of the computer, into the body, a world model is needed to make a pacemaker program, most of it lives outside of RAM.

To be honest, my threshold is quite low, I think in 1 year we will be able to take any piece of code and audit it with `claude`, and I will put my life on the line using it. So my requirement will be the system must be open and all code and specs must be available. But.. what will it take to put my daughter on a car with AI programmed breaks, even if its open and my oracle says its amazing? My instinct is, no fucking way, but I also know 1 bug per 5000 lines of code is the pinnacle of human achievment. The pressure to produce is immense, all organizations are racing, beurocracy has its limits, the 737 MAX issue is an example. We are not ready for the flood of AI code, before AI can actually code.

Radical transparency is the only answer. I am convinced that open code, specs and procesees must be requirement going forward.

Trust is taken it is not given.

PS: I rewrote this post like 3 times, when I started writing, it was about personal programs, and allowing people to program their own vibrators or whatever, but as I was writing it I discovered that I am truly horrified that my car's breaks will be programmed by a contractor using some local 7b model that specializes in writing MIRSA C:2023 ASIL-D compliant software.
PPS: I learned a lot about pacemakers though, and really enjoyed reading some real pacemaker code.