← jackdoe
CODE AND TRUST: vibrators to pacemakers
I want to discuss two programs, one the program that controls a vibrator, and
one that controls a pacemaker.
First pseudocode for a program of a vibrator:
def fib(n):
if n <= 1:
return n
return fib(n-1) + fib(n-2)
def pulse(ms, freq):
start_vibration(freq)
sleep(ms)
stop_vibration()
frequency = 2
while True:
pulse(2000, frequency)
frequency = fib(frequency)
sleep(1000)
This program will vibrate with increase frequency using the Fibonacci
numbers 2, 3, 5, 8, 13, 21, 34..
Pacemakers are
implant devices that have electrodes connected to chambers of the heart and
can force it to contract and pump blood. Its goal is to create stable heart
rate, of course the heart might pump without it, so it also needs to monitor
what is going on and when it should send an electric pulse. This is how much
we trust other people's programs, we allow them to control our heart.
Can you imagine? Just think of the simplest part of a pacemaker: "send a 2ms
pulse every 810ms", ignoring the inhibited mode where it senses cardiac
activity and does nothing if present, but focus on just sending regular pulse
to maintain 74 beats per minute. A naive and absurdly reduced pacemaker program:
def pulse(ms):
start()
sleep(ms)
stop()
while True:
pulse(2)
sleep(810)
Notice how similar it looks to the vibrator program: loop, pulse, sleep. On
the surface the vibrator program even looks more complicated because of the
Fibonacci sequence.
Oh, I forgot to say, the pacemaker's computer and battery are inside the human body.
If the battery runs out they have to cut it out.
If you read some pacemaker code you will see in the comments they put not only
how many cycles an instruction takes but also how much current it will use,
its very impressive. The real pacemaker code is ridiculously complex, tens of
thousands of lines, true fault tolerant code is very difficult. It has taken
many many years of talent and politics to make those kinds of programs
possible, standards like
ISO
14971, or
IEC
62304:2006, IEC 60601, ISO 13485, DO-178C and hundreds more. Thousands of
people have spent their life thinking: How do you write programs when a bug
can kill their user?
Lets make an argument, who will write a better
program,
djb (Daniel J. Bernstein: a brilliant programmer) or FDA
audited and ISO certified contractor company. Or maybe ask, which pacemaker
would you buy? Or car? At some point we go down the critical list you will
chose djb for sure, but when your life depends on it? Most people will prefer
the FDA audited program.
Now, imagine in 1 year we have new AI models, and we are absolutely convinced
that they can actually program. They go beyond tokens and patterns. Now, which
code would you choose if you can use the all-seeing AI that can look at the
code, each hardware component and their interconnection, the manufacturing
process, the QA process? I will choose the "better" program, completely
agnostic to who wrote it.
The same AI can of course write the program itself, now what would you choose?
AI, djb, FDA? It should be possible that holistic AI writes better code than
human specialists that communicate through abstractions and interface. Imagine
the AI is djb in software, hardware, electrophysiology, in biology,
biochemistry and physics.
At this very moment Opus 4 can absolutely code a vibrator together with its
user, but what is the path to coding a pacemaker?
What about programs that are not individual, but affect all of us? What about
people driving AI programmed cars? And I don't mean
in
comma.ai or Tesla way, I mean saying
something like "hey the new v-jepa is out, can you retrain and condition on
it, and lets hit road when you are ready", and then getting in their car with
their fresh model and not crashing with other people's newly programmed cars.
It will lead to complete chaos. Not even self driving, imagine if just the ABS
is programmed by AI and the car's user.
How do we get to a point to `trust` it? I am not sure, I just know its
not solving
SWE-Bench. There is of
course a whole spectrum of programs between the vibrators and pacemakers, and
SWE-bench will do very well for most, but when push comes to shove, programs
leak outside of the computer, into the body, a world model is needed to make a
pacemaker program, most of it lives outside of RAM.
To be honest, my threshold is quite low, I think in 1 year we will be able to
take any piece of code and audit it with `claude`, and I will put my life on
the line using it. So my requirement will be the system must be open and all
code and specs must be available. But.. what will it take to put my daughter
on a car with AI programmed breaks, even if its open and my oracle says its
amazing? My instinct is, no fucking way, but I also know 1 bug per 5000 lines
of code is the pinnacle of human achievment. The pressure to produce is
immense, all organizations are racing, beurocracy has its limits, the 737 MAX
issue is an example. We are not ready for the flood of AI code, before AI can
actually code.
Radical transparency is the only answer. I am convinced that open code, specs
and procesees must be requirement going forward.
Trust is taken it is not given.
PS: I rewrote this post like 3 times, when I started writing, it was about
personal programs, and allowing people to program their own vibrators or
whatever, but as I was writing it I discovered that I am truly horrified that
my car's breaks will be programmed by a contractor using some local 7b model
that specializes in writing
MIRSA C:2023 ASIL-D compliant software.
PPS: I learned a lot about pacemakers though, and really enjoyed
reading some real pacemaker code.