This is my post for day 17 of the Inkhaven writing retreat.

Every so often, I have this conversation:

Them: So you know how the other day we talked about whether we should leave for our trip on that sunday or monday?
Me: …doesn’t sound familiar…
Them: And you said it depended on what work you had left to do that weekend…
Me: Hm… where were we when we had the conversation?
Them: Um… we had just arrived at my house and I had started making food-
Me: Ooooh yeah yeah okay. And I was sitting on the black stool facing the clock. Okay cool, I remember the conversation now, please continue.

…What the heck is up with this? Does it happen to anyone else? Apparently, my brain decides to index conversations to be efficiently looked up by quite precisely where I was in physical space when the conversation occurred. I have no conscious experience of this indexing happening. It’s also pretty strange that it happens for locations that I use on a regular or even daily basis; it’s not like I could just start listing all the conversations I’ve had while sitting on that kitchen stool.

I do believe that I’m quite above-average aware of what’s happening in my visual field. I always notice when people come in and out of a room, I tend to see new objects or decor right away, and I somehow spot every insect. I’m often the first to spot a leak or mold. I almost never run into stuff or knock things over. So maybe it’s just increased attention to my surroundings?

---

Here’s a similar pattern I’ve noticed.

I’m in a phase of my life where I read a lot of books, and especially textbooks. My field of study is interdisciplinary, and I am frequently looking up something that I’ve read before. When I do, I will frequently have the sense of roughly where it was, physically, in the book. This includes:

• how far into the book,
• whether it’s on the left or right page,
• how far down the page,
• roughly where within a paragraph it is,
• and a vague sense of what the rest of the page looks like.

To be clear, I’m not claiming that I have any kind of “photographic” memory. I have no idea what almost all of these books say. I don’t have any degree of verbatim retention. But when I remember that there was a particular interesting part and want to go look for it, my brain brings up these visuo-spatial associations. These associations feel blurry but confident, like some kind of hash function. Textbooks are heavily formatted, so there will be lots of white space, diagrams, section headers et cetera to anchor off. When I try to recall the “location” of events in flat prose fiction books, nothing comes up.

This is, I think, one reason why I have struggled to switch over to digital forms of books. I’ve tried it a lot, but they always fade out of use. There are many other reasons (if they’re not on my shelf I tend to forget it exists, I find physical books far easier to skim) but the fact that I can’t physically index my knowledge to it is noticeable. It’s just some big infinite scroll that looks and feels indistinguishable from all the other big infinite scrolls.

---

I’d love to hear how others relate to either of these experiences!

This is my post for day 16 of the Inkhaven writing retreat.

I knew I wanted to do science and math from a very early age. And I didn’t want to spend my life investigating just some particular phenomenon; I wanted to understand “everything”. You obviously can’t do that in a literal sense, so I focused on understanding things that were increasingly general. Generalizations are in some sense more “efficient” ways of understand things.

In physics, the field that claims to see the “theory of everything”, there are two obvious directions you can go with this. One is “up”, to astronomy and cosmology and the overall structure of the universe. The other is “down”, where you can look at the smallest particles. I was very into both of these, though it’s clear that the “down” direction is in some sense more fundamental. If you understand the laws behind the behavior of the smaller things, you can, in theory, use them to calculate what will happen to the bigger things. At 12 I knew that people had made huge progress toward finding the fundamental physical laws, and I was very excited to catch up on it.

But there also seem to be some other “directions” to generalize in if you want to efficiently understand everything. Mathematics is one of them, which is something like the “symbolic” direction. Philosophy is perhaps in the “conceptual” direction. And there’s also a direction that is something like the study of yourself, of how minds work. The study of what’s up with being the kind of thing that is inside the universe, observing and trying to understand it. This is a meta type of direction.

Over the years my preferences between these has shifted, but my 12-year-old self would not have been too surprised if I ended up going deep in any of these directions. Treatise on ontology? Awesome. Unified theory of the neocortex? Let’s go. But what’s agent foundations?

Well, I moved into agent foundations because I decided we needed to solve a problem, namely existential risk from AI. But it’s mostly trying to help with that problem by figuring out what the heck is going on with the phenomenon of agents. Which is to say, by understanding it.

I think my younger self would be pretty confused for a while that I’m into this, but I could probably explain it given enough time.

Above I mentioned the study of physics at the biggest scale and the smallest scale. There is obviously a lot going on in the middle, but from some perspective it feels mostly arbitrary. Like, there just happens to be water and flowers and binary star systems. All those things are interesting insofar as they are in the set of “everything”, but it doesn’t feel like understanding them has much generalization power.

I claim (to my 12-year-old self) that there is actually a generalized theory of things going on in the middle. That is, a generalized theory, not about specifically what’s going on in the middle, but about what it means that something could be said to be going on in the middle. (That sentence may have lost the reader. It also may have lost my 12-year-old self, but he’s now very excited to understand what I meant by it.)

For example, what exactly does it mean when we say that Newtonian mechanics is a good approximation of the true laws of physics? If you handed someone only the true laws of physics, how could they have figured out, in principle, that Newtonian mechanics was a good approximation of what they were holding? Are there other possible good approximations they could have figured out instead? Can we well-define the set of all possible good approximations, given some true physical laws?

This is relevant to agent foundations because agents have models of the world inside them. They use these models to successfully achieve their goals, so the models must be good approximations by that standard. If we want the agent to achieve our goals, then it probably needs a world model that is compatible with ours, at least in the parts that describe our goals.

We currently do not know how to formally state this, and I think that’s a barrier to being able to ensure it in practice.

Another word you could use for “world model” or “good approximation” is “theory”, so in some sense this part of my work is studying the theory of theories. Which, yeah, my 12-year-old self would be pretty thrilled about.

This is my post for day 14 of the Inkhaven writing retreat.

My boyfriend and I live four short blocks away from each other. An eight minute walk. We are in our eighth year. I walk there often, most often at night, around 7:30. For some part of the year, this is during sunset. Sunset is when the neighborhood cats come out. Donut lives closest to me; he’s a black cat, the rockstar of the neighborhood. A couple houses to the right is a crazy grey and white cat. It will chase you and tackle you, harass you and hiss at you. But in a playful way.

I leave my door and turn left, walking half a block toward the sun and the bay. Then I turn left again. At this house, there’s a Little Free Library box. I always stop and look in it. The books are different almost every day. I’ve wondered if the owners cycle the books. I can’t imagine passersby taking them all. I once left a copy of the Epic of Gilgamesh in the box. It was the Penguin edition from 1960. It was outdated; since then, we’ve found many more tablets, which add to our knowledge of the Epic. The book was gone the next day.

Then I go through the park. The park is very long and skinny, and I only cross it for half a block. I cross the first street. This one is reasonably low-traffic, but the visibility isn’t great. I walk down the second block. There are memories here, but they’re all faint. There were bees once. I look into the house that never closes their curtains. I rarely see them in there. I think they’ve moved out, now; the curtains have started being closed sometimes.

I cross the second street. This intersection has barriers that turn it into two separate right-angle turns. Only emergency vehicles should drive straight through. This makes it almost always empty of cars, much nicer to cross. Half the time that I encounter cars here, they’re making a mistake and turn around.

I get to the next street. On the left is the Mexican tile & ceramics store. Every item in this store is heart-achingly beautiful. I’ve bought some from there before. But I mostly don’t have need for tiles.

Crossing this street is the worst. It’s super wide. The pavement reflects the sun and it’s hotter somehow and the whole world suddenly feels like Los Angeles. There’s no stop light. There’s only the lights where you can press a button and make them blink. The blinking is not particularly persuasive.

Crossing this street is the worst, but it’s better than going the other way. The other way, I’d have to cross the worst intersection I know of. That intersection turns drivers psychotic. I’ve had to jump out of the way of the cars more than once. Those same cars are the ones that also cross this intersection, but by this point they’ve regained their sanity. That intersection has a stop light and a pedestrian cross light. Apparently, it is not particularly persuasive either.

Crossing this street is the worst, but it’s the last one.

I cross the street and pass a burger place like an off-brand McDonalds. I’ve never been in. Just the other day I realized that it’s open 24 hours a day. I don’t know anyone who goes there.

At the end of the block I look into the other house whose curtains never close. There is what looks like a stripper pole in there, but I’ve never seen anyone use it. I almost never see anyone in there. I think new people have moved in, now; I see them, and they close the curtains sometimes.

I turn left, and walk toward my boyfriend’s house. There are three ways to get into his apartment. If I go through the front door, there might be people hanging out in the living room. It’s easy to get caught up in conversation with them. But I’m not here for that; I’m here for him. My favorite way is going up the driveway and into the back door. It’s closer to his bedroom. But if I arrive after dark, my passing makes lights turn on. Click, click, click, click. Super bright. Too much attention. I’m trying to remember to go through the gate down the other side. The gate used to be closed all the time. It’s open now, but it still feels weird to walk past the downstairs unit’s door.

He’s always surprised when I show up. He never checks his phone.

Sometimes he comes right back with me to my house for the night. We call it doing a fetch. The other day, we did this just when it started downpouring. We were prepared, me with a raincoat, him with a poncho. We splish-splashed in the puddles and torrents all the way home. It was the best part of my day.

This is my post for day 12 of the Inkhaven writing retreat.

I place a high value on humanity’s cultural heritage. I find the question of what exactly makes it valuable quite interesting and unclear. According to my theory of metaethics, one doesn’t need to justify all of one’s values. But often there are deeper reasons, and it can be very useful to introspect on one’s values.

For me, here are some things that make artifacts valuable;

• They give us information about what, specifically, happened in human history.
• They give us insight into human nature.
• They’re aesthetically beautiful.
• They’re old. I’m unclear how much I value this on its own, but there’s something to it.
• They give us connection to people long gone.

There’s aspect in particular about this type of value that confuses me. It seems to get created at some point, but this point is very unclear.

Almost all cuneiform tablets are the equivalent of paperwork. They were essentially worthless at the time of creation. They continued to be essentially worthless for quite a while; as cool as it is to see your great-grandmother’s receipts from the grocery store, they are not actually rare, sentimental, or insightful about the human condition. But somewhere along the line, we stopped writing in cuneiform, and then lost knowledge of how to read it, and then lost knowledge of its existence, and then lost knowledge of the very civilizations that created it in the first place. So now, cuneiform tablets are highly valuable, despite the fact that we have found hundreds of thousands of them.

This phenomenon is a little confusing to me, but it gets much more confusing to me if we layer it. The Metropolitan Museum of Art in New York City contains a large room devoted to displaying a small ancient Egyptian building in its entirety: the Temple of Dendur. This is a straight-forward example of a culturally valuable artifact. But on the side of the temple is graffiti. Terrible! Someone damaged the value of the temple by writing graffiti on it. …or did they? It turns out that the graffiti is 200 years old. It exists because the Napoleonic campaigns in Egypt brought a wave of European tourists into the country. Like it or not, Napoleon’s actions were of enormous historical consequence, and thus have become part of humanity’s cultural heritage.

If some punk visiting the Met used spray paint on the temple, they would surely be harshly reprimanded, the action universally denounced, and the paint cleaned up. …but we if we waited another 500 years? Would the action then be considered an enrichment of the history of the temple?

In Berkeley there is a tendency to call for the historic preservation of houses whose cultural value is highly dubious. If some quirky author lived here for a bit in the 60s, that does not really justify cordoning off that plot of land indefinitely, especially when there are tens of thousands of people who could be using that land to, you know, live.

This tension also exists in a continual way for cities which were of immense importance in ancient times and which, well, never stopped being swarmed with people, because they’re cities. Examples include Jericho, Rome, London, and Mexico city, just off the top of my head.

I want to preserve the cultural artifacts that humans produce. And I want humans to keep being able to live and produce more artifacts. And they should be able to keep living where they have been living. This all feels like a big tangled conundrum to me. I’d like to hear more people talking about it.

This is my post for day 11 of the Inkhaven writing retreat.

Sometimes it’s hard to tell what we will love. A life-changing career or hobby could be right around the corner, or right under your nose. As a seventh grader, I sneered at my friends for collecting Pokemon cards. Weeks later I was begging my parents for booster packs.

Sometimes it’s all around you and always has been. Let me tell you about how I met coffee.

I should start by saying that I’m a 99.99th percentile picky eater. (I give that number because I would guess that I’ve been acquainted with roughly 10,000 people, and I’ve only ever met one person who was more picky than me, and he had gotten over it by his early 20s.) Me and food is a topic for another time, but needless to say, trying new foods has essentially never been an enjoyable experience. I’ve always loved the “coffee flavor”, mostly in ice cream. And it smelled amazing. But it was far down the list of things to try.

When I was 20, I got a job at Starbucks. As part of the job, they ask you to try each of the drinks once. This was where I learned that you can basically create a drink version of coffee ice cream. Since then I’ve slurped down quite a number of iced tall decaf breve one-pump white mochas (no whipped cream). But the black coffee did not get added to my list.

Fast forward a decade. While continuing my quest toward higher agency, I decided it was finally time to take more seriously the option of regular stimulants. Caffeine was high on the list, since it is a consumer good and has extremely low side-effects. I took caffeine pills occasionally for a few months, and then at some point I remembered that coffee existed. It seemed like people had a good time with coffee. There was a whole coffee culture. Maybe I could be having a bit more fun with my caffeine?

I had my first deliberate tasting of coffee in February. I didn’t take to it immediately. It’s actually hard to remember how drawn out this part was, and I only know because fortunately I took fairly extensive notes about it. I was already keeping a stimulant journal, and when I started drinking coffee, I started recording some of the other parts of my experience, like how it tasted, and whether I liked it.

It was not until my eleventh coffee that I finished a whole cup. It’s not like I LIKED it, though. It was just… interesting.

I had a good friend who was a BIG coffee nerd. We’re talking “has a tabletop home roasting machine” levels of nerd. And I just so happened to be coworking with him at his house every monday. In between my sampling from random cafes, he introduced me to specialty coffee. That is, single-origin, light roast coffee.

Over the next few months my notes become peppered with increasingly many exclamation marks. But it is quite a while before I start to interpret my experience as “tastes good”. Instead, there is faint praise like “very drinkable”, “First sip is better than yesterday”, and “maybe the most palatable one I’ve had so far”.

By July, I am writing notes about how the whole thing is extremely interesting, even though it really did not map onto what I would call “tastes good”. Writing now, years later, with much longer hindsight, I am not sure what this phase was about. Perhaps what happened is that I was thrown into the deep end of a very high-dimensional sensory world, and just stayed disoriented for a long time. But since then I have become a twice-daily black coffee drinker, and it has added so much color to my life.

One mystery is how a 99.99th percentile picky eater could come to enjoy such a notoriously acquired taste. I now believe that I got quite lucky. The major types of flavors and tasting notes in light roast coffee — acidity, Maillard products, grains, chocolate, caramelization — are all flavors that I already liked. Texture is a big reason I dislike foods, but coffee is totally homogeneous. It’s also worth noting that light roast coffee does not taste bitter to me at all. Dark or even medium roast still does. But lots of my friends say that my coffee tastes bitter, so I may have gotten lucky with my particular gustatory perception, there.

And, let’s be real; the caffeine probably helped.

There’s a lot more I could say about my relationship with coffee. But that’s how we met.

This is my post for day 10 of the Inkhaven writing retreat.

“The ending was terrible.” You’ve probably heard this opinion many times. People will often claim that the ending ruined something, like the Game of Thrones TV series or the Mass Effect video game.

Whether it’s a book, movie, or DnD campaign, story-telling is hard, and wrapping everything up nicely is one of the hardest parts.

Over the years I have struggled to enjoy many popular movies, especially big climatic action movies, which seem to be addicted to escalation and whose endings defy any sense whatsoever. The bad guys can’t aim, the hero gets physically stronger just by resolve, and the power of love always saves the day.

This happened to me over and over, and got to a point where I started feeling like maybe I should just stop consuming popular media. Eventually I realized that I didn’t have to take the story all-or-nothing. I could just… “pretend” it didn’t end that way.

I think I first had this thought when I saw a typical Hollywood movie about AGI, and it somehow managed to get most of the details right, according to me. The AGI lived in a data center and not in a robot. The AGI stayed low-profile while it developed the technologies to control infrastructure. It built solar farms and nanotech. Most of the humans were unaware of it for most of the time. But then at the end of the movie, something weird happened, and then love saved the day. It was like someone designed a movie optimized for breaking me on this particular point.

So then I thought, fine. If you’re going to ruin your movie, I’m just going to, I dunno, revoke your right to tell me how the story ends, or something. Instead of deciding whether “I like the movie” was true or false, I just decided to carry around the part that I liked and throw away the ending.

The reason this feels weird to me is because you’re not allowed to pretend that things aren’t true. You’re not allowed to decide “I love America” by deliberately ignoring the parts about slavery and indigenous mistreatment and the Vietnam war. You’re not allowed to ignore the icky parts of true things, because then you’ll make wrong predictions and take worse actions, and then you won’t achieve your values.

But a movie isn’t an event that literally happened. I’m not advocating that you go around claiming and believing that The Matrix literally had no sequels. You should still maintain your beliefs about actual facts, like what types of media humanity tends to produce, or whether your friends liked the ending.

But if you find a painting in a thrift store that you absolutely love, except for this one weird dog that creeps you out, you can just paint over it. It’s allowed. It’s not a lie to change the painting, because the existence of the painting is not a proposition about reality.

The purpose of a story isn’t to make an assertion about what happened. The purpose of a story is — well, there are a lot of purposes. It can encode societal wisdom in a memorable way, or serve as hopeful inspiration of what the world could be like, or be a cautionary tale, or help you understand the inner experience of other people, or just be a rollicking good time. But none of the purposes obligate you to swallow the story as a whole or spit it out. Some group of people made up the story. You’re allowed to re-make up the story.

Though I explicitly noticed this option for the AGI movie, I’d been doing this already without realizing it. I absolutely love the movie 2001: a Space Odyssey, despite the ending making the least sense ever. The opening shot is ecstatic. HAL 9000 was a unique and formative representation of AI. The special effects were stunning. Whenever I thought of the movie it was a fond thought.

And really, this goes for anything you don’t like about a piece of media. It doesn’t have to be the ending. You can love a setting and throw away the characters. You can decide that the villain should have won, because it makes for a more poignant story. You can decide that the villain is actually the hero, because the author is wrong about morality. A lot of people operate this way, c.f. the entire fanfiction community.

This is something I also did unconsciously with the Indiana Jones franchise. When I rewatch the movies, I notice that there is some concerning treatment of side characters, and reckless handling of priceless artifacts, and that somehow every supernatural entity is real. But when I reflect on the movies, I feel fondness about the idea that an intellectual can also be adventurous and brave, that humanity’s collective cultural heritage is held as sacred and to be protected, and that Harrison Ford is incredibly dashing in a fedora.

I’m not sure why I manage to do this sometimes but not other times. I care so strongly about my practice of understanding how to make sense of reality, that I think I can get out of practice of enjoying stories.

Stories are creations made from innumerable ingredients. Use the ingredients that you like and bake your own cake.

This is my post for day 9 of the Inkhaven writing retreat.

Finding treasure at the British museum

I recently visited London for the first time. I didn’t have much free time to do sight-seeing, but if there one thing I was going to visit, it was the British museum. Among the tablets and tapestries, there was a gallery holding an exhibit on currency. The exhibit spanned everything from cowry shells to gold doubloons to Zimbabwean trillion dollar bills. But the item that caught me by surprise was a £50 note.

Not being from the UK, I wasn’t sure if this was a standard £50 note being displayed for comparison purposes or a special edition one. A quick google told me that it has been the standard issue note since 2021. It featured computer scientist Alan Turing. This was awesome to see. It reminded me of how the portrait on the US $100 note is not a former president, but instead scientist (and founding father) Benjamin Franklin. But what really struck me was what was beside Turing.

It was a table specifying a Turing machine.

Reader: I don’t know how to convey to you my excitement about this table of letters. Turing machines are practically a religious symbol for me. Seeing this symbol prominently featured on the largest-denomination note of such an important currency was like seeing my religion validated. And it wasn’t just the pretty graphical version; it was an actual table. I needed to get to the bottom of this.

Following the paper trail

I was leaving London soon, but I decided that I was willing to pay £50 for this very cool souvenir.

Being a normal, modern person, I had not actually had reason to acquire any physical British cash while in London. So I decided that I’d try to get a £50 note when I inevitably walked by the currency exchange desk at Heathrow airport. This worked fine, except that I had to get three £20s from the ATM and then exchange them for a £50 and a £10. I spent most of the £10 on airport snacks.

After I arrived home, I finally got around to looking up what the internet had to say about this design. Since I’m pretty familiar with Turing machines, the notation on the note did look familiar, and I had a pretty good guess that it was from the paper in which Turing first introduced his machines. (Of course, he didn’t name them after himself; he called them “automatic machines” or “a-machines”.) I can never quite remember the title of this paper, because the title is On computable numbers, with an application to the Entscheidungsproblem. The word “Entscheidungsproblem” is German for “decision problem”, and Turing used the German word because of the profound influence of Hilbert’s program.

The Bank of England’s official website for this note says the following;

The design on the reverse of the note celebrates Alan Turing and his pioneering work with computers. It features:

• A mathematical table and formulae from Turing’s seminal 1936 paper “On Computable Numbers, with an application to the Entscheidungsproblem” Proceedings of the London Mathematical Society. This paper is widely recognised as being foundational for computer science.
• The Automatic Computing Engine (ACE) Pilot Machine which was developed at the National Physical Laboratory as the trial model of Turing’s pioneering ACE design. The ACE was one of the first electronic stored-program digital computers.
• Ticker tape depicting Alan Turing’s birth date (23 June 1912) in binary code. 
• Technical drawings for the British Bombe, the machine specified by Turing and one of the primary tools used to break Enigma-enciphered messages during WWII. 
• The flower-shaped red foil patch on the back of the note is based on the image of a sunflower head linked to Turing’s morphogenetic (study of patterns in nature) work in later life.
• A series of background images, depicting technical drawings from The ACE Progress Report.

Which is all lovely, but I notice that it does not actually say what the table means.

Looking through other search results, many people were talking about what this choice of portrait meant in relation to Turing’s former conviction for homosexuality and posthumous pardoning. Or they were joking about how only criminals use £50 notes. But there was essentially no one talking about the technical details.

The paper itself was easy to find, and from a quick visual skim I quickly found the table that matched the note.

Since the sentence immediately before says “The lines of the table are now of the form”, I realized that this table does not specify a Turing machine, but instead just shows us how the rules of a Turing machine can take one of three forms.

Digression on how Turing machines actually work

I might as well take this moment to give you a quick description of how Turing machines are defined.

They are an extremely simplified abstract model of computation. The computing “machine” has a finite number of “internal” states, and can access an unlimited “external” memory in the form of one long tape. The tape is made of discrete cells. Canonically each cell holds either a zero or a one, but you could have a fancier tape if you wished. At any given time, the machine is in one of its states, and can read one cell of the tape. Each state is just a rule about what the machine will do next. An example rule looks like this;

if tape cell = 0
    write 0 to the tape cell
    move read head left
    go to state 3
if tape cell = 1
    write 0 to the tape cell
    move read head right
    go to state 2

All the states are exactly like this, except they differ in what they write, which way they move, and what state they go to next. In the notation on the bank note, each q is a name for a state, each S is a name for a symbol of the tape, L means “move left” and R means “move right”. The last q number is the state we should go to next.

So the table is just saying that, according to this particular notation, there are three different types of rules: ones where you move left, ones where you move right, and ones where you don’t move at all. That’s why it’s not a specification of a particular Turing machine.

Under the table

But below the table, the note also has this line;

which is not immediately below the table in the paper. Instead, it’s in the middle of the next page.

This page is just showing how to convert between several different ways of notating a Turing machine, which are variously used in other parts of the paper. Earlier on the page it says “Let us find a description number for the machine I of §3.” So to find out what this specific machine does, let’s head back up to section 3.

But before we go there, since we now know roughly how this notation works, let’s try to reason it out for ourselves. Some Turing machines have behavior so complicated that the only way know what they’ll do is to run them. But sometimes you can eyeball the rules and see that it’s simple.

The first thing I noticed is that it’s specifying rules for four states, q1 through q4. The semicolons are separating the rules.

The second thing I notice is that the goto-states are also labeled q1 through q4, so this could conceivably be a complete Turing machine.

But the third thing I notice is that the read-symbol in all four rules is S0. That means the machine is underspecified; if we give it a tape with S1 on it, it will not have a rule for what to do.

The fourth thing I notice is that the move-symbol for all four states is R. That means that no matter what, the tape head is moving right. So this is basically a “print-only” machine.

Final answer

Putting these things together, we can conclude that, as long as we start the machine on a tape filled out with S0 in every cell, then it will just print something and keep moving right. But what will it print? Since there are a finite number of states, it must print something periodic. Now that we know the machine is very simple, we can confidently trace out the exact steps of the machine.

Assume we start in state q1 with a tape full of S0. State q1 prints S1 and then moves to state q2. State q2 prints S0 and moves to state q3. State q3 prints S2 and moves to state q4. (It looks like we have three tape symbols, so Turing opted for a slightly fancier tape in this example.) State q4 prints S0, and finally moves us back to state q1. So, dropping the S, the machine just prints this;

102010201020…

A little disappointing, but at least it’s well-defined, and not the most trivial possible Turing machine.

Now let’s check out work, and read what the paper says.

3. Examples of computing machines.
   I. A machine can be constructed to compute the sequence 010101….

Huh… well, we were close. We thought it was a machine that printed a pattern of period 4, but apparently it’s a machine that prints a pattern of period 2.

If you read through enough of the paper, you find out that Turing is deliberately putting in “spacers” between every printed cell as a sort of scratchpad cell, which is erased at the end of the computation. But this is just a trick that is very useful for an example machine later in the paper. It’s not essential for the definition of Turing machines. So I claim that our original guess is a more accurate description, and that the Turing machine on the bank note is one that prints 1020 repeatingly.

If you’re interested in understanding this paper more deeply, the book The Annotated Turing by Charles Petzold is a very gentle line-by-line journey through the entire paper, including much historical context. If you’re comfortable with any formal mathematics, the original paper itself is quite readable.

---

All this makes me wonder how exactly the Bank of England decided on this design. Presumably they paid a computer scientist to check that it made sense? Or maybe a historian who specialized in Turing and understood his paper? I think that a much cooler Turing machine choice was possible, but I understand prioritizing historical accuracy and simplicity over putting Easter egg puzzles on your currency. Overall, it was a satisfying micro-quest. Perhaps this souvenir will fit nicely between my Knuth check and my Bristol pounds.

This is my post for day 8 of the Inkhaven writing retreat.

I’m trying to figure out what’s up with what I’m calling “being a policy”. I’d like to get better at it.

The classic way to decide what actions to take is to generate a bunch of options, evaluate the outcomes of taking each one, and pick the action corresponding to the best predicted outcome. In other words: to think about it.

Thinking, however, is expensive and slow. So we develop a ton of ways to shortcut the process.

One of these ways is to develop a habit. A habit is something that you had to practice a few times, but that eventually became automatic, and that you now do essentially unconsciously and involuntarily. You could stop doing it if you wanted, but first you’d have to notice, then you’d have to decide to stop, then you’d have to practice stopping. I think I would classify things like muscle memory or skills under this. Every time you use the scissors, your brain does not have to recalculate the optimal way to move them. Habits can also be purely internal or cognitive, like if you find yourself feeling annoyed about something, automatically try to generate something about it that you’re grateful for.

Another type of non-deliberative decision is what I would call a heuristic. Perhaps you have a heuristic that you don’t eat until you’re hungry. By using the hunger signal to fire off the behavior, you save yourself from having to constantly re-decide whether to eat or not. But you might sometimes decide to go against this heuristic. If you’re about to go on a long road trip, you might want to eat a big breakfast right away, so that you don’t have to stop for food for a while. Or perhaps you’re still in the office at 7pm and you’re starving, but you’ve just got a little bit of work left to do before you can send off this email, so you decide to keep working and eat after you’ve sent the email. Heuristics can save you 95% of the cost of deciding, while still being flexible to the context.

There’s a third kind of decision procedure that I would call a policy. A policy is a well-defined rule that you always follow. I probably got this term partly from its use in reinforcement learning, but it also matches the connotation of a company policy. The reason you have a policy is because you deliberated for a while on what the best course of action would be in this recurring, well-defined context, and decided that you always want to take that action. So after you decide on the best course of action, you then install it as a policy. Like a habit, you will now take that action every time, but unlike a habit, it may be very conscious and difficult. Unlike a heuristic, you will not reconsider under varying contexts.

Part of the purpose of a policy is to ensure that you take the action even when other circumstances would give you reason not to. Policies help ensure fair action, and better long-term consequences, even at the cost of short-term consequences.

A normal person might call a policy a “principle”, a “virtue”, or just “the right thing to do”.

In the field of advanced, mathematically-founded decision theory, the right decision procedure is to maximize expected utility, according to your best predictive models and consistent values.

But according to doubly advanced, meta-mathematically-founded decision theory, the right decision procedure is to first identify the decision procedure that maximizes expected utility, and then install that decision procedure.

I am generally quite good at having predictive models, reasoning through the implications, and then taking the action with the best predicted outcome. But I sure do have some big flaws in that last part. I seem to do a form of hyperbolic discounting, according to which it always feels like a good idea to do the somewhat easier action right now.

I do have some policies. For example, I do not drink alcohol. I don’t remember ever deciding on this policy, but it just has zero appeal. There are apparently some good things about alcohol, and I understand that drinking a tiny bit will have no observable effects. But that is not relevant, because I Do Not Drink Alcohol. Another example for me is that when Musk bought twitter, I stopped using twitter. It did not feel like a choice, it felt like “shoot, I can’t use that now, that’s too bad.” Would there still be benefits to using twitter? Absolutely. It seems like the whole machine learning community uses twitter as their primary social network. I’d be more informed about my work. And it really is quite a lot of fun. But that is not relevant, because I Cannot Use Twitter, now.

Both of these examples (and others that I’ve thought of) have two things in common; 1) the rule is about not doing something rather than doing something, and 2) the condition is extremely well-defined. A change I really want to make is something like “choose productive activities more often, and consumptive activities less”. But I don’t want to totally stop consumptive activities. How much is enough?

I think it’s actually extremely common for people to have what I’m calling policies, and to use them as tools for living better lives. I suspect that one reason I don’t seem to know how to pick up this tool is because I’m in love with “reason”, and I’ve spent all my life refining my ability to make good predictive models, and assess what actions and outcomes would be valuable. Since installing a policy is partly for preventing you from thinking about what to do each time, it feels somewhat “anti-reason”. But I do in fact endorse and understand the doubly-advance version of decision theory, and so I would endorse installing more policies.

I just need to find where this tool’s handle is first, before I can pick it up and start using it.

This is my post for day 6 of the Inkhaven writing retreat.

I think a lot about models. Mental models. The world is too big and complicated for us to memorize everything we experience, and it would take far too long to think through all the implications of everything that we do remember. So what we do instead is build lots of smaller models, small enough to remember, and simple enough to use for predicting and decision making.

We conditionally deploy these models largely based on bottom-up sensations. I haven’t memorized exactly which aisle or shelf my favorite bread is on, nor which breads are immediately to the left and right of it. But I know that when I run out of bread, that will trigger an action for me to decide when to go to the store, and that when I get to the store, the visuals around me will match up enough to a remembered template that I will just skim a bit and then find the bread more or less right away.

All of this will happen automatically, involuntarily, because it is required for navigating the world. But you can also do it intentionally. You can think about whether a given model is failing a bit too often, and whether you should look for a better model. You can try to merge two models into one. You can take a model that’s complicated to use, and see if you can find one that is much more elegant while still being sufficiently accurate.

You can also build models of content that you’ll never personally experience. You can try to understand what went so wrong in 18th century France, or how what’s up with Uranus being sideways.

All these models live in our heads. They are little bits of software that we programmed inside our brains by going around living. So they are written in brain language. This is ultimately mathematical, but when you are the math, it doesn’t feel like math.

It turns out that there are other entities in the world, doing the same thing. And they’re pretty cool! We like to interact. But they think in terms of their mental models, and we think in terms of our mental models. Fortunately, there is substantial overlap between how these models work. It turns out that reality has joints, along which any functional agent will have delineated some concepts used in their mental models. We can point at an apple and say “apple”, and then the other entity will know to assign the word “apple” to the mental model that is activated inside their mind when they look at what we pointed at. Language is the means by which we bridge across our individual mental models.

Many people have a craft. They spend their careers paying deep attention to something and practicing it. I am a craftsman of mental models. Unfortunately, you cannot just hand someone a mental model in the same way that you can hand them a sword or fresh vegetables. By default, all my work is stuck inside my head. To invite someone into the shop of a mental model craftsman, you have to communicate the models. If we’re using language, then I have to move around the structure of the model and linearize it. (Models do not like being linearized.)

Fortunately I do enjoy the process of communicating models. It can make the models better, but it’s also just a blast to give people the same experience I had while making it. But it’s still much, much more natural for me to just keep on crafting. So by this point I feel as though I’m sitting in a shop filled with thousands of little gadgets. And the gadgets themselves could be put to better use if other people could use them.

I want to get better at sharing my models. That’s why I’m at Inkhaven.