The Unity of Experience: How the free energy principle builds reality

We only experience a single, stable perception at a time. How  bistable viusal figures and Karl Friston’s big idea explain how we build a coherent prediction of ourselves in the environment and keep our selves sane in an uncertain world.

We only experience a single, stable perception at a time. How  bistable viusal figures and Karl Friston’s big idea explain how we build a coherent prediction of ourselves in the environment and keep our selves sane in an uncertain world.

By James Vornov, MD PhD
Neurologist, drug developer and philosopher exploring the neuroscience of decision-making and personal identity.


It’s not just an accident of biology that our brains work as single-threaded engines, engaging only one input stream at a time, one view of the world at a time. I’m going to argue here that a real-time control system operating under conditions of uncertainty needs to minimize error and settle on its best guess prediction of the state of the world. Maybe developers of autonomous systems like self-driving cars could learn something from the nature of human consciousness.

One stable perception at time

Give it a bit of thought and you’ll see that we have a stable awareness of just one thing at a time. Reading comes one chunk of meaning at a time. We can’t listen to the radio while we’re reading. Pausing, we may turn our attention to the background music, but the sound was excluded from awareness while engaged with the text.

The brain is processing ambient sound all the while; we are just not attending to it, so it is not presented in awareness. If among the sound is a doorbell ringing, brain systems outside of awareness signal the system controlling the flow of sensory information to switch from reading to listening to the environment. We become aware, a bit after the fact, that the doorbell rang. If it rings again, it’s less of an echo of a recent event and more present in awareness.

Continue reading “The Unity of Experience: How the free energy principle builds reality”

Greetings from Helsinki and the European Stroke Organization Conference (ESOC)

I think I was last in Helsinki about 20 years ago. It’s a great place for a conference and I truly have enjoyed Finland. Compared to the rest of Europe (and home, the US), it has a modest, less consumer oriented culture. The favored drink here is tap water, which I think speaks volumes about the culture. They have adopted sustainability as a core philosophy, but it’s not performative in any way. It seems to fit with the Finnish culture of having enough.

Cloudy with a bit of rain. Cool. No real opportunity for photography, but I did grab some establishing shots with the phone.

My aim at the conference was to connect with those who are in the trenches treating stroke and figuring out what all of the new data really means for patient care. While I no longer practice medicine, in designing trials, the single biggest success factor is fitting into to the standard of care. And in an area like stroke treatment, where that standard changes on a yearly basis, understanding what the field is thinking is absolutely critical.

Not much discussion of mind and consciousness among these neurologists. They’re too busy trying to save brain.

To this neurologist, Severence got self-identity exactly right

When the recent season of the Apple TV show Severance concluded, I sent a pitch to a few mainstream publications to write about how close to reality the series is from the point of view of a neurologist. I never got a reply, so I’m going to move the discussion here as a post.


It turns out that the premise of the show, splitting human identity, is pretty close to some real, albeit pathological conditions that I’ve seen clinically as a neurologist. And there’s an important point to be made about how the brain constructs its model of self-identity as a continuous experience of the world.

If you haven’t seen the show, maybe you’ve at least heard of the premise. No real spoilers here, as I can base this discussion on just the opening episode of the series.

It goes like this. You open your eyes and don’t know who you are.

A woman’s voice is asking you questions. You’re in a conference room, wearing a suit. You don’t remember arriving, don’t recognize your own name on the folder in front of you. “Who are you?” she asks again. Behind this is a brain implant that splits your life in two. One is your normal, daily continuous existence outside of your work at this mysterious company, Lumen. That’s your Outie. Then there’s a self that exists only at work, the Innie. When the Outie descends to work in the elevator, the device is triggered, and the Innie emerges. The two selves share some base personality, but no narrative knowledge is carried over from one to the other.

A real case of identity loss

Let me tell you a story that happened to a colleague and friend. He found himself in an emergency room and had no idea how he had gotten there. But worse, when the nurse asked his name, he was unable to remember it. He couldn’t provide an address, telephone number, or employer. He found his wallet, which helped the ER staff contact family. Other than this confusion, he felt normal. The staff would have seen him as someone who was cooperative and alert but adrift.

This is called Transient Global Amnesia or TGA. For my friend, it lasted for about 90 minutes in the ER. As things came back, he felt embarrassed and silly but was reassured that this was a relatively rare occurrence and was unlikely to repeat itself or be indicative of a more serious condition. He stayed in the hospital overnight with scans and, as I recall, a cerebral angiogram, since this was in the days before we had the software and scanners to do angiograms with CTs and MRIs as we do now. In my days as an academic neurologist, I saw a few cases, one in the ER while it was happening. It looked like an odd, dreamy, disconnected state for an older woman. It’s like floating outside of identity. Dissociated from self.

In general, TGA comes on suddenly: the person remains alert but unable to recall recent—or even distant—events. They may ask the same questions repeatedly: “Where am I? What happened?” In some cases but not all, they can’t recall personal details, like their address or even their name. Then it fades away as quickly as it started.

What’s remarkable is that this happens without permanent brain injury. Most people still speak fluidly, understand language, and retain general knowledge of the world—they know what a hospital is, even if they’re not sure why they’re in one.

On MRI, there’s a consistent but very odd finding reported in some cases. As you might have guessed from the effect on global core memory, the findings have been in the hippocampus, a seahorse-shaped structure buried deep in the temporal lobe that’s part of the ancient “inner cortex”. We don’t think that memory is actually stored there, but it seems to be how cortical activity gets linked together and time-stamped to form episodic memory, the story of past experience. Like other brain systems, it seems to act as a gating system to store and recall previous activity stored diffusely in the cortex itself.

An abnormal signal has been detected in the hippocampus after TGA, generally after a day or two. These bright spots show up late and disappear, leaving no sign of permanent damage. The kind of MRI sequence that shows the change is called Diffusion Weighted Imaging (DWI), which is sensitive to the movement, the diffusion, of water molecules. When there’s injury, stress, or inflammation, water movement becomes restricted, showing up as bright spots. So something’s going on in the hippocampus, but the nature of the change that causes the hippocampus to go offline temporarily is a matter of speculation. Early in my career in studying mechanisms of stroke damage, this was an area of interest for me since the hippocampus is uniquely sensitive to metabolic stress. So we’ll guess for now this is some kind of localized metabolic disturbance that gets corrected after a while, some vascular or metabolic glitch that “turns off” episodic memory.

So in the world of Severance, we have an implant that switches off the Outie’s hippocampal function, leaving the Innie with TGA, at least at first. I can’t speculate how one could then turn it back on in a segregated, separate new identity. For fun, I’ll speculate that it’s using lateralization to achieve this. There’s a hippocampus in both the right and left temporal lobes. Like many of these global control systems, losing one side has no observable effect; the other side can carry on the function perfectly well. It’s only bilateral lesions that cause devastating loss of memory function, as I discuss in a moment. So we’ll say that the Severance implant works by creating something like a “split brain” condition where the outie continues to use the left hippocampus, but the innie uses the right for new storage and retrieval.

Time stamped memories

Now, is this really believable? Are memories time-stamped so that you could actually turn off some based on when the events occurred? Actually, there’s another neurological condition where this time stamp becomes very clear: the person who wakes up every day believing it’s the morning of their accident.

The first such report was in the 1950s—a man injured in an oven explosion who, for decades, believed it was still the day before the accident. Every morning, Henry Molaison, or HM as he was known in the published reports, awoke in the same world. He could speak, reason, and recognize family, but he couldn’t lay down new memories. The brain’s machinery for encoding experience into lasting memory had been irreversibly disrupted. We call this anterograde amnesia— loss of memory going forward with previously recorded memory preserved.

But what gets really strange is there have been other cases reported where memory isn’t simply turned off, but gets frozen in the past. In some cases, it’s been hours or days before the injury, but in one case, there was a near 20-year gap!

WS had a case of severe brain inflammation, encephalitis, that damaged his hippocampus and temporal lobes, just like Henry Molaison. But when WS recovered in 1997, his memory was frozen in 1979. In extensive interviews and testing with neuropsychologists, it appears he believed Jimmy Carter was president, that he was still in his early 30s, and his children were still young even though they were now adults. His recall of pertinent details perfectly lined up with the facts of 1979 with evidence of leakage from the ensuing years.

So you might wonder what happened when he looked in the mirror and saw a man in his 50s? When he saw a newspaper or documentary about the events between 1979 and 1997? Well, he’d become a bit disoriented and distressed about these facts that contradicted his beliefs. Sometimes he would make excuses for the discrepancy, like it was a joke or bad lighting. We call this confabulation, and it’s a very common human coping mechanism when memory systems fail. But since he had lost the ability to form new memories, there was no lasting effect. He would reset back to 1979 with no recall of what he had been shown minutes before. He had the usual inability to update the timeline, but for some reason, his contextual time stamp had been moved years back from his injury.

So if the Severance implant could keep the timestamp context intact, it’s possible that memory could be stopped and started reversibly. Personal identity is preserved in anterograde amnesia. The person knows who they are, what they value, their life history—up until the moment of trauma. After that, memory becomes patchy, and the ability to carry events forward is lost.

Choosing to split the self

But in truth, the real-world condition that Severance most closely resembles isn’t from brain injury or malfunction at all. It’s the condition we now call dissociative identity disorder (DID), part of the spectrum that included classic multiple personalities.

In the emergency room, you’ll see a patient walk in completely coherent, well-spoken, and oriented—except they insist they’re 20 years old filling out forms when they appear much older. They’re shocked when they see themselves in a mirror. They can recount vivid details from earlier in life but draw a blank on anything that happened afterward. There’s no injury. No drugs. Just a complete, time-bound void in memory.

What makes these episodes so remarkable is their precision. It’s not that the memory is fuzzy—it’s surgically absent. The patient remembers everything up to a specific point—a breakup, a trauma, the loss of a child—and nothing after. The person sitting before you seems to have stepped out of the past. It’s as if the timestamped memory has been tampered with to turn off access to everything beyond a certain date.

It appears that time-delimited memory shutdown is a real capability, engaged without injury. In the DID we recognize as multiple personalities, personal identity is fractured—different states or “alters” emerge, often with different postures, voices, preferences, and worldviews. Memory between them is sometimes leaky, sometimes sealed.

The two identities in Severance seem to most resemble dissociation controlled by the implant, rather than intrinsic brain processes as we see in the clinic. It’s as if the implant induces dissociation, but with a blank identity to be filled in by the Lumen workplace. In DID, the switch is often involuntary, triggered by stress or context. In Severance, the transition is mechanical—an elevator controls the shift. But what’s imagined in fiction already happens in function: the brain can selectively partition identity, assigning one set of memories to one context and hiding the rest.

Personal identity is a brain construct

The neurological conditions I’ve discussed, TGA and anterograde amnesia with frozen memory, reveal something about the mechanism of personal identity and timestamping of episodic memory. Cases of DID show the ability of these systems to be controlled by intrinsic brain mechanisms.

I thought, for example, that some patients develop DID as a way to survive trauma by disconnecting experience from awareness. If the brain can do that on its own, maybe a future where it’s done on command really might be possible.

So Severance is taking the real events of identity fragmentation not as disorder, but as a plot device. It doesn’t invoke trauma, repression, or psychological conflict. It posits a world where the brain’s natural capacity for compartmentalization is switched on and off like software.

But take the lesson that this is how all of our brains work all the time. We live as if the self is fixed: I am who I am. But neurologically, that’s never been true. The brain maintains identity through dynamic integration of memory. Personal identity is a story we present to ourselves and the world. But the self can be dissociated under the right conditions. Whether through trauma, inflammation, or an elevator at Lumen, what we see is this: the mind feels whole, but the brain is capable of dividing or at least putting on convincing mass that temporarily allows us to change identity based on context to achieve our aims.

Why the simplest computer is faster than any human?

The bottlenecks in our brains create awareness, meaning and coherence

Does it make sense to simulate the brain in a computer to produce a human-like intelligence that’s way more capable than a biological hum? I’ve make the argument that machine minds are different kinds of minds, and we should focus on intelligent function of machine minds based on their unique advantages rather than emulating how people think. 

During the discussions of uploading a human mind, I repeatedly asked myself why we were interested in doing this in the first place. The obvious sci-fi intent is digital immortality. Perhaps an expansion of the nature of being human by replacing biological embodiment with a digital embodiment that had the potential for greater bandwidth and augmentation of sensation and physical ability. We know the abilities of the human brain are severely limited in bandwidth and speed. 

The processing speed and storage limits

So let’s start with the brain’s limitations. You may have seen a recent paper showing that the brain processes speech at the remarkably low rate of 39 bits per second. The paper by Coupe et al. analyzed 17 spoken languages and found that while speech rates and information density per syllable vary between languages, the overall information transmission rate is remarkably consistent at 39 bits per second. 

Continue reading “Why the simplest computer is faster than any human?”

You want to upload a human mind, but you can’t even upload a worm

Reposted from my Substack as an experiment

Our friend, C. elegans

“What do you consider the largest map that would be really useful?”

‘About six inches to the mile.’

‘Only six inches!’ exclaimed Mein Herr.

‘We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!’

‘Have you used it much?’ I inquired.

‘It has never been spread out, yet,’ said Mein Herr: ‘the farmers objected: they said it would cover the whole country, and shut out the sunlight!’

—Lewis Carroll from Sylvie and Bruno Concluded, 1893

If you’ve been following my posts over the last few weeks on the failure of uploading a mind and not simply emulating what would appear like a person, you might object that this is all very sci-fi and not grounded in reality. Maybe the technology now just can’t be imagined, but someday a high-resolution scan or some very precise mapping function will provide a model so precise that the emulation will be for all intents and purposes an uploaded individual mind. Who knows, it might be self-aware with preserved self-identity.

I don’t think so. I think this is, as I’ve said, a confusion between copying a thing and building a model of a thing. An uploaded toaster can’t make toast, and a model of a hurricane won’t make you wet. The idea of uploading a brain is seductive — but it confuses a structural map with the thing itself.

Uploading the world’s simplest brain

Lets put aside the human brain for now. We can see this confusion of map and territory clearly in the failures and successes in uploading the brain of the worm, Caenorhabditis elegans into a computer. We’ll see that the bottom-up approach of mapping the C. elegans connectome didn’t work, but a top-down use of the model as explanatory has been increasingly useful as a way of understanding how a simple nervous system generates complex, state dependent behaviors. Models can be powerful tools to understand and predict behavior, but aren’t a replication of the thing itself, expected to just be an uploaded copy in the computer.

Continue reading “You want to upload a human mind, but you can’t even upload a worm”

Can We Understand the Brain?- Jeff Lichtman

As part of editing my book manuscript and starting my Substack posts, I turned my attention to being a better communicator at this intersection of neuroscience, philosophy and self-help. It’s really complex and many of the ideas are radical because they peer behind the illusion that the brain presents us through awareness.

As part of that effort, I’m been reading some of the recently published books on brain and consciousness like Anil Seth’s Being You and Donald Hoffman’s The Case Against Reality. By the way, I can recommend the former with reservations, but avoid Hoffman at all costs. In their efforts, these philosophers of mind are hopelessly trapped in what I’ve started calling “Mentalism”, the explanation of brain function not through neurophysiology but through metaphor and theory all of which is untestable and without consequences in the real world.

I was so happy to hear a real neuroscientist discuss our understanding of the brain on Sean Carroll’s podcast: Jeff Lichtman on the Wiring Diagram of the Brain – Sean Carroll. Jeff Lichtman is a contemporary of mine, an MD, PhD who stuck with basic research on brain connections through a very productive career.

Like Jeff, I’m astounded both at how far we’ve come in understanding the brain compared to where we started, but in awe of how far we are from a real understanding of how the brain functions as a complex system of networked neurons.

it’s an enormous amount of effort. And I don’t want to make fun of this, because this is the field I’ve been in for my entire professional career, and I’m old. And we have learned an enormous amount. The question is, if you’re climbing Mount Everest and you’ve gone three feet, have you made a lot of progress? You have made an infinite amount of progress relative when you started and you had gone zero. But we are still very far from having a deep understanding of how the brain works. And I will probably say at some point, I’m not sure that is what we should be aiming for anyway.

It’s also interesting to hear Sean Carrol, a physicist as smart questions about the brain from what for us is a very naive perspective. He’s had some of those Philosophers on his podcast over the years and they’ve given him some wrong ideas about what’s represented in the brain and our understanding of encoding. But Jeff put it perfectly:

I think of a nerve cell as a living creature. It’s a single celled organism, like a paramecium or an amoeba, but it’s living in a very weird pond, which is your head. It doesn’t know that it’s inside your head. It doesn’t care about whether you’re eating a sandwich or listening to a podcast. It’s just there. And it has to do certain things to stay alive. And so all the things it does are for its own survival because it’s a single celled organism with a will to survive. And those things end up generating learned based wiring diagrams. That from their perspective, they don’t know that that’s what they’re doing. They just know that if they don’t do that, they’re going to be punished and die.

This is a point I’ve been working on presenting more clearly. We have an introspective sense that I call awareness that provides some access to brain function. But outside of what we’re presented through this limited mechanism, we have no access to the functioning of the brain at all. And when we use our tools to look inside, we don’t understand at a fundamental level what these neurons are doing and how they’re organized. They’ve been assembled into networks through developmental programs to process sensory input through channels of sight, sound, position, etc and produce purposeful behavior.

And Jeff makes a great point about the difference between describing the structure of the brain and understanding it in the sense of predicting behavior.

I think even simulating a worm in a computer is a challenge, though, with only 300 cells, so I’m not too worried. I think, as I said, the best we may be able to do is describe in complete detail what’s there, but it’s a big difference from saying, I now can predict its behavior, I understand it. No human being, honestly, no human being could hold this amount of information. The analogy I give is like, do you understand New York City? And you’d say, that’s a stupid question. What do you mean? There’s so many things happening at the same time, and there’s so much complexity, and I would say if you can’t understand New York City, forget about the brain. It’s way more complicated than New York City.

I agree that if we can’t understand a 300 neuron worm brain, understanding the complex biological system of the human brain with its billions of neurons is beyond reduction down to understanding. So we describe. Make general statements about regions of brain, their connections, their participation in aspects of awareness and behavior, but the goal is better and better description.

This is what David Chalmers famously termed “The easy question”, which is describing the mechanisms of brain function that underly cognition and behavior. I think Jeff underestimates our progress. I think we’ve come more than just a few steps, but I agree there is much more to learn. I’m confident that the understanding gained will provide insight into human problems like depression, schizophrenia and, ultimately, allowing each of us to attain our potential. Because even though the mechanisms are an ad hoc, tangled mess, their purpose is emergent functions like optimism about the environment or fear. And we have already found a number of levers to shift the system toward a more desirable state. I think we just need to be a bit more sophisticated and realistic in how to accomplish change in a complex system we can describe but never truly understand.

Substack- So Far So Good

A week ago I posted for the first time on Substack and result has been quite nice. More reads of a post coming out of nowhere than I get here on my 25 year old blog based on analytics, but of course RSS reads these days are silent, so total audience size of a blog is an unknown. The community at Substack is really nice, reminiscent of the early days of blogging but enriched with many serious creators. It seems like the audience is looking for engaging content.

Writing on a schedule is a challenge that I’ll have to work on, probably by prepping more than a week in advance. At this point my goal is to see if I can get better at explaining my take on the topics where neuroscience and life meet. So I’m trying to take a single one of the concepts each week and try to make it understandable. So last week it was the illusion of self. This week it I’ve written about externalization of mental events as feedback.

In the meantime, I’m talking to some publishing industry pros about what to do with the manuscript. At this point, all options are still open, but I think a plan is taking shape. This writing, like my photography is my art. It’s for me and it’s my pleasure to share it.

My New Substack

When I saw that Venkatesh Rao, was “retiring” his blog of 17 years, Ribbonfarm and moving to Substack, it solidified my plan to try building audience there as well.

Here’s my first post there: Forget Cognitive Dissonance.

I’ve been writing in this space, hosted one way or another for just about 25 years now. The early days were exciting and we built up each other in a blogging community for a few years. Then with the rise of social media and commercialization of websites with ad driven clicks, blogging died a quiet death for all but a few high profile public intellectuals. I get traffic here from what I think is a small group of regulars plus some pages that Google serves up in the first page or two on note taking tools and camera equipment.

At least for right now, Substack feels a lot like a 2024 version of that early blogging environment with easy interaction with writers and low barrier to entry. Of course the monetization is a big problem with the balance between free and paid content varying wildly between creators.

For now, I wouldn’t think of charging for content as I’m after some more engagement than one gets at a site like this. I’m putting myself on a once a week schedule there with. more frequent notes here.

My other motivation, besides engagement, is to work on writing for a general audience to develop the ideas I’ve put into my manuscript. That’s put aside for the moment, but I know that the next edit should be an effort to be clear and systematic in exposition of ideas. I’m also curious about what subjects a general readership is interested in besides tools for thinking so that I can focus my next big writing project on something that might be more broadly read.

Blog Engine Running: Tools for Deciding

Dave Rodgers always notices when I start feeling the need to write in this space and I thank him for that. I’m glad to know that there are a few consistent readers.

A little while ago, I wrote about my review of what actually gets read here. It’s not the book reviews or philosophy, it’s the discussion of tools. And that makes sense, because it’s by far the most useful information for the largest audience.

And the last section of my manuscript is about positive steps and techniques for making better decisions. So it makes sense to focus there while trying a bit harder to lay out the unique perspective I’ve gained by looking into the brain mechanisms behind our choices. Because it turns out that it’s not so easy to make better decisions when almost all of them take place entirely outside of awareness by processes we have no access to or influence over.

Second Draft of ODB Manuscript Done

After many false starts over the years, I started from scratch and finished a first draft detailing what I’ve learned over the years almost 3 years ago. I let it sit for a long time, but after picking it up again about 18 months ago, I’ve made it through a full reorganization and revision and have something that more resembles a book.

Along the way there were a few long detours into researching probability (focusing on Jaynes’ big book and) Polya’s two part treatise on plausible reasoning) and the neuroscience of perception and valence. So it’s about 75,000 words. A sizable book, but one that’s not really commercially viable in today’s competitive publishing world.

Why would it be tough to get published? Two reasons really. One is that I’m not known in academia, philosophy or as a public intellectual. This little corner of the web is visited occasionally and I’ve never built an audience eager to spend a ton of money on a book. Given the investment a publisher makes in preparing for print and distribution, it’s not a reasonable business proposition. Second, the subject of decision making is a very crowded area in the publishing space. For the most part, Economists have flooded the zone with their discussions of “irrationality” and “bias”. So books on choice and rationality abound, even if they don’t really help us much since most of our decisions are made outside of awareness and can’t be adjusted to avoid our built in biases.

It’s good to have this big project behind me and begin to think about next steps. As I’ve written recently, I’m look at options and but I think there’ll be more effort here as part of the path forward.