Magic Cap: machine language

The origin. With Will Gundy, I video-recorded two friends setting up email on an early agent-based PDA and read the session turn by turn, in Jefferson transcription, as conversation. Thirty years later its two findings describe how people break down with AI almost exactly.

The device was a Motorola Envoy running General Magic’s Magic Cap — a 1994 “personal communicator” whose interface extended the desktop metaphor into a whole world of rooms and streets: a desk, a hallway, a downtown. It was built for novices, so we chose novices: two women, best friends since the fourth grade, neither an early adopter, who approached the machine the way real people do — together, narrating to each other. That collaboration was the method, not an accident of it. Following Lucy Suchman, two people working aloud produce a naturally generated protocol; the talk we needed was the talk they made for each other.

The stance was stated against the grain of the field. A usability test, the paper argued, should act “not as a prism, which would destroy the data’s particularity, but as a lens.” (I have since, with Patrick Squire, helped build a lab literally called PRISM — the irony is not lost on me; more on that on the arc.) Rather than scoring task completion, we returned again and again to the tape and asked Harvey Sacks’s question: why that now?

Breakdown one: the world loses its frame

The instruction said “go downtown.” It should have been trivial. But the screen the users were on — a phone-setup panel — had quietly stopped being a place. Magic Cap’s whole promise was a world of scenes you move through; here that world had silently collapsed into a bare screen with no visible way out. Hunting for “downtown,” the users reached for what was local and ready-to-hand: an unlabeled garbage-truck icon (“whadaya mean take the truck?”), then a building marked with a big “I.” Both are sensible guesses — a truck and a building both belong to a downtown — and both are wrong, because the frame that would have made the right move obvious was gone. They could not proceed until a developer in the room stepped in.

The finding: consistency of frame is paramount. When the governing frame disappears — when the system shifts, in the paper’s phrase, “from scenes to screens” — users are left with no resources to locate themselves, and even good local reasoning leads them astray.

Breakdown two: “and then it jus- stopped”

Later, retrieving a test message, a progress bar crept across the screen and froze. There was a real server error, but the software said nothing. The users had to manufacture an account of the system’s state out of whatever was at hand — and adjacency did the rest: when they tapped “In box,” a help window happened to open at the same instant the frozen bar vanished, so the coincidence was read as a response. As Suchman observes, whatever happens next gets heard as the answer to what you just did, whether or not it is.

The repair, when it came, was physical. Correcting the developer’s wrong theory of what had gone wrong, one user traced the bar’s motion in the air and cut it off with a chopping hand — “Cause it kept going across… and then it jus- (.) stopped” — a gesture that did three things at once: it conveyed the timing that words had failed to convey for several turns, it corrected the expert, and it told him to stop talking and listen. Talk and gesture were mutually contextualizing, exactly as Goodwin & Goodwin describe.

The finding: usability is a function of the resources available to users for the repair of breakdown. Breakdown is inevitable; what separates a usable system from an unusable one is whether the software, the instructions, and the people around the user give them enough to find their way back.

A breakdown is not a negative situation to be avoided. It is a moment of unconcealment, in which the recognition that something is missing reveals the nexus of tools we were using all along.

Why a 1994 PDA predicts 2026

Magic Cap itself did not survive, and the study names a reason deeper than market timing: a quasi-mimetic interface tied to a desktop metaphor is incoherent. A world of rooms and streets promises that place will carry the meaning everywhere — and the moment the metaphor runs out, as it did on that phone-setup screen, the user is stranded more completely than a plain screen would ever have stranded them.

Read the two breakdowns again with an AI agent in mind and they stop being history.

Frame loss is what happens every time an agent switches context — into a tool, a sub-task, a handoff — and drops the thread the user was holding. The user is suddenly “on a screen, not in a scene,” with no way to locate themselves in the agent’s world. The frozen progress bar is the original illusion of execution: a system that asserts a state it has not actually reached, and a user who takes it at its word until the gap shatters. And adjacency — reading the next thing the system does as a reply — is the deepest trap in conversational AI, because a fluent model produces a plausible “next turn” for anything, related or not.

The design responses are the ones I still argue for: build the agent for repair rather than one-shot intent capture, and make the interface an inspectable contract whose seams are visible — so that when the frame breaks, as it will, the user has somewhere to stand.

Case study · 1995, with Will Gundy. Running title: “Machine ‘Language’: The Construction of Meaning Between System & Users in a Software Usability Test.” ← The Practice