The Deictic Interface

The most important word in an agent interface may be “this.” Not because it is expressive on its own. It is not. “This” is almost empty until it is tied to a situation. This chart. This sentence. This error. This product. This person. This window. This thing I was just looking at. This claim the other agent made. This version, not that one.

In ordinary software, the user has to do a lot of work to make “this” explicit. They copy a URL, select a paragraph, upload a file, take a screenshot, name a tab, paste an error, describe the object, or navigate through menus until the system knows which thing is at issue.

Agents make that burden feel increasingly strange.

If the assistant can see, remember, search, act, and coordinate, why should the user have to reconstruct the visible world in prose? Why should a user have to say “the third row of the table in the window on the right, the one with the unusually high fee” when they can point?

That is the promise of deictic agent interfaces.

A deictic interface lets the user bind language to situated context by pointing, hovering, circling, drawing a box, selecting a region, or relying on a recent trail of attention. It turns words like “this,” “that,” “here,” and “the thing I was just comparing” into usable references.

This sounds small. It is not.

Deixis is the bridge between conversation and the world in front of the user. Without it, the assistant lives in a separate text channel and the user remains the translator between screen and speech. With it, the user can treat the visible world as part of the conversation.

The next interface is not only something the user types into. It is something the user can point through.

Pointing is not a product

The first mistake is to treat pointing as a product category.

It is not. Pointing is an input primitive.

The same gesture can feed very different systems. Point at a date, and a fast system might show action chips. Point at an error, and a work assistant might bring the screenshot, logs, and recent files into a conversation. Point at a hotel listing, and a forum could invite relevant service agents to explain availability, fees, and alternatives. Circle a paragraph, and a memory system might preserve it as the referent for a later question.

The gesture is the same. The architecture behind it is not.

That distinction matters because deictic interaction can serve two competing futures. In one future, pointing is a more convenient trigger for app-local actions. The system classifies the target, matches it to a known schema, and shows a small menu. This is useful, fast, and sometimes exactly right.

In the other future, pointing is how a user gives a continuing assistant precise present context. The gesture does not have to resolve immediately into a menu. It can become a reference object inside a conversation: something the assistant can inspect, remember, compare, hand to another agent, or ask the user to clarify.

The first future is a smarter shortcut.

The second future is a different relationship to the computer.

The fork is partly a matter of clock speed. A pointer that must produce a menu in under a second has to classify the target against a schema and offer whatever actions the catalog already holds. A pointer that feeds a conversation can afford slower, generative interpretation of the target — because the gesture is creating a referent, not racing a menu. Choose the deadline and you have chosen the architecture.

After the App is interested in the second future. The point is not that action chips are bad. The point is that the more consequential design question is what happens after the system knows what “this” refers to.

This is no longer hypothetical. Google DeepMind’s Magic Pointer ships the gesture: nudge the cursor over a paragraph, image, product, or block of code and Gemini binds to it. It is the best on-ramp anyone has built — and it lands the pointed-at thing inside one assistant, in Chrome. That is exactly the fork above. A year before it shipped I was testing an adjacent idea, AI actions on the ordinary right-click menu, and the lesson was the same: keep the gesture, and decide carefully where it goes. Pour every point into a single assistant and you have built the on-ramp to the everything assistant (Against the Everything Assistant); route it into a visible room and you have built something else.

The reference comes before the action

A pointer is often treated as an action trigger. Click here, do this, open that, move there.

For agents, the pointer should first be treated as a reference.

This distinction is subtle and important. If the system assumes that every deictic gesture is a request for immediate action, it has to decide too much too soon. It has to classify the object, choose the relevant action set, rank options, and render something quickly. That works best for narrow objects with predictable actions: dates, addresses, images, contact cards, product listings, document snippets.

But many useful gestures are not action requests yet. They are attention markers.

“This feels wrong.”

“Compare this to yesterday.”

“Why is this different?”

“Ask the finance agent about this.”

“Remember this for the house project.”

“Use this as the example.”

In these cases, the user is not asking the system to pick a predefined action. The user is creating a shared referent. The assistant's job is to preserve that referent accurately enough that conversation can proceed around it.

This is where deictic interfaces become more than convenience. They let the user move from describing the world to marking the world. Marking is lower friction, but it is also more precise. The gesture carries visual and spatial information that language often loses.

The product rule: bind first, act second.

The system should be able to say, in effect: “I am using the table row you selected,” or “I am answering based on the paragraph you circled and the chart beside it.” That acknowledgement gives the user a chance to repair the binding before the assistant acts.

Four ways to say this

Deictic interfaces need more than one gesture.

The simplest form is hover-over-element. The user points at a DOM element, UI control, file, row, chart mark, image, message, or visible object. If the underlying application exposes structure, the system can bind the pointer to a clean object: this button, this cell, this heading, this person, this issue.

The second form is region selection. The user draws a box, lassos an area, brushes over an image, or highlights part of the screen. This matters because many important targets do not map cleanly onto application structure. A user may care about an awkward cluster of fields, a visual anomaly, a group of overlapping objects, or an area where the layout itself is the problem.

The third form is whole-situation selection. Sometimes “this” is not one object. It is the current state: the arrangement of windows, open tabs, visible files, partially completed form, warning dialog, and user's apparent task. Whole-screen reference lets the assistant reason about the situation rather than forcing the user to reduce it to an item.

The fourth form is recent pointer trace. The user may not explicitly select anything. They may hover, circle, pause, move back and forth between two options, or visually compare several items before asking a question. A remembered trace can let the assistant infer that “this” refers to what the user just emphasized. The strongest version treats the trace as a gesture buffer: pointer motion captured continuously, like screen memory, and consumed lazily at prompt time only if the question makes it relevant. The user circles an error and simply types “what is wrong with this?” — no gesture button, no attachment step. Low-pressure movement becomes a recoverable input channel rather than an invocation.

These modes are not interchangeable. Element binding is precise when structure exists. Region selection handles visual ambiguity. Whole-situation reference handles task state. Recent trace handles attention.

A mature interface will need all four.

The market already has each piece in partial form. Browser extensions are DOM-aware and selection-bound — they can act on the element but rarely leave the page. Desktop overlays use capture and vision — they can point at anything but mostly cannot write back into the app underneath. AI image editors prove that a click, brush, or lasso can be a model-grounded edit target — inside image pixels only. Pointing, in other words, is not one invention. It is the composition of DOM-aware selection, screen-level understanding, region-grounded generation, durable reference objects, and governed write-back — and the composition is what nobody has shipped.

The present needs memory

Pointing is usually imagined as present-tense. The user points at what is visible now.

But the more powerful version binds the present to memory.

“Compare this to the version I was editing yesterday.”

“Is this the same error I saw last week?”

“Find the product I was looking at before this one.”

“Was this clause already in the draft?”

“Show me where this chart changed.”

This is where deictic input becomes part of an agentic workstation rather than a screen overlay. A pointer supplies the present referent. Memory supplies the earlier referents. Tools supply the ability to compare, retrieve, transform, and act.

Without memory, pointing is local convenience. With memory, pointing becomes temporal navigation.

The user no longer has to remember filenames, version numbers, window titles, URLs, or exact phrases. They can gesture at the current object and ask the assistant to connect it to the relevant past. The assistant can then ask for repair if the binding is ambiguous: “Do you mean the chart from yesterday's draft or the one in the meeting notes?”

This is also why silent context capture is both powerful and risky. A system that can remember recent screen state and pointer movement can make “this” work with very little ceremony. But the same capacity can feel invasive if the user cannot tell what was captured, when it is being used, and how to correct it.

The correct posture is not surprise. It is disclosure.

“I am using the area you just circled.”

“I am comparing this screen with the document you had open yesterday.”

“I saw two possible referents; choose one.”

Deictic memory needs repair affordances. Otherwise it becomes invisible inference by another name.

Against silent help

Deictic interfaces sit at the boundary between ambient and conversational AI.

Ambient systems try to help without being asked. Conversational systems wait for the user to pull help into a thread. Pointer-mediated AI can borrow from both. It may use ambient plumbing: screen perception, object recognition, schemas, action catalogs, recent attention traces. But its healthiest interaction contract is conversational: the user points, asks, corrects, and continues.

That user-initiated contract matters.

When a system pushes an unsolicited suggestion into a workflow, the user has to interpret not only the suggestion but the system's reason for appearing. Why now? What did it see? What did it infer? What else is it watching? If the answer is wrong, there may be no natural repair path. The card appeared, or it did not. The interruption felt helpful, or it did not.

When the user points and asks, the social geometry changes. The user has licensed the system to bind context. Silence is acceptable if the system cannot help. A clarifying question is acceptable if the referent is ambiguous. A visible acknowledgement of the binding is useful rather than creepy.

This is why deictic input is a bridge primitive. It can keep the speed of local context capture without accepting the arrogance of uninvited inference. It gives the assistant more of the world while preserving the user's role as initiator.

The design target is not an assistant that watches everything and decides when to intervene.

The design target is an assistant that can use the world the user indicates.

Pointing into the forum

The deepest version of deictic design is not single-assistant control. It is forum entry.

Imagine the user points at a product listing and says, “compare this with the alternatives.” One architecture routes the listing through the user's assistant, which searches, summarizes, and returns a blended answer. That is useful.

Another architecture lets the user's assistant use the pointing gesture to invite visible specialist agents into the forum thread: the seller's agent, a delivery agent, a warranty agent, a budget agent, perhaps a household agent that knows the user's constraints. Each participant can speak in its own name, post structured proposal artifacts, and be challenged by the others.

The gesture did not merely select a thing. It opened a situated episode.

This matters because many pointed-at objects belong to other parties. A hotel listing, insurance denial, invoice, contract clause, medical bill, work ticket, or appliance manual is not just data on a screen. It is a boundary object in a social and institutional relationship. The user may need explanation, negotiation, evidence, authority, or escalation from the party that owns the underlying service.

If the user's assistant privately interprets every object, the other participants remain hidden. The user gets convenience, but loses contestability.

If the pointing gesture can become an invitation into a forum, the object can summon the right speakers. The user's assistant remains present, but it does not have to impersonate everyone else.

That is the After the App version of deictic interaction: not merely “show me actions for this thing,” but “bring this thing into a visible coordination thread with the right participants and the right evidence.”

The fallback is still screen operation

The world will not expose clean agent interfaces all at once.

Sometimes the assistant will bind “this” to a visible object and still have no structured way to act on it. The site may have no API. The application may have no semantic contract. The object may exist only as pixels. The only available path may be browser or screen operation.

That fallback is necessary.

It is also the highest-risk rung. Once an agent operates an existing human interface, the system is no longer merely reading context. It may click, type, submit, change records, send messages, place orders, or expose data. The user needs visible sessions, scoped authority, confirmations for writes, and audit trails.

Deictic input can make this safer if it is treated as scope, not carte blanche.

Pointing at a form should not authorize the assistant to operate the whole account. Selecting a region should not grant access to every visible system. Asking “what is wrong with this?” should not become permission to fix it without review.

The gesture should define the referent and, where possible, the task boundary. Action still needs its own contract.

This is where deictic input and UI contracts meet. The user points in. The assistant proposes a governed surface out. If the next step requires operating a legacy UI, the system should show what it plans to do before doing it.

Design rules for deictic agents

First, make the binding visible. The system should show or state what it thinks “this” refers to, especially before consequential action.

Second, separate reference from command. A point can mark attention without authorizing action. The assistant should not treat every gesture as permission to execute.

Third, support repair. The user needs to correct the referent quickly: not that row, the one below; not the whole chart, the outlier; not this window alone, the comparison across both.

Fourth, preserve provenance. If the assistant uses a selected region, recent pointer trace, screen memory, or previous version, it should be able to say so.

Fifth, expose ambiguity. When several referents are plausible, the system should ask rather than guessing silently.

Sixth, let pointing compose. A user may point at two things and ask for comparison, point at an object and a target style, or point now and refer to something remembered. The interface should support multi-deictic work.

Seventh, route to participants, not only tools. Some referents call for another agent's voice, not just another function call. A visible forum needs to be one possible destination for a gesture.

Eighth, treat continuous capture as sensitive. Recent pointer traces and screen memory should be inspectable, bounded, and easy to disable. The value of low-friction context does not erase the need for trust.

Ninth, keep output governed. Deictic input should lead to structured proposals, previews, approvals, or explanations, not hidden action.

The common thread is simple: pointing should make the assistant more situated without making the system less accountable.

The world as shared context

The app era made the user responsible for translating between action and context. If the system needed to know what mattered, the user had to navigate, select, copy, paste, name, or upload it.

The agent era can reduce that burden. The user can point.

But pointing alone is not enough. The product has to decide what pointing means. Is it a shortcut? A reference? A command? A memory marker? An invitation? A scope boundary? A permission request?

Those answers determine whether deictic interfaces become another layer of ambient suggestion, or whether they become part of a richer conversational architecture.

The better path is clear. Let the user indicate the world. Let the assistant show what it understood. Let the thread carry the referent forward. Let other agents enter when the object belongs to a broader social situation. Let action require visible authority.

In that design, “this” becomes more than a word.

It becomes the smallest possible bridge between the user's attention and the system's ability to help.

After the app, the interface is not only the screen, the chat box, or the button.

It is the act of making something in the world available for shared work.

The input-side companion to The Interface Is a Contract. ← All essays