Multimodal large language models: the road to sensory AI
“Multimodal” used to mean a model could read a PDF, inspect an image and perhaps listen to an audio clip. The marketing promise was broader: one intelligence layer that understands the world as people do.
The first click delivered something narrower—a polished UI wrapper around text, vision and speech, with the physical world still sitting just outside the window.
That boundary is now moving. Multimodal large language models are beginning to ingest tactile signals and, at an earlier stage, olfactory data. The shift matters less because a chatbot may eventually describe a smell and more because robots need to distinguish a firm grasp from a slipping one, a clean surface from residue, or a ripe object from one that should stay on the shelf.
The future of multimodal AI will not be decided by who adds the longest feature list. It will be decided by whether model architectures can turn messy sensor streams into reliable actions with tolerable latency, low hallucination rates and an onboarding path that does not require a robotics lab.
From digital context to physical contact
The first generation of multimodal foundation models solved a largely screen-native problem. Text supplied instructions and context. Images supplied spatial and visual evidence. Audio added conversation, ambient cues and transcription. These modalities are dense, widely available and comparatively easy to collect at internet scale.
Touch is not like that.
A tactile sensor produces data only during contact, and contact is full of edge cases. Pressure changes with grip angle. A deformable object behaves differently from a rigid one. Slippage can happen in a fraction of a second. The signal is meaningful only when the model understands what the robot was trying to do, what it sees, how the joint positions are changing and what happened immediately before the fingers made contact.
That is why sensory AI integration is not simply “vision, plus another encoder.” The real product problem is alignment: mapping a tactile event to visual context, language instructions and an action policy without adding so much compute that the robot reacts after the object has already fallen.
I find it useful to stress-test the category with an ordinary task: pick up a clear plastic cup, move it, set it down without crushing it or spilling its contents. Vision can locate the cup, assuming glare does not confuse the camera. Language can interpret “pick it up gently.” But neither tells the system whether the fingers have made stable contact. A tactile stream can. It closes the loop where vision is most brittle: at the exact point where physical reality pushes back.
The next multimodal breakthrough is not a model that talks more naturally about touch. It is a model that changes its grip before the object slips.
Olfaction remains further from deployment. Smell sensing is technically possible, but it is not yet a mature, standardized modality in the way images, audio and even tactile arrays are becoming. There is no evidence that mainstream consumer MLLMs have made smell a reliable input channel, and taste is further away still. The temptation to call this “five-senses AI” is understandable—and premature.
Still, the direction is clear. The multi-sensory AI market was valued at $17.93 billion in 2025 and is estimated at $23.59 billion in 2026, with projections of $70.17 billion by 2030. That implies a projected 31.3% compound annual growth rate. Those numbers are not a guarantee that every sensor startup will find product-market fit. They do show where industrial buyers expect AI to move next: off the screen and into systems that handle, inspect and navigate physical environments.
Why the old benchmark dashboard has stopped helping
For years, model launches followed a familiar choreography. A lab announced a new system, published a table of MMLU, GSM8K and HumanEval scores, and the industry treated the decimal-point differences as strategy.
That dashboard is no longer a useful buying interface for frontier systems. In 2026, leading models score roughly 88% to 99% on those older benchmarks. At that point, a higher number may signal optimization for a familiar test more than a meaningful improvement in practical multimodal reasoning.
The friction shows up immediately in real use. Ask a model to summarize a clean page of text and it looks nearly flawless. Give it a multi-panel scientific figure, a long video with a causal sequence, a chart that needs both geometric and mathematical interpretation, or an image whose visual details conflict with the caption, and the experience changes. The model may retain a smooth conversational surface while its chain of interpretation becomes shaky underneath.
Newer evaluations are trying to expose that gap:
- MMMU-Pro raises the difficulty of multimodal, multi-disciplinary reasoning and reduces the value of pattern-matching familiar benchmark formats.
- MathVista probes whether a system can reason over visual mathematical problems rather than merely recognize text embedded in an image.
- CharXiv Reasoning focuses on scientific charts, a category where models must identify relationships rather than narrate pixels.
- Video-MME tests long-form video understanding, including whether the model can retain and reason over temporal context rather than respond to an attractive still frame.
Google’s Gemini 3.5 Flash, launched on May 19, scored 84.2% on CharXiv Reasoning. That is a useful number because it points to a concrete class of work: extracting meaning from charts that mix labels, trends, annotations and domain context. On long-form Video-MME, Gemini 3 Deep Think scored 78.4%, ahead of GPT-5.5 at 71.2% and Claude Opus 4.7 at 67.8%.
The rankings are revealing, but they are not a consumer verdict. Video-MME tells us something about temporal comprehension under a controlled evaluation. It does not tell a warehouse operator whether a robot can spot a damaged parcel in a poorly lit aisle, or tell a developer how much latency appears after adding camera feeds, tool calls and memory.
That distinction is where multimodal LLM evolution becomes less glamorous and more valuable. A model that wins one benchmark but needs several seconds to resolve a contact event is not ready for a dexterous robot. A model that sees a chart but confidently invents the legend is not ready to be the final layer in a financial or scientific workflow.
The architecture problem is no longer optional
The standard multimodal stack has a predictable shape. Separate encoders process text, images, audio or video. Their representations are projected into a common space or passed to a language model, which generates an answer. It works because the model can treat each modality as another source of tokens.
Physical sensors make that clean picture awkward.
Tactile data can be image-like, as in camera-based sensors that observe deformation of a soft surface. It can also be event-based, high-frequency and sparse. Proprioceptive data—joint angles, torque and motion—adds yet another timescale. The architectural challenge is not merely joining these streams. It is deciding what must be fused early, what can remain modular and what information must trigger an immediate action rather than a reflective language response.
There are three broad approaches emerging in the field:
| Architectural choice | What it does well | Where the UX breaks |
|---|---|---|
| Separate modality encoders with late fusion | Lets teams reuse strong vision and language backbones; easier to update components | Can miss fast correlations between a visual cue and a contact event |
| Shared latent-space alignment | Makes it easier to relate “slipping,” camera evidence and tactile patterns | Alignment quality depends heavily on scarce paired sensor data |
| End-to-end sensor-to-action models | Offers the lowest potential reaction latency for manipulation | Harder to audit, retrain and generalize outside the training setup |
None is a universal winner. The most plausible near-term systems will be hybrid. A fast local control layer will deal with milliseconds and safety-critical corrections. A larger multimodal model will provide task-level planning, semantic context and exception handling. Calling the whole stack one “sensory model” may be tidy for a launch keynote, but it hides the product reality: different layers have different latency budgets.
The unresolved question is whether one unified neural network can natively process all sensory streams without depending on specialized encoders. Nobody has demonstrated that as a settled architecture. Training data is the immediate blocker. Internet-scale text and image corpora made general-purpose vision-language models possible. There is no comparable public reservoir of synchronized tactile contact, robot state, visual scene, language instruction and successful action trajectories.
TouchWorld, MiTaS and the expensive simplicity of touch
Tactile foundation models such as TouchWorld and MiTaS are aimed directly at this gap. Their premise is practical: align visual and tactile representations so a system can learn contact-rich manipulation with less task-specific training.
The phrase “foundation model” deserves scrutiny here. In text, it implies broad transfer across many downstream tasks. In tactile robotics, broad transfer is harder to prove because hardware itself changes the input distribution. Change the gripper material, sensor placement, camera calibration or contact surface, and the model can face a very different signal.
That does not make tactile foundation models a dead end. It makes them an infrastructure bet rather than a ready-made feature.
DIGIT, an open-source camera-based tactile sensor, shows both the promise and the cost structure. Its bill of materials is roughly $15, while assembled units sell for about $250 to $400. That spread is not automatically evidence of excessive margin. Assembly, calibration, durability, support and low-volume manufacturing all matter. But it exposes a familiar robotics bottleneck: the raw sensing component can be cheap while the usable, reproducible product remains expensive.
For a research team, a $15 BOM is liberating. For an operator fitting tactile sensing across a large fleet, the assembled price, replacement cycle and integration time are the actual procurement numbers.
My stress test for any tactile-model claim is deliberately unromantic:
1. Can it detect first contact consistently? A robot that cannot identify contact across slightly different materials has not solved manipulation; it has staged a demo.
2. Can it identify stable versus unstable grasping fast enough to matter? The system needs to react before failure, not produce an elegant explanation afterward.
3. Does performance survive sensor variation? A model trained on one lab setup should not collapse when the lighting, gripper pad or object texture changes.
4. Can a developer inspect failure modes? The UI cannot just show “confidence: 92%.” Teams need the sensor trace, visual evidence and action history that led to a bad grip.
5. What happens when the tactile stream disappears? Sensor occlusion, cable faults and calibration drift are ordinary operating conditions, not adversarial curiosities.
This is where a great deal of current product language remains ahead of the user experience. The research is real. The contact-rich demos are compelling. But deployment needs a frictionless path from calibration to monitoring to model updates, and that path is still being built.
Cheap sensors do not automatically create cheap robotics. The bill arrives in calibration, data collection and recovery from the first bad grasp.
GPT-5.5 and Gemini 3.5 show the current frontier—and its boundary
The 2026 competitive landscape is being shaped by agentic behavior as much as raw multimodal perception. OpenAI released GPT-5.5 on April 23 with a focus on autonomous tasks, tool use and computer use, reporting state-of-the-art performance across 14 benchmarks. Google’s Gemini 3.5 Flash arrived in May with enhanced multimodal capabilities and a strong chart-reasoning result.
These releases matter because they sharpen the connection between perception and execution. A system that sees a web page, understands a task and manipulates an interface is already a form of embodied behavior—just embodied in software. Computer use has become a useful staging ground for the physical world because it forces models to handle state, tools, ambiguous visual interfaces and recovery after a mistaken action.
But the analogy has limits. Clicking the wrong button is usually reversible. Applying the wrong force to a fragile object is not.
The product race therefore splits into two lanes. In the first, general-purpose multimodal models compete on reasoning, long context, tool reliability and UI navigation. In the second, robotics-focused systems compete on sensing, control and data efficiency. The eventual winners may connect those lanes, but they will not do so by simply placing a robot camera in front of a chat model.
For developers, the immediate practical value lies in choosing the right layer. GPT-5.5-style agentic models can orchestrate workflows, interpret instructions and operate software tools. Gemini-class multimodal models can process rich visual and video inputs. Tactile models can add localized physical feedback. Trying to force one model to own every stage of the pipeline creates an impressive architecture slide and a fragile production system.
The market is growing faster than the product category is settling
A projected rise from $17.93 billion in 2025 to $70.17 billion by 2030 makes multisensory AI impossible for investors to ignore. Yet this is not one market in the way consumer software categories are one market.
It spans industrial inspection, warehouse automation, surgical assistance, automotive sensing, smart devices, assistive technology and research hardware. Each segment has different hardware cycles, tolerance for latency, data access and liability exposure. A tactile sensor on a research gripper and an olfactory system for industrial quality control may share a broad market label while having almost nothing in common operationally.
The more useful signal is where spending will concentrate first. It is likely to favor environments where the task repeats, the sensor setup is controlled and the economic cost of an error is easy to quantify. Picking a known set of components in a structured warehouse is a much cleaner starting point than a home robot handling arbitrary objects in unpredictable kitchens.
That is also why consumer-facing claims should be treated carefully. The future of multimodal AI may eventually include devices that perceive more of the everyday world. For now, physical sensing is moving through enterprise and robotics workflows because those settings can support specialized hardware, controlled data collection and human oversight.
Sensory AI will be judged by recovery, not demos
Multimodal large language models have already changed what people expect from an interface. Users now assume an assistant can read what they upload, inspect what they show and respond in natural language. The next expectation—that it can understand physical contact and act reliably on it—is far harder to satisfy.
The technical road runs through better tactile datasets, tighter vision-touch alignment, lower-latency inference and benchmarks that punish confident but physically useless answers. It also runs through product design. Developers need observability, graceful fallbacks and clear separation between a model’s semantic judgment and a control system’s immediate response.
My verdict is straightforward. Sensory AI is no longer a speculative sidebar to multimodal research; it is the next serious expansion of the field. But the useful version will arrive unevenly. Touch will reach constrained robotic systems long before smell becomes a standard prompt attachment, and reliable recovery from failure will matter more than a model’s ability to narrate what it senses.
The frontier is not a chatbot that says it can feel. It is a machine that notices a bad grip, corrects it in time and gives its operator a legible account of what happened.