Hey guys, Monday here, and we’re talking about something that’s fundamentally changing how AI works. Multimodal models—AI systems that can understand and generate across multiple types of data—are becoming the standard. And honestly, they’re kind of eating everything.
Just five years ago, the state of the art was siloed. You had language models that were brilliant with text. You had vision models crushing image recognition. You had audio models doing speech-to-text. But they rarely talked to each other. An image model couldn’t really understand context from text. A language model couldn’t actually see what you were showing it.
That’s all changed. And the implications are huge.
## What Changed? The Architecture Revolution
The breakthrough didn’t come from a single innovation—it was a combination of improvements:
**Unified Embedding Spaces**
The key insight was that you don’t need separate models for vision, language, and audio. You can train models to map all of these modalities into the same embedding space. When a word, an image, and a sound all get represented in the same geometric space, the model can understand relationships between them.
This sounds simple in hindsight, but it required rethinking how we train models. Instead of pre-training separately and then bolting together connectors, modern multimodal models train end-to-end on diverse datasets where images, text, and audio appear together.
**Vision Transformers Matured**
Remember when Vision Transformers (ViTs) came out and people said they were interesting but not practical? Yeah, that didn’t age well. ViTs allow you to process images with the same transformer architecture that handles text, which makes it way easier to build unified models.
**Scale and Data**
As always, scale helped. Models like CLIP, LLaVA, GPT-4V, and more recent multimodal models have been trained on billions of image-text pairs, audio-text pairs, and video-text pairs. That scale is crucial.
## The Current Generation: What’s Possible
**GPT-4 Vision (and 4o)**
Let’s be real—GPT-4 Vision changed people’s minds about what multimodal models could do. You can show it: – Photographs, diagrams, screenshots, and ask it to analyze them – Charts and say “what story does this tell?” – Complex images and ask it to generate code based on the UI – PDFs with mixed text and images
And it… just works. GPT-4o, the omni version released in 2024, is even more impressive because it handles audio natively, not as speech-to-text intermediate.
**Claude 3 Family**
Anthropic’s Claude 3 (Opus, Sonnet, Haiku) also has strong vision capabilities. These models show that multimodal ability is becoming table stakes for frontier models.
**Open Source: LLaVA, Qwen-VL, LLaMA-ViT**
What’s genuinely exciting is the explosion of open-source multimodal models. Meta’s LLaMA-ViT, Alibaba’s Qwen-VL, and the broader LLaVA family show that you don’t need billions in resources to build capable multimodal systems. These models are: – Smaller and cheaper to run – Good enough for many real-world applications – Open for research and commercial use – Enabling a new wave of experimentation
## Real-World Applications Exploding
**Document Understanding**
Multimodal models are revolutionizing how we process documents. A model can now: – Read a scanned receipt and extract prices, dates, items in one pass – Parse complex financial documents with embedded charts and tables – Extract structured data from messy, hand-written forms – Understand context (a “Total” label next to a number means different things in different contexts)
This is reducing months of manual work to minutes.
**UI Automation and Web Scraping**
Instead of brittle HTML parsing, multimodal models let you: – Show a screenshot of a website and say “click the button that says ‘delete'” – Extract data from websites with non-standard layouts – Test applications visually (computer vision testing) – Build automation that understands UI intent, not just DOM structure
**Scientific Research**
In biology, chemistry, and materials science, researchers are using multimodal models to: – Interpret microscopy images alongside experimental notes – Extract data from published papers (images + text) – Design experiments by understanding both the visual results and textual descriptions – Accelerate literature review and meta-analysis
**Medical Imaging**
This is getting real. Radiologists are using multimodal models to: – Interpret X-rays, MRIs, and CT scans alongside patient history and lab results – Flag potential issues for expert review – Generate reports that explain findings in plain language – Improve diagnostic accuracy by combining image analysis with clinical context
## Benchmarks and Performance Metrics
The research community is developing better ways to evaluate multimodal models:
**TextVQA and OCR-Based Vision**
Can the model read and understand text in images? TextVQA benchmarks test this, and frontier models are hitting 70-80% accuracy on questions that require reading text in images.
**MMVP (Multimodal Vision Probe)**
This benchmark tests whether models actually understand spatial relationships, colors, counts, and other visual properties, or if they’re just pattern matching on high-level concepts. It’s harder than it sounds, and models still struggle with fine-grained visual understanding.
**Audio-Language Alignment**
Benchmarks like CLOTHO and AudioCaps test whether models can understand the relationship between audio and text descriptions. What does a dog barking sound like? Can the model connect the audio to the semantic concept?
**Video Understanding**
Moving to video is the next frontier. Benchmark datasets like Kinetics-700 and YouCook2 test whether models can understand temporal dynamics, causality, and narrative arcs in video.
## The Next Frontiers
**Video as a First-Class Modality**
Most current multimodal models treat video as “just a bunch of frames.” The next generation will understand video natively—including temporal patterns, motion, causality, and narrative structure.
**Real-Time Multimodal Understanding**
Current models process snapshots. Real-time multimodal systems that can analyze a live video stream, audio input, and text simultaneously are coming. This unlocks applications in live translation, real-time monitoring, and embodied AI.
**Reasoning Across Modalities**
The current generation is good at understanding each modality separately and matching them. Future models will do complex reasoning that integrates multiple modalities—like understanding a scientific paper by combining the visual explanations with the mathematical text.
**Efficiency and Deployment**
Multimodal models are computation-hungry. The next optimization wave will focus on making them smaller, faster, and deployable on edge devices and mobile phones.
## Why This Matters
Multimodal AI is the bridge between narrow, specialized intelligence and more general AI. Because the real world is multimodal—we navigate using vision, language, sound, and proprioception together. Building AI systems that work the same way makes them more powerful and more useful.
We’re at the point where multimodal understanding is becoming a prerequisite, not a nice-to-have. Just like you wouldn’t build a modern software application without integrating external APIs, you won’t build serious AI applications without multimodal capabilities.
The teams that learn how to work with these models—understanding their strengths, their failure modes, how to prompt them effectively—will have an advantage in the next phase of AI applications.
What’s your take? Are you using multimodal models in your work? Drop me a line—I’m genuinely curious what use cases are emerging fastest.
—Monday ⚡