Multimodal Models

Background

CLIP (Radford et al. 2021) demonstrated zero-shot vision-language alignment; GPT-4V (2023) brought multimodal reasoning to conversational AI
Modality fusion has been studied since the 1990s in speech recognition (audio+lip) and media analysis
Modern multimodal models unify vision, language, audio, and video in a single Transformer backbone

The Concept

Multimodal AI processes and relates information from multiple modalities (text, images, audio, video) simultaneously. A shared embedding space aligns representations across modalities - enabling cross-modal retrieval, visual question answering, and image captioning.

How it Links

CLIP contrastive pretraining connects to self-supervised learning and metric learning
Vision-language models (LLaVA, GPT-4V) use a vision encoder (ViT) connected to an LLM via a projection layer
Connects to cross-modal attention, which is a generalization of cross-attention in seq2seq Transformers

Concrete Example

Searching a photo library with text: CLIP embeds both the query ("sunset over mountains") and all photos into the same vector space. Photos with matching visual content cluster near the text embedding. No manual tags needed - the alignment is learned from 400 million image-text pairs scraped from the web.

Prerequisites

Attention Mechanisms, Transformers, Tokenization & Embeddings, Convolutional Neural Networks, Autoencoders & VAEs

Learning outcomes

State the symmetric InfoNCE loss used by CLIP and derive it from a noise-contrastive view.
Compare the four dominant architecture patterns for vision-language models: dual-encoder, cross-attention into a frozen LM, projector-into-LLM, and early-fusion.
Explain why CLIP supports zero-shot classification through prompt templates and where that fails.
Describe the Q-Former in BLIP-2 and the Perceiver Resampler in Flamingo as solutions to the same bridging problem.
Run zero-shot image classification with CLIP and reason about its known failure modes.

TL;DR

Multimodal models map two or more sensory streams into a shared computational substrate. The post-2021 stack is dominated by four patterns: dual-encoder contrastive models like CLIP [4] and ALIGN [5] that align modalities through a symmetric InfoNCE loss; cross-attention bridges into a frozen language model like Flamingo [8]; lightweight projectors that feed visual tokens into a pretrained LLM like LLaVA [10]; and early-fusion native-multimodal architectures like Chameleon [13] and Gemini [11]. The choice between them trades data scale, compute, modality flexibility, and how much of an existing LLM you can reuse.

Intuition

A unimodal model lives inside one sensory channel. A multimodal model needs to compare or compose representations from at least two. The hard part is not encoding each channel separately - those are solved by ViTs [15] for images, Transformers [16] for text, and Whisper-style encoders [12] for audio. The hard part is the bridge: how do you make a vector that came from a 224x224 image comparable to a vector that came from the sentence “a cat sitting on a sofa”?

There are two answers and they correspond to two camps in the literature:

Train a shared embedding space from scratch on huge paired data. This is CLIP. Both encoders learn from the same loss; the bridge is the loss function itself. The output is a single embedding space where image vectors and text vectors are directly comparable by cosine similarity.
Take a pretrained LLM and inject the other modality into it. This is Flamingo, BLIP-2, and LLaVA. The bridge is a small set of new parameters - a Q-Former, a Perceiver Resampler, or just an MLP - that maps image features into “tokens” the language model already knows how to read. The output is a generative model that can describe, answer questions about, or reason over images.

The first path makes a great retrieval system. The second path makes a great chatbot.

The bridging problem in one sentence

Multimodal learning succeeds or fails based on the quality of the smallest module that translates between the modalities, not on the size of the encoders themselves. Spend your parameters wisely on the bridge.

A short pre-history

Before CLIP, vision-and-language was mostly task-specific architectures trained on small curated datasets:

Show and Tell [1] (2015) ran a CNN encoder into an LSTM decoder for image captioning, treating it as machine translation from pixels to English. It worked, but every new task needed a new model.
VQA [2] (2015) introduced visual question answering as a benchmark and stimulated a decade of attention-based fusion architectures (stacked attention, bilinear pooling, MUTAN, etc.).
ViLBERT, LXMERT, UNITER, OSCAR (2019-2020) brought BERT-style pretraining to vision-language pairs, with masked language modeling, image-text matching, and masked region prediction objectives. They closed gaps on benchmarks but still trained from labeled data.

The shift between this era and the modern era is that everything pre-CLIP curated its data, and everything post-CLIP scraped it. CLIP’s 400M image-text pairs from the web - a number it openly says is undersampled compared to what is achievable - changed the data calculus permanently.

The CLIP loss

Math: deriving the symmetric InfoNCE loss

CLIP’s loss is the symmetric form of InfoNCE [3]. Start from the noise-contrastive view.

Given a batch of $N$ image-text pairs $\{(x_i, y_i)\}_{i=1}^N$ , encode each into unit-norm embeddings $\mathbf{u}_i = f_\theta(x_i) / \lVert f_\theta(x_i) \rVert$ and $\mathbf{v}_i = g_\phi(y_i) / \lVert g_\phi(y_i) \rVert$ . Define the cosine similarity $s_{ij} = \mathbf{u}_i^\top \mathbf{v}_j$ scaled by a learned inverse-temperature $\tau$ :

\ell_{ij} = \tau \cdot s_{ij}.

For the image-to-text direction, the model is trained to recognize $\mathbf{v}_i$ (the text matched to image $i$ ) as the correct match against the $N-1$ other texts in the batch acting as negatives:

\mathcal{L}_{\text{i2t}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\ell_{ii})}{\sum_{j=1}^{N} \exp(\ell_{ij})}.

This is exactly the cross-entropy of an $N$ -way classification problem with logits $\ell_{i,\cdot}$ and target class $i$ . It is a Monte-Carlo lower bound on the mutual information $I(\mathbf{u}, \mathbf{v})$ [3]:

I(\mathbf{u}, \mathbf{v}) \geq \log N - \mathcal{L}_{\text{InfoNCE}}.

Define the symmetric text-to-image direction analogously:

\mathcal{L}_{\text{t2i}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\ell_{ii})}{\sum_{j=1}^{N} \exp(\ell_{ji})}.

The CLIP loss is

\mathcal{L}_{\text{CLIP}} = \frac{1}{2} \left( \mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}} \right).

Two implementation details matter:

The temperature $\tau$ is learned, not fixed. CLIP parameterizes it as $\tau = \exp(t)$ with $t$ a scalar parameter, clamped at $\exp(t) \leq 100$ to prevent runaway sharpening. A higher $\tau$ pushes the softmax toward one-hot, making the loss more demanding.
The batch size is the negative sampling budget. With $N$ pairs you get $N-1$ negatives per query. CLIP used a global batch of 32,768 across 256 V100s; ALIGN [5] used 16,384 across 1,024 TPUs. The loss is essentially a giant retrieval task and the more negatives the harder and more informative each step.

Why two directions, not one? The asymmetric InfoNCE only trains the image encoder to be predictive of text, or vice versa. The symmetric version forces both encoders to converge to a shared space where similarity is meaningful in both directions. Empirically this is significantly better at retrieval and zero-shot classification.

Connection to the alignment objective in machine translation. CLIP is doing online hard-negative mining at batch scale. It is conceptually the same as Negative Sampling for word embeddings (Word2Vec): pick a positive pair and rank it above $k$ noise samples. Just much, much larger.

Architectural patterns

The post-2021 multimodal landscape clusters into four patterns. They differ on what is frozen, where the bridge lives, and what the output is.

Pattern	Examples	Bridge module	What is trained	Output	Strengths	Weaknesses
Dual-encoder contrastive	CLIP, ALIGN, SigLIP, OpenCLIP	Loss function (symmetric InfoNCE)	Both encoders end-to-end	Joint embedding space	Zero-shot retrieval and classification with no head; cheap inference; modality-agnostic at inference time	Cannot generate text; cannot reason; loses fine-grained spatial detail; "bag of concepts" representation
Cross-attention into frozen LM	Flamingo, IDEFICS, OpenFlamingo	Perceiver Resampler + gated cross-attention layers	Resampler + new cross-attn weights only	Generative LM with image conditioning	Few-shot image-text in-context learning; preserves LM capabilities; works with arbitrary image counts in context	Expensive to train; resampler is opaque; needs intricate gating to keep LM stable
Projector into LLM	LLaVA, MiniGPT-4, BLIP-2, InstructBLIP	Q-Former (BLIP-2) or 2-layer MLP (LLaVA)	Bridge + LLM (LoRA or full fine-tune); image encoder frozen	Generative LM that consumes image tokens	Cheap to train (a few hours on 8 GPUs for LLaVA); reuses entire LLM stack; clean instruction-following	Visual tokens compete with text for context; weaker on dense visual reasoning; bottlenecked by projector capacity
Early-fusion native multimodal	Chameleon, Gemini, GPT-4o	Shared tokenizer + transformer over interleaved tokens	Everything end-to-end on mixed corpora	Unified token-level model over all modalities	No separate bridges; can output any modality; scales cleanly with data and compute	Massive training cost; data balancing is hard; hard to ablate which modality is contributing what

The trend over 2021-2024 is clearly toward early-fusion. Gemini and GPT-4o are early-fusion at training time. Chameleon [13] is the open paper that explains the recipe in detail: tokenize images into discrete visual tokens, interleave with text tokens, train a single transformer with a single next-token-prediction loss over the joint stream. The cost is enormous; the payoff is that the same model emits text or images and reasons across both natively.

CLIP, in one diagram

   image                   text
     |                       |
     v                       v
  +-----+                +-----+
  | ViT |                | TE  |
  +-----+                +-----+
     |                       |
   normalize              normalize
     |                       |
     v                       v
   u_i  --- cosine(u, v) ---  v_j
                |
        scale by tau, softmax over batch
                |
         symmetric cross-entropy

ViT is a Vision Transformer image encoder; TE is a text Transformer with a final pooling step. Both project to a shared dimension (commonly 512 or 768). Inference is just both encoders in parallel followed by cosine; no fusion module.

BLIP-2’s Q-Former

BLIP-2 [7] introduced the Q-Former (Querying Transformer) as a parameter-efficient bridge between a frozen image encoder and a frozen LLM, building on the contrastive-then-generative pretraining recipe from BLIP [6]. The idea: instead of trying to project all $N$ image patch tokens into the LLM’s input space, learn a fixed set of $K$ query embeddings (typically $K=32$ ) that cross-attend into the image features and then feed the resulting $K$ tokens to the LLM as a soft prompt.

The Q-Former trains in two stages:

Vision-language representation learning: image-text contrastive (like CLIP), image-text matching (binary classification), and image-grounded text generation, all on the Q-Former with a frozen image encoder. This makes the queries extract text-relevant visual features.
Vision-to-language generative learning: feed the queries into a frozen LLM (OPT, FlanT5) as a soft prompt and train the LLM to caption / answer questions, with a small projection layer between the Q-Former output and the LLM input dimension.

The whole bridging stack is ~108M parameters - tiny relative to the LLM and image encoder it sits between.

Flamingo’s Perceiver Resampler

Flamingo [8] faces a different challenge: in-context learning with multiple images. The Perceiver Resampler [9] is a small Transformer that compresses the variable-length output of the visual encoder into a fixed number of visual tokens (Flamingo uses 64). The compressed tokens are then fed into the frozen LLM through gated cross-attention layers inserted between LM blocks.

The gates start at zero and are learned, so at initialization Flamingo is exactly the underlying LLM. This is the safest possible way to add a modality to a pretrained model: degrade gracefully to the original LLM, then train the gates to open just enough to let visual information through without destabilizing the language model.

LLaVA’s projector

LLaVA [10] is the simplest and most reproducible pattern. The bridge is a 2-layer MLP. Take a frozen CLIP ViT-L/14 image encoder, take a Vicuna LLM, train an MLP to project ViT patch features into the LLM’s token space, and fine-tune on GPT-4-generated visual instruction data. The whole thing trains in a few hours on 8 A100s.

The reason it works is the visual instruction tuning data: synthetic conversations about COCO images, generated by feeding the captions and bounding boxes to GPT-4 and asking it to write Q&A pairs. This crystallized the recipe for a generation of open multimodal models.

Code: zero-shot CLIP

The canonical use of CLIP is zero-shot image classification. You do not need a labeled training set or a classifier head; you just write the class names in English and let the text encoder produce the classifier weights.

python

# pip install open_clip_torch torch torchvision pillow
import open_clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load a CLIP model and its preprocessing pipeline.
model, _, preprocess = open_clip.create_model_and_transforms(
  "ViT-B-32", pretrained="laion2b_s34b_b79k"
)
tokenizer = open_clip.get_tokenizer("ViT-B-32")
model = model.to(device).eval()

# Image side.
image = preprocess(Image.open("cat.jpg")).unsqueeze(0).to(device)

# Text side: write the classifier as natural-language prompts.
classes = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
text = tokenizer(classes).to(device)

with torch.no_grad():
  image_features = model.encode_image(image)
  text_features = model.encode_text(text)
  image_features /= image_features.norm(dim=-1, keepdim=True)
  text_features /= text_features.norm(dim=-1, keepdim=True)
  logits = (100.0 * image_features @ text_features.T).softmax(dim=-1)

for cls, p in zip(classes, logits[0].tolist()):
  print(f"{cls:30s} {p:.3f}")

open_clip is the LAION-trained reproduction with stronger checkpoints than OpenAI’s original. The 100.0 logit scale matches the trained inverse temperature.

The key idea: the prompt template ("a photo of a {label}") matters. CLIP was trained on web captions, not bare class names, so wrapping each label in a sentence pattern brings the text distribution closer to training. The original CLIP paper provides 80 prompt templates per ImageNet class and ensembles them; the gain over a single template is several percentage points.

Prompt ensembling

templates = [
    "a photo of a {}.",
    "a photo of the {}.",
    "a close-up photo of a {}.",
    "a photo of a small {}.",
    "a photo of a large {}.",
    # ... 80 total in the original paper
]
 
class_embeddings = []
for cls in classes:
    prompts = [t.format(cls) for t in templates]
    text = tokenizer(prompts).to(device)
    with torch.no_grad():
        emb = model.encode_text(text)
        emb /= emb.norm(dim=-1, keepdim=True)
        emb = emb.mean(dim=0)               # average over templates
        emb /= emb.norm()                   # re-normalize
    class_embeddings.append(emb)
 
text_features = torch.stack(class_embeddings)   # [num_classes, d]

This is the closest thing CLIP has to a classifier head. The class weight matrix is just the prompt-averaged text embedding for each class.

Evaluation

Multimodal evaluation is a moving target. The standard set in 2024:

Benchmark	Modality	Metric	What it measures
ImageNet zero-shot	image -> class	top-1 / top-5 accuracy	Generalization of the joint embedding to a curated label distribution.
COCO / Flickr30K retrieval	image <-> caption	R@1 / R@5 / R@10	Bidirectional retrieval quality on natural captions.
VQA v2	image + question -> answer	accuracy (open-ended; majority-vote over 10 humans)	Visual question answering across object/attribute/count/scene.
MMMU	image + multi-discipline question	accuracy	College-level reasoning across 30 subjects with diagrams, charts, equations.
MMBench	image + question	circular eval accuracy	Granular skill breakdown (perception, reasoning, OCR, etc.) across 20+ leaf abilities.
TextVQA / OCR-VQA	image + question -> answer	accuracy	Reading text rendered in the image; isolates OCR capability.
POPE	image + yes/no question	accuracy / F1	Object hallucination - asks "is there a [object] in the image?" with negative samples.

A model that does well on retrieval (CLIP-style) is not the same model that does well on VQA (LLM-style). Retrieval rewards a clean shared embedding; VQA rewards instruction-following plus a connection to world knowledge. The two objectives pull in different directions and that is why the architectural patterns above stay distinct.

Pitfalls

Hallucinated grounding

LLM-based multimodal models will confidently describe objects, text, or attributes that are not in the image. The POPE benchmark exists exactly to measure this. Mitigations are imperfect: better instruction data (LLaVA-1.5, ShareGPT4V), grounding-aware decoding, and refusal training help, but no current open model is reliable for safety-critical scenes.

OCR is harder than it looks

CLIP and most retrieval encoders cannot read text in images reliably. The patch tokenization throws away exact pixel positions and a 32x32 patch covers an entire short word at low resolution. Specialized OCR (DocVQA, TextVQA) needs higher-resolution encoders, native-multimodal training (Gemini, GPT-4V), or a separate OCR module piped in.

Spatial and counting reasoning

“How many people are in the image?”, “Is the cat to the left of the dog?”, “What is in the bottom-right corner?” are systematic weak spots. Patch-token global pooling drops fine-grained spatial structure. Cross-attention models (Flamingo) and high-resolution models (LLaVA-NeXT, Gemini 1.5) do better but the gap to humans is still large.

The binding problem

When two objects share an attribute (“a red cube next to a blue sphere”), models often “leak” attributes between objects (“a blue cube next to a red sphere”). This shows up most clearly in text-to-image generation but the encoders are also affected. Compositional generalization is an open research problem.

Open-vocabulary detection

If you need to localize objects by name, CLIP alone is not the right tool - it returns global similarities, not bounding boxes. Use OWL-ViT [14] or GLIP, which are CLIP-style trained on detection-style data. They keep the open-vocabulary advantage but emit boxes.

Interview Q&A

Derive the symmetric InfoNCE loss used in CLIP from the noise-contrastive principle, and explain why both directions (image-to-text and text-to-image) are needed.research

Start from a single direction. Given a batch of $N$ paired examples $\{(x_i, y_i)\}$ , encode them into unit-norm embeddings $\mathbf{u}_i$ and $\mathbf{v}_i$ . The image-to-text InfoNCE loss is

\mathcal{L}_{\text{i2t}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\tau \, \mathbf{u}_i^\top \mathbf{v}_i)}{\sum_{j=1}^{N} \exp(\tau \, \mathbf{u}_i^\top \mathbf{v}_j)},

which is the cross-entropy of an $N$ -way softmax classifier whose logits are scaled cosine similarities and whose target class is $i$ (the matched text). This is a Monte-Carlo lower bound on the mutual information between image and text features [3]:

I(\mathbf{u}; \mathbf{v}) \geq \log N - \mathcal{L}_{\text{InfoNCE}}.

Asymmetric InfoNCE only encourages $\mathbf{u}$ to be predictive of $\mathbf{v}$ . The reverse direction $\mathcal{L}_{\text{t2i}}$ uses the same denominator structure but with the columns (texts) querying the rows (images):

\mathcal{L}_{\text{t2i}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\tau \, \mathbf{u}_i^\top \mathbf{v}_i)}{\sum_{j=1}^{N} \exp(\tau \, \mathbf{u}_j^\top \mathbf{v}_i)}.

CLIP minimizes the average:

\mathcal{L}_{\text{CLIP}} = \tfrac{1}{2}(\mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}}).

Why symmetric. Asymmetric training is enough to learn a one-way mapping (image features predictive of text). It is not enough to make text features useful for retrieving images, which is half the use case. Symmetric training enforces that both encoders converge to the same shared space. Empirically, symmetric InfoNCE outperforms one-direction InfoNCE on every retrieval benchmark [4].

Connection to the temperature. $\tau$ is a learned parameter, not a hyperparameter. As $\tau \to \infty$ the softmax becomes one-hot and the loss demands perfect separation; as $\tau \to 0$ the softmax becomes uniform and the loss saturates. Letting the model choose $\tau$ is what lets the same architecture train on web-scale data without ever needing manual scheduling.

Why does the LLaVA recipe (frozen vision encoder + small MLP projector + fine-tune LLM) work as well as it does empirically? Where would you predict it to break?research

It works because of three reinforcing properties:

The frozen vision encoder is already linguistic in expectation. CLIP’s ViT was trained against English text. Its patch tokens already live in a space where dimensions correlate with semantic concepts (“dogness”, “redness”, “indoorness”). A 2-layer MLP is more than enough to reorient that space into the LLM’s token space because the underlying signal is already concept-aligned.
The LLM is doing most of the reasoning. The visual tokens act as a soft prompt; the actual chain-of-reasoning happens in the LLM’s parameters. Vicuna-13B already knows what cats are, what kitchens look like, and how to write captions. The projector just needs to deliver enough visual evidence to activate the right priors.
The instruction-tuning data is GPT-4 quality. The synthetic conversations LLaVA trains on are generated by feeding COCO captions and bounding boxes into GPT-4 and asking it to write Q&A pairs. This crystallizes a high-quality teacher signal at low cost - one of the under-appreciated reasons the recipe works.

Where it breaks.

Dense visual understanding: counting, spatial relations, OCR. The MLP cannot synthesize information across distant patches; you would need cross-attention between visual tokens or higher-resolution input. LLaVA-NeXT addresses some of this with multi-tile inputs.
Fine-grained categorization: distinguishing 200 dog breeds. The CLIP encoder was not trained for this and the MLP cannot recover what was lost in patch pooling.
Long-context with many images: the LLM’s context window fills up. Flamingo’s Perceiver Resampler explicitly compresses to a fixed token budget per image to avoid this.
Faithfulness: the LLM’s prior often dominates the visual evidence, especially when the question matches a common pattern. This is the hallucination regime the POPE benchmark targets.

A predictive heuristic: LLaVA-style projectors break wherever the answer requires evidence the CLIP encoder did not preserve. If you can look at the question and the CLIP feature map and decide the answer is “in there”, LLaVA can probably extract it. If not, you need a different bridge.

Compare the Flamingo Perceiver Resampler to the BLIP-2 Q-Former. What problem does each solve and when would you choose one over the other?onsite

Both are bridges between a frozen vision encoder and a frozen language model. Both compress a variable-length stream of patch tokens into a fixed-size set of “soft tokens” the LM can consume. The differences are in how they train and what they optimize for.

Perceiver Resampler (Flamingo): a small cross-attention block where $K$ learned queries (Flamingo uses 64) attend into the patch tokens. Trained jointly with new gated cross-attention layers inserted between LM blocks. The whole bridge plus the gates are trained on multimodal next-token-prediction; the LM and image encoder are frozen. Critically, the gates start at zero, so at initialization the model is exactly the unmodified LM, and training opens the gates only as much as necessary.

Q-Former (BLIP-2): also a small Transformer where $K$ learned queries attend into the patch tokens, but it is trained in two stages on three contrastive-style objectives (image-text contrastive, image-text matching, image-grounded text generation) before being plugged into the LM. The output is a fixed set of $K=32$ tokens that go into the LM as a soft prompt through a single linear projection.

Choose Flamingo when you need few-shot in-context multimodal learning - interleaved sequences of images and text, with the model picking up the pattern from a handful of examples in the prompt. The gated cross-attention preserves the LM’s in-context behavior because it is interleaved at every layer.

Choose BLIP-2 when you want a clean instruction-following multimodal model and you want to train cheaply. The Q-Former pretraining stage gives you image-grounded representations before any LM is involved; the LM-stage training is a single soft-prompt task. BLIP-2 was the recipe most copied by LLaVA, MiniGPT-4, and the open multimodal cascade of 2023.

The convergence point is that both architectures are now legacy patterns in 2024. Native multimodal pretraining (Gemini, Chameleon, GPT-4o) trains the entire model end-to-end on mixed-modality data and outperforms both. The Q-Former and Resampler era was 2022-2023; from 2024 onward the open question is how to scale early-fusion economically.

A frontier lab is deciding between (a) training a CLIP-style dual-encoder at 10x larger scale and (b) building an LLaVA-style projector model on top of an existing frontier LLM. Which would you recommend, and what data and evaluation would you ask for first?staff

The right answer is: it depends on the product, and you should never commit to either before naming the deployment. Frame the question by deployment:

If the deployment is search, retrieval, or moderation, choose (a). A dual-encoder gives you a single embedding space with cosine similarity at inference time. A 10x scale-up of CLIP buys you better recall on long-tail concepts, better robustness to caption phrasing, and zero new inference cost - the encoders run once per item and you compare with cosine. SigLIP-style sigmoid losses and longer training give large compounding gains here. The thing to ask for first is a paired evaluation set that mirrors the deployment: queries and items drawn from the actual production distribution, with hard negatives.

If the deployment is a chat-like product or anything that needs to generate text, choose (b). A projector model gets you 80% of frontier multimodal capability for 1% of the cost, and it inherits every instruction-following and reasoning improvement made to the underlying LLM - you ride the LLM’s roadmap for free. The thing to ask for first is visual instruction tuning data at the quality level of the deployment. LLaVA-1.5 vs LLaVA-1.0 was almost entirely a data quality story; the architecture barely changed.

The trap to avoid: pretending these are interchangeable. They are not. A CLIP scale-up will not give you a better chatbot; the dual-encoder loss does not produce generation. A projector model will not give you a better retriever; it does not produce a clean shared space. Frontier labs typically run both tracks in parallel because they serve different products.

The frontier-2024 view: if you have the compute and the data pipeline, neither (a) nor (b) is the right ceiling. Native multimodal early-fusion (Gemini, Chameleon, GPT-4o) outperforms both on generation tasks and matches or beats CLIP scale-ups on retrieval after a contrastive head is added. The honest staff-level answer is: do (b) for the product roadmap this year, do (a) as a calibration baseline, and start a third track on early-fusion that ships in 12-18 months.

Check yourself

Score: 0 / 5(0 answered)

1. In the symmetric InfoNCE loss used by CLIP, what role does the temperature $\tau$ play?
2. Why does CLIP support zero-shot image classification without ever seeing the target classes during training?
3. What is the primary architectural difference between Flamingo and LLaVA?
4. In BLIP-2, what is the function of the Q-Former queries?
5. Which of the following is NOT a known weakness of CLIP-style models?

Practical project

Build it: a CLIP-powered image search over a personal photo library

Goal: index your local photo collection (or a public dataset) and search it with natural-language queries.

Steps

Install open_clip_torch and faiss-cpu. Pull a strong open checkpoint such as ViT-L-14 pretrained on laion2b_s32b_b82k.
Encode every image once with the image tower, L2-normalize, and store the vectors in a FAISS IndexFlatIP (inner product = cosine for unit vectors).
Query: encode the user query through the text tower, normalize, run a top-k search.
Display: open the top 9 thumbnails in a grid.
Stretch goals:
- Add a prompt template ensemble for the query (the 80 ImageNet templates) to see how much the wording affects retrieval.
- Try a negative query: search for “a photo of a cat” minus “a photo of a black cat” by subtracting the embedding.
- Index 100k+ images and replace IndexFlatIP with IndexHNSWFlat for sub-linear search.
- Compare ViT-B-32, ViT-L-14, and a SigLIP checkpoint on the same library.

The only model file you need is the CLIP checkpoint (~600 MB for ViT-L). Everything else is your own metadata. This is the simplest serious multimodal system you can build, and it is the same architecture that powers production tools like Apple Photos search and Google Lens.

Research frontier

Where multimodal research is going

Three threads to watch:

Native multimodal at scale. Gemini and GPT-4o demonstrated that unified token-stream training over text, image, and audio can match or beat the modular pattern at sufficient scale. Chameleon [13] is the open paper that documents the recipe. Open questions: how to balance modality mixtures during training, how to evaluate without leaking benchmark data into the corpus, how to scale to video and 3D.
Long-context multimodal. Gemini 1.5 pushed multimodal context to 1M+ tokens, enabling whole-movie reasoning. The challenge is that visual tokens are expensive (each image is hundreds of tokens at high resolution). Compression strategies, hierarchical context, and KV-cache reuse are active research areas.
Multimodal reasoning. Static visual question answering is largely solved at the level of perception. The frontier is multi-step reasoning over images: chain-of-thought with visual intermediate states, visual program synthesis, world models. Recent work on multimodal o1-style models suggests RL on reasoning traces transfers to vision.

If you are preparing for a research interview, the InfoNCE derivation, the Q-Former two-stage training scheme, and the early-fusion vs late-fusion trade-offs are the three most-asked topics in this area.

References

Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015). Show and Tell: A Neural Image Caption Generator. CVPR 2015. arXiv:1411.4555
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). VQA: Visual Question Answering. ICCV 2015. arXiv:1505.00468
van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding. arXiv preprint. arXiv:1807.03748
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. arXiv:2103.00020
Jia, C., Yang, Y., Xia, Y., Chen, Y., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. ICML 2021. arXiv:2102.05918
Li, J., Li, D., Xiong, C., and Hoi, S. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022. arXiv:2201.12086
Li, J., Li, D., Savarese, S., and Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023. arXiv:2301.12597
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. arXiv:2204.14198
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., and Carreira, J. (2021). Perceiver: General Perception with Iterative Attention. ICML 2021. arXiv:2103.03206
Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). Visual Instruction Tuning. NeurIPS 2023. arXiv:2304.08485
Gemini Team, Google. (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint. arXiv:2312.11805
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv preprint. arXiv:2212.04356
Chameleon Team, Meta. (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint. arXiv:2405.09818
Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., and Houlsby, N. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022. arXiv:2205.06230
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021 (arXiv preprint 2020). arXiv:2010.11929
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762

Diffusion Models Interview Prep Guide