Open Problems in AI

TL;DR

Hallucination and grounding

The problem. LLMs are trained to predict the next token, not to be true. The training signal rewards fluency and statistical plausibility, not factual accuracy. When the model does not know an answer, it produces a plausible-sounding continuation---a hallucination. Hallucination rates on factual QA benchmarks range from 3% (large frontier models on well-covered topics) to 30%+ (smaller models on niche topics).

Why it is structural. The next-token prediction objective has no ground-truth grounding. A model that hallucinates confidently and a model that is accurately uncertain both minimise the same cross-entropy loss if their output distributions match the training corpus. Factual accuracy is not directly rewarded.

Current mitigations. Retrieval-augmented generation (RAG) grounds the model on retrieved evidence and asks it to cite sources. RLHF with human raters who penalise hallucinations shifts the model toward more cautious responses. Self-consistency sampling (generate many responses and take the majority answer) reduces variance. Verification chains (the model generates an answer and then checks it step by step) catch some errors. None of these is a complete solution.

What a research interview question looks like: “Why do RAG systems still hallucinate even with retrieved context, and how would you design a system to reduce this?”

Reasoning: chain-of-thought and RL-on-reasoning

The problem. LLMs that answer questions directly without intermediate steps fail on multi-step logical reasoning, arithmetic, and planning tasks. The model needs to “think before it answers.”

Chain-of-thought prompting (Wei et al. 2022) showed that prompting with “let’s think step by step” or providing exemplars of reasoning chains dramatically improves performance on GSM8K (math word problems) and similar benchmarks. The key insight: token-by-token generation provides a “scratchpad” where intermediate computations can be verified and built upon. However, CoT prompting is brittle---the model can produce incorrect intermediate steps that lead to a wrong conclusion.

o1-style RL-on-reasoning. OpenAI’s o1 model and DeepSeek-R1 [1] train models to reason by applying reinforcement learning with outcome-based rewards (correct final answer = positive reward) on mathematical and coding tasks. The model learns to generate a chain of reasoning that leads to verifiable correct answers. DeepSeek-R1 showed that a relatively small RL training stage on top of a strong base model can dramatically improve reasoning, achieving performance competitive with o1 on math benchmarks.

Test-time compute scaling. A related paradigm: instead of (or in addition to) training, allow the model to spend more compute at inference time by generating multiple reasoning chains and selecting the best one (majority vote, self-evaluation, or process reward models that score intermediate steps). This creates a scaling dimension orthogonal to parameter count.

What a research interview question looks like: “What is the difference between a model that is good at math benchmarks because it has memorised solutions and one that genuinely reasons? How would you test for this?”

World models

The problem. Current LLMs have no internal model of physical reality. They can describe what happens when a glass falls off a table because such descriptions appear in their training data, but they do not have a grounded simulation of gravity. This limitation shows up in tasks requiring multi-step physical reasoning, spatial navigation, and embodied manipulation.

Ha and Schmidhuber [2] proposed that an agent should learn a compressed internal model of its environment---a world model---that can simulate future states given current state and action. The agent can then plan by “dreaming” in its world model rather than always interacting with the real environment. The Dreamer family of models (Hafner et al. 2019-2023) instantiates this in continuous-control RL and achieves superhuman performance on the Atari benchmark while using far fewer environment interactions than model-free RL.

For LLMs, world models remain elusive. A language model might implicitly encode some physical knowledge in its weights (it “knows” that heavier objects fall faster to a first approximation), but it has no mechanism to simulate the consequences of novel physical configurations. Connecting LLMs to learned physical simulators or 3D scene representations is an active research area.

What a research interview question looks like: “Describe an experiment that would distinguish a language model that has learned a genuine world model from one that has memorised physical descriptions.”

Embodied AI

The problem. Real-world intelligent agents must perceive, reason, and act in physical environments. The gap between language-based reasoning (where LLMs excel) and physical manipulation (where robotic systems are brittle) is wide.

Current approaches include: (1) Language-conditioned policies: use an LLM as a high-level planner that issues commands to a low-level motor policy (RT-2, SayCan). (2) End-to-end vision-language-action models: directly map observations and language instructions to motor actions in a single model. (3) Simulation-to-real transfer: train in photorealistic simulation and transfer to the real world.

Key challenges: long-horizon planning (a task may require hundreds of steps with no intermediate reward), robustness to visual distribution shift (trained in one environment, deployed in another), and data scarcity (robot trajectories are expensive to collect).

What a research interview question looks like: “What are the three main failure modes of language-model-based robot planners, and what architectural choices address each?”

Mechanistic interpretability

The problem. We do not understand what computations large neural networks perform internally. Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable circuits.

Circuits in vision. Olah et al. (2020) showed that features in CNNs are composed of circuits of neurons, and many features are universal across architectures (curve detectors, high-frequency detectors, multimodal neurons that respond to both visual and textual concepts).

Induction heads. Olsson et al. [7] identified “induction heads”---a two-layer attention circuit that implements in-context learning: the first head copies keys to later positions, and the second head attends to those positions to complete patterns. Induction heads emerge at a specific training step and coincide with a “phase transition” in in-context learning ability.

Superposition and sparse autoencoders. Elhage et al. [6] showed that transformer MLP layers represent more features than they have neurons, by encoding features in nearly-orthogonal directions (superposition). This makes neurons polysemantic---each neuron responds to multiple unrelated concepts. Sparse autoencoders (SAEs) address this by learning to decompose neuron activations into monosemantic features (each SAE feature responds to a single interpretable concept). The Anthropic monosemanticity line of work (2023-2024) scaled this to frontier models, identifying millions of interpretable features.

What a research interview question looks like: “Explain the superposition hypothesis. Why does it make interpreting individual neurons difficult, and how do sparse autoencoders help?”

Efficient architectures beyond Transformers

The problem. The Transformer [8] has $O(n^2)$ attention complexity. For long sequences or resource-constrained deployment, this is prohibitive. Alternative architectures aim to match transformer quality while improving efficiency.

Mamba [3] (Gu and Dao, 2023) is a selective state space model (SSM). Unlike prior SSMs that process all input tokens with the same parameters, Mamba’s parameters are input-dependent: the state transition matrix is a function of the current input token. This selectivity allows Mamba to filter irrelevant tokens and focus on relevant ones, similar to attention but with $O(n)$ sequential complexity and $O(1)$ memory for autoregressive generation. Mamba matches transformer perplexity on language modelling at the 1-3B parameter scale and is significantly faster at long sequences.

RWKV [4] (Peng et al. 2023) is a recurrent architecture that reformulates as a transformer-like attention but with a recurrence that enables $O(1)$ per-token generation. RWKV uses a time-mixing mechanism with a linear attention-like structure. Training is parallelisable (like a transformer); inference is serial (like an RNN). This combination is compelling for on-device deployment.

Trade-off. Both Mamba and RWKV give up the attention mechanism’s ability to attend to arbitrary past tokens. They have bounded memory (the SSM state is fixed-size), so very long-range dependencies that require remembering a specific token from thousands of steps ago are harder. Hybrid architectures (alternating SSM and attention layers) are a practical compromise.

What a research interview question looks like: “What is the key difference between Mamba and a vanilla RNN, and why does the selectivity mechanism matter for language modelling?”

Long context

The problem. The quadratic memory and compute cost of attention limits context to 2K-8K tokens in most trained models. Many real tasks (legal document analysis, multi-document summarisation, long-horizon coding) require context windows of 100K-1M tokens.

RingAttention [5] distributes the attention computation across a ring of devices. Each device holds a block of queries and a block of keys/values. Devices pass their key-value blocks around the ring while computing their local attention, so the full $n \times n$ attention matrix is never materialised on a single device. This scales context length linearly with the number of devices.

Other approaches. Sliding window attention (Mistral, Longformer) computes local attention within a window plus global attention sinks. Sparse attention (BigBird, LongFormer) uses structured sparsity patterns. Retrieval-augmented generation effectively extends context by retrieving relevant passages. Positional encoding extrapolation (RoPE NTK scaling) allows models trained at 4K context to generalise to longer sequences with fine-tuning.

What a research interview question looks like: “How does RingAttention scale attention to 1M tokens? What are its limitations?”

Data quality frontier

The problem. The dominant paradigm assumes more data is always better. The Phi line of models (Phi-1, Phi-2, Phi-3, Microsoft) challenged this: models trained on small, extremely high-quality “textbook quality” data matched or exceeded models trained on 10-100x more internet text. The implication: data curation matters more than data volume at the frontier.

Synthetic data. Large frontier models can generate training data for smaller models. This creates a potential self-improvement loop: a model generates data, a smaller model trains on it, and the smaller model improves. The risk: amplification of errors and biases in the generator model. The Alpaca / Orca / OpenHermes lines of work show that GPT-4-generated instruction data can dramatically improve smaller open models.

Data attribution. Which training examples were responsible for a specific model behaviour? Influence functions (an approximation of leave-one-out retraining) can attribute a model output to specific training examples. This is relevant for copyright, safety (finding which training examples caused a harmful behaviour), and data quality improvement.

What a research interview question looks like: “Describe the Phi experiment’s key finding and its implication for the scaling laws hypothesis.”

Summary: the frontier as a map

Pitfalls

Interview Q&A

References