Blog

On the arXiv today, with Purvam Jain, Preethi Jyothi and Vihari Piratla. https://arxiv.org/pdf/2605.30788 Modern LLMs have become increasingly good at a variety of languages. How does one detect cross lingual gaps in their abilities? We propose a set of synthetic puzzles that can be generated using base templates and then scaled up dynamically. This avoid translation errors; the tasks are clearly quantifiable; are comparable across languages; and as models improve it is trivial to scale up the test. The flip side is that these tests can’t detect more subtle issues like lack of awareness of nuance, or differences in creative-writing skill. Nevertheless, empirical experiments across many models, and seven languages — English, Hindi, Arabic, Chinese, Japanese, Tamil, Telugu — show that this benchmark can successfully detect cross lingual gaps. The complexity-accuracy graphs that we studied with Praneeth from a theoretical perspective in https://arxiv.org/abs/2601.14175 turned out to be useful. By studying the entire complexity-accuracy curve, we sidestep issues that arise if one tests the model at only a single complexity level.

This is an article I wrote 7 years ago analyzing the allegations against the Iranian nuclear program and the history of the conflict between the US and Iran. Reading it again, I think it is still useful to understand some of the context and history, which is usually glossed over in the media. The article includes a number of references to further reading. World history moves slowly on human timescales. But looking back, some broad trends were quite clear. The last line of the article is definitely one that I would write again today.

https://mronline.org/2019/07/01/imperial-overreach-in-iran/ (Originally written for the Research Unit in Political Economy, but then republished in Monthly Review online.)

Today, with Hao, Andreas, Mark, Lisa and Carlos: https://arxiv.org/abs/2602.06543

We tried to ensure that our main points are clear from the intro, even for a non-expert. I encourage you to click through and read at least the first few pages.

Here is a longish semi-popular summary of some of the background for those who are interested.

While formulating the information paradox, Hawking assumed that observables inside the black hole are independent of those outside. (More precisely, algebras of operators inside and outside the black hole commute with each other.) This assumption is so common that Hawking called it a “basic assumption of quantum theory”. Page disagreed with Hawking’s conclusion and suggested that information would emerge gradually from the black hole, according to what is now called a “Page curve.” But Page’s argument relied on precisely the same assumption of independence of degrees of freedom.

In quantum gravity, this assumption fails. Instead, under weak assumptions, one finds a “principle of holography of information”: observables in the black-hole interior (or any compact region) can be rewritten in terms of observables in the exterior when the global state is pure.

“Holography of information” does not give us a full-fledged holographic duality like AdS/CFT. On the other hand, it applies to asymptotically flat space and does not require us to know details about the UV-complete theory.

This leads to the following picture. Consider surrounding a black hole with a detector. If the detector makes only coarse-grained observations, it naturally loses information, and will see a rising curve for the von Neumann entropy of the exterior. But if the detector makes suitably fine-grained observations, it will find a von Neumann entropy that is constantly zero. The exterior observer knows about the interior at all times — and not just after the “Page time.” So holography of information resolves the information paradox but simultaneously trivializes the Page curve.

This is somewhat different from the historical bias within the hep-th community, which has been that Hawking was wrong but Page was right. And, several recent computations of the Page curve in toy models of black holes seem to vindicate this bias.

In 2021 the G7(=all of us + Sanjit) noted that these models relied crucially on a nongravitational bath. In the presence of this bath, the bulk theory of gravity follows nonstandard dynamics. (The graviton becomes “massive”.) The G7 showed that when gravity was turned on in the bath, the Page curve trivializes. In the presence of a bath, the exterior observer can reconstruct only a part of the black-hole interior called an “island”. But the G7 showed that islands are inconsistent in standard gravity because we cannot have exactly-defined algebras for a compact region when the global state is pure. I’ll call this the “inconsistency paper.”

To be clear: the “inconsistency paper” doesn’t suggest that computations with a bath are wrong. These computations are nice and lead to interesting questions about the gravitational path integral that we are still exploring. But they provide a misleading physical picture for realistic black holes.

Last year, Stefano, Henry, Chang-Han and Geoff wrote an “Apologia for islands” arguing that islands and the Page curve are relevant even in standard gravity. I’ll refer to this as the “apologia paper.”

The “apologia paper” doesn’t raise any technical objections to the “inconsistency paper.”

The “inconsistency paper” relied on the observation that the exterior observer can measure the Hamiltonian in gravity. The “apologia paper” sets up configurations where the Hamiltonian is inaccessible to the exterior observer. For example, in AdS, one can prevent the observer from making measurements on part of the boundary. This is like having a detector with a “blind spot” and moreover, rather than studying a generic black hole, one studies a black hole that is localized right next to the blind spot. In flat space, it is possible to consistently discard the Hamiltonian by hand at null infinity. If the Hamiltonian is inaccessible to the exterior observer, the argument from the “inconsistency paper” obviously doesn’t apply.

These caveats were known even before the “inconsistency paper.” In 2020, when we studied the holography of information with Alok, Pushkal, Siddharth, we discussed a Page curve of this kind in AdS. And we also showed that the Hamiltonian can be discarded from the set of observables at null infinity to get a Page curve.

But these Page curves don’t teach us about information “emerging” from the black hole. They simply tell us about how information is redistributed between the “blind spot” and the rest of the detector. And the islands one gets this way are not the islands we studied in the “inconsistency paper.” Rather than being compact regions, they always have an asymptotic piece corresponding to the “blind spot” in the detector.

This is why our paper today is titled “Seeing Page Curves and Islands with Blinders On.” We have many more details, including a detailed discussion of “relational observables” and why they can’t be used to make islands consistent; and an explanation of how holography of information is important even in the presence of a bath.

I doubt that there are technical disagreements on any of these points. So, I’ll end with some personal perspective. Arguably, the effect that observables outside a compact region are sensitive to observables inside the region is one of the most interesting facts that we have learned about quantum gravity. It explains how information from the interior ends up in the radiation. So I feel that it makes sense to embrace and explore this physics, rather than finding innovative ways of obscuring it, as one must necessarily do to see a Page curve.

If you are in Hyderabad over the weekend, here are a couple of fun events that I’ll be part of on Saturday. A panel discussion at the Hyderabad Lit. Fest and then a public outreach lecture at Lamakaan. Both events are free and open to the public.

Here is something different, with Praneeth Netrapalli.

https://arxiv.org/abs/2601.14175

LLMs have made spectacular progress over the past few years. Yet they still make errors on relatively simple tasks and we don’t have a good understanding of why these errors arise. This is reflective of a broader problem that should bug all theorists: our scientific understanding of these systems greatly lags the technology.

This paper examines these errors in a very simple setting — the ability of the model to implement deterministic tasks like arithmetic, list reversal etc.

Even state of the art LLMs are poor at these tasks. It is hard to see this from the web interface because the models have been trained to invoke external tools like Python when they encounter these tasks. So if one asks ChatGPT to implement a long multiplication, it returns the right answer by invoking an infinite-precision tool behind the scenes, even though the model itself doesn’t have the ability to perform the task natively. Because models can be taught to use these tools, their weakness at tasks like arithmetic is not a question of direct practical interest. But it is still of scientific interest because these tasks offer a controlled set of problems where one can set up and verify theoretical models.

We theorize that LLMs make errors because “noise” in the “attention mechanism” accumulates to cross a threshold. This leads to a quantitative prediction for the relationship between the expected accuracy and the length of the task. (For those who are interested, the prediction is that the accuracy, a, and the complexity c are related via: $a=\gamma(q/2, q/(2 r c^2))/\Gamma(q/2)$, where q,r are two parameters with a simple interpretation that depend on the prompt and the model and \gamma is the lower incomplete gamma function. )

Our derivation is inspired by the philosophy of “effective field theory.” The LLM itself has hundreds of billions of raw parameters. But by thinking about it the right way, one can argue that these parameters reorganize themselves into two effective parameters q,r.

Of course, this is not a rigorous EFT analysis of the kind that we have in physics let alone the kind of theorem that computer scientists might want. But its still very nice that a clear set of assumptions and arguments lead to a simple two-parameter effective model.

We validate this formula using 200,000 prompts across many different tasks and three state-of-the-art models. It works surprisingly well. But, interestingly, it doesn’t always work and when it doesn’t work, it teaches us something about the functioning of the model.

There has been a fair amount of previous work on this question. One set of papers suggested that the failure of models on tasks like arithmetic and dynamic programming indicates a fundamental limitation in its expressive power to implement “compositional” functions. And last year, a paper from Apple suggested that these failures indicated a “collapse of reasoning.” But our diagnosis is different — and we have a quantitative model that fits the empirical data nicely not just when a = 0 and a = 1 but all the way in between.

P.S: One contribution that we definitely make to the ML literature is to add the notion of error bars! (I haven’t yet read an ML paper that has error bars on its empirical graphs. 😅)

There has been debate on the suspension of Israel from the IOAA. Alok and I wrote this oped for the Hindu today, which can be read here: https://drive.google.com/file/d/1RjqaVpzwJliMJexl5iRK-pVv1qSZlfQJ/view?usp=sharing

I would also recommend the article by Madhu and Aditi in the Scroll that appeared a few days ago: https://scroll.in/article/1086205/why-we-signed-a-petition-asking-for-israel-to-be-suspended-from-from-the-astronomy-olympiad

There is a news article here: https://www.thehindu.com/sci-tech/science/rift-among-indian-scientists-as-international-olympiad-on-astronomy-and-astrophysics-bans-israel-from-future-editions/article70009127.ece

and also a post here: https://sciencechronicle.in/2025/08/31/ioaa-statement-clears-the-air-on-israels-suspension-from-future-astronomy-olympiad-events/

I’m happy to have more discussion, but I request that you read all the articles above before commenting.

I wanted to think through various issues at the interface of science and society in relation to recent progress in AI/ML. So I reviewed “The AI Con” by Emily Bender and Alex Hanna for The India Forum. If people get a chance to read through the review, I’d be glad to hear comments/criticism.

https://www.theindiaforum.in/book-reviews/sharp-industry-thin-science

Israel has been suspended from the International Olympiad on Astrononomy and Astrophysics (IOAA). The IOAA is part of a network of science Olympiads that seek to identify the most talented high-school students in each country.

This year the IOAA was held in Mumbai. Israel had pre-registered for the event although, eventually, it did not send a team. Members of the Indian and international scientific community sent the letter below to the IOAA requesting it to suspend the State of Israel, while allowing students from Israel to participate as individuals.

The initial signatories of the letter were Alok Laddha, Suvrat Raju, Ronak Soni, Sandip Trivedi, Ravinder Banyal, Ashoke Sen, Nissim Kanekar, Ahmed Abbes and Pierre Vanhove. In all, more than 500 academics signed the letter.

The IOAA board, comprising representatives from 64 countries discussed this proposal on 18 August. While the issue was not relevant for this year due to Israel’s absence, a significant majority of the board voted to suspend Israel from future Olympiads while allowing individual students to participate.

We greatly appreciate Prof. Aniket Sule, the president of the IOAA for his integrity and commitment to democratic values in placing our letter before the international board for an open discussion and vote.

This sets a significant and important precedent: countries that are guilty of genocide and practice apartheid are not welcome to send official representatives to international scientific and cultural events. Individuals from those countries can participate but not as official representatives of their governments.

I hope that, from next year, the other science Olympiads and even the Olympic games follow this precedent.

Of course this is only a small step in the larger scheme. But to whatever small extent, I hope this conveys the horror and disgust with which the rest of the world views Israeli policies in Palestine and encourages the Israeli government to change course immediately.