Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.

On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.

So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.

Yeah, makes sense; e2e is different and valuable, KLD is not a replacement.

As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?

Nope.

If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.

I am curious what's the actual formula.

I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?

It's really simple, basically just the magnitude of the value vector, weighted by QK dot product, summed across all attention heads and layers.

When I started, I expected I'd have to experiment a lot to find something comprehensible. But this simple computation can already show some patterns.

Might be cool to try different colors for the different attention heads instead of summing them across all attention heads, and making the backgrounds composed of stacked color layers? So you can see how each attention head attends to tokens individually.

Nice! Sometimes the simplest approaches work the best.

If you want quick access look at google images for "transformer attention formula" there are some interesting depictions

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.

The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless.

I'm not that well versed in statistics, but a standard box plot is probably the best alternative

A single result is binary. All we get from a run is which tasks were solved, which weren’t.

I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.

So, if there are irrelevant remarks, these are mine. :)

Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).

Hi,

If you ever get to writing a blog post about kv cache quantisation, i'm interested in quantising K differently than V

If you like, you can run these tests yourself as well. It is ~$500 per a single combination, and assuming no failed runs.

Nice! One thing I am missing is an easy „go up” a taxonomy group.

Is it a subtle effect that cannot be explained by Newtonian gravity (i.e. a different potential affecting, V(z) in the Hamiltonian)?

No. From the paper linked above [0]:

> The phase of free fall is predicted in a purely quantum manner to have a dependence m/6 g^2T^3/ℏ + gmzT on the free-fall time T, where m is the mass of the object, g is the gravitational acceleration relative to the surface of Earth, and z in the spatial coordinate in the direction of gravity. This prediction follows the calculated phase accumulated by an object accelerating in a linear potential, and has been made starting from almost one hundred years ago by Darwin, Kennard and others.

Apparently the phase shift is derivable from just adding a linear potential term mgz to the Hamiltonian.

[0]: https://arxiv.org/pdf/2502.14535

Don’t ask. Start from a few blog posts and see traction, read feedback.

You will also see hos long it takes - and what is thd difference between an idea and making it real.

An interesting catch!

Maybe it suffices to make this pre-title "the ai-design-slop fingerprint · defs 2026.09" lowercase.

In any case, from what I see what is AI aesthetics on the website: pre-title and horizontal lines.

With Internet, we live in an always-on culture.

It contrasts with times before, when we actually had to wait for a monthly magazine, and even if we wanted to watch a movie, it was aired (say) the next Tue 7PM.

While now we have all convenience, there is no anticipation - or rest.

Hey. We still have TV shows that air once a week (and people loudly complain that it takes too long)

While I like this index, calling in "Intelligence" might be confusing - it is a mix of coding and knowledge.

Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench).

In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted:

> Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.