> General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.

I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.

Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.

Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.

Models don’t know that they don’t know.

> know that they don’t know

And we are waiting for architectures that do - because it's duly.

Honestly it seems like a job for the harness, rather than the model. Sample the model with the same question, perhaps with varying temperature (?), and use that to establish a degree of confidence in the answer. If the model provides very different answers every time, respond that it doesn't know. If it responds with the same answer usually but a different answer sometimes, respond with moderate confidence. If the model always responds with the same answer, respond with certainty.

If we want to implement Intelligence, and especially now that "the box is open" we must, we can play with the "intuitive" LLM architecture to understand it and squeeze it to its potential yeld, but at some stage we have to actually implement intelligence. That implies notions of confidence and a Foundational Theory of Knowledge (knowing why you know something), among the rest (one shot learning, update through reflection etc.).

that could work sometimes but that's terribly hacky engineering

I... really don't agree. Being confident in using documentation to program in a variety of environmnets is not "terribly hacky engineering". Good engineering involves leveraging documentation well.

Ideally it would run a web search to check questionable knowledge. Google-fu has been one of the most important SWE skills for a long time!

I actually disagree that this doesn’t mean anything. I understand the contention that it’s not measuring the quality of the model in general, but I think it is measuring something useful.

A good example of this is planning hardware projects - a larger 200b plus model like DeepSeek V4 flash will recommend parts like motors, real time clocks, voltage regulators etc and it will do so providing exact model names and specifications.

I wouldn’t expect a smaller model to encode all of this information, but it is helpful to understand where that cutoff is because it changes what the model might be useful for. It is a very crude way of measuring because it comes down to the balance of training data at sizes this small - but I do think it conveys something that is helpful in real world tasks.

Agreed. It's just to see how much knowledge the model still has for its size. For comparison, no other model of this size has ever given me the correct answer to the Keel question. Even GPT-OSS-120B got it wrong, but it's expected for its relative old age.

I concur, treating models as question and answer machines and judging them on recall is meaningless, unless you're measuring quantisation impact on a foundation model maybe.

100% agree, it's really just for kicks&giggles. The fact that the model answered correctly, unlike every other model of its size before it, still pleasantly surprised me.

> on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode.

Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!

Which setup did you use? MLX/GGUF, Quant, Engine (e.g. llama.cpp or MTPLX, etc)? There’s so much variety these days.

It was in LMStudio (llama.cpp), Q4 by Unsloth. Applied the recommended defaults published by Unsloth.

Meaningless yet fun fact: DeepSeek V4 Pro 0813 made a much worse icon for the same app, and only produced an SVG I had to convert manually to .png. Qwen3.8 made a perfect icon in .png. I don't yet know how it did it, but it did it.

Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.

copilot isn't a model. it's a model hoster.

> write a fully-featured todo list app in JS

Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).