Always has been. My prediction is that both OpenAI and Claude will go bust unless they deliver a killer product. And unlike scrappy startups, they have a pretty serious deadline because creditors will come a-knockin'.
There's little to no functional difference between Kimi, Qwen, Sol, Opus, etc. All flagship models are within like 1-5% of each other and the real moat will be what's always been the hard part: making a good product.
> All flagship models are within like 1-5% of each other
Don't know about that.
I'm using code review of my lone lisp project as a benchmark. It's a massive parallel code review where a coordinator cuts up the codebase into sections and dispatches agents to consider each part from different perspectives like quality, maintainability, consistency, correctness, rigor, etc.
Ran a complete Fable/max code review. Took over a month on a subscription. Now I've switched to OpenAI and am repeating the exact same review with Sol/max.
It's still not done yet but preliminary findings suggest Sol can only reproduce 70-90% of Fable's findings. So I think these models aren't as close as we've been led to believe.
Absolutely. I have an independent audit pass to verify claims.
My methodology consists of launching a 242 cell parallel code review matrix and committing all Fable/Sol max effort agent prompts and their full reports to a private orphan branch on the repository. This is the part that is taking me months to complete. This thing can kill my $100 subscription in about 12 hours.
When done, these raw findings will be semantically deduplicated and merged into a list of findings per model. This list will then be audited for hallucinated or otherwise made up findings. This will refine the list, and hallucination rate is its own data point. I'm also counting things like cybersecurity refusals and downgrades.
When all this is done, I'll analyse the final results and publish them on my website.
Some preliminary analysis:
Which code review lenses were the most valuable, where value is defined as number of serious issues identified? Rigor, followed by tests, robustness, correctness, and so on. I was able to create a tier list of reviewer personas using evidence! I can now run focused code reviews using the highest value lenses.
What's the most expensive code review? Correctness and rigor, of the lone lisp machine specifically.
API costs per finding? $0.91 to $3.77. API costs per serious finding? $8.94 to $28.51. All Fable.
How long did it take? 28.2 calendar days, 66.3 agent-hours.
Is it worth it to run the code review multiple times? If a review matrix's defect capture probability is 57%, then a second run captures 81% of the estimated/extrapolated defect population, a third run captures 92%, a fourth run captures 97%, and further runs yield severely diminishing returns. Probably worth it to code review important stuff three times.
What's the impact and cost of the safety classifier? Out of the 52 Fable review cells that triggered the safety classifier, 35 died without producing any output whatsoever, so 67.3% of the cells were a complete waste of tokens. 25% produced at least some output.
Does the safety classifier trigger most often on the important code that actually needs SOTA models? For the most part, yes. Fable was most often barred from reviewing the most important and complex files in the codebase, such as the virtual machine, the parser and I/O layer. These files also have the most CRITICAL+HIGH severity findings. Only a couple outliers broke this pattern.
It's not hard to sell a dollar for 50 cents. Imo, these businesses are pretty clearly not doing well financially (the revolving door of unvested C-levels is a good hint, the constant postponing of S-1s is another).
That graph looks a lot less impressive when not "annualized". Annualized revenue can be gamed in a number of ways, and is a big reason companies tend to compare YoY to investors once they're public.
Their quarterly revenue looks exactly the same. It's literally the same data. Annualized revenue can be gamed when it is fluctuating. It can't really be gamed when it is growing exponentially for three years straight. There's nothing to cherry-pick.
The Chinese models are cheap because no one is using them. But they can't actually afford (or have capacity) to serve enough people to kill the giants. This is evidenced by them all recently hiking prices or limiting usage.
It's possible that they build out in China at an unreal pace, China doesn't have concept of "community input" to drag down state projects, but then you are left giving your IP to China. Just ask western hardware businesses how well that goes.
No one is in a better position to more efficiently use it - that's what happens when you poach every top 0.01% engineer/researcher in AI.
They're guaranteed to get over whatever hump you think they're in unironically. Uber/Tesla have been in far worse situations and despite Elon being an idiot/liar you see how they performed when even the most bullish of investors called for their heads
I think I generally agree with you - OpenAI and anthropic will probably succeed here. If the ai bubble pops, they’ll come out on top.
I just think the whole “moat” discourse is silly. OpenAI and Anthropic’s success depends on the same thing every business’s success depends on: their customer base, and their continued delivery of services their customers want to pay for. Not their tech or their compute or anything else. They have no moat because moats aren’t a thing.
There is a massive difference even between Opus and Fable, same provider, before various harnesses and other optimizations come into play. Don't be deceived by rankings and benchmarks, try for yourself.
The problem is that most of the volume doesn't come from proprietary products, it comes from API use which has no stickiness.
Claude already has a killer product (claude.ai/chat is a Swiss army knife) but just relying on people typing stuff into chat is not enough to sustain the company.
The other strategy is entrenching yourself as the LLM of choice into existing products (like ChatGPT is on Apple products).
Always has been. My prediction is that both OpenAI and Claude will go bust unless they deliver a killer product. And unlike scrappy startups, they have a pretty serious deadline because creditors will come a-knockin'.
There's little to no functional difference between Kimi, Qwen, Sol, Opus, etc. All flagship models are within like 1-5% of each other and the real moat will be what's always been the hard part: making a good product.