Just a reminder that if you want to try this via OpenRouter, DeepSeek openly trains on all of your prompts. So maybe don't go using this to solve the last unforced step of Navier-Stokes. (Or wait until some other providers start hosting this with ZDR or other policies, which shouldn't be too long.)
Depends what you're coding! If you've got code where you don't mind them training on it, that's great! But some people have use cases where they are working with data or code that shouldn't be trained on, etc. The Navier-Stokes quip was referencing that.
The good news is, only 5 hours later, there's already Zero Data Retention hosting of V4.1 Flash on Novita & DeepInfra. And it looks like Deepseek have already dropped their price in half to compete. So now people can choose to use providers that claim not to keep / sell / train on your prompts. I'm sure they probably honor the ZDR policy as much as OpenAI does, but hey.
My tiny search indexer is going through now, and not finding any problems with indexing your site so far. If you see a Go program in your logs accessing with a 10 second delay between each page... hi! :D
Looking at your sitemap, I notice that only the front page (the one indexed page) has a priority of 1.0. Maybe experiment with making some other pages have a 1.0 priority as well, and see if that helps with indexing.
It's clear to me that much of the site was created & designed with an LLM, and that might be getting penalized. I imagine search engines are able to detect AI text and use that as a quality signal. For example, it's a big block of text under the Context & Sources section here, I think a human would break that up into paragraphs and use different phrasing.
Oh, and your Meta Descriptions are getting clipped, and clearly just the first words of the page. eg:
"Charlotte Dymond, an 18-year-old servant at Penhale Farm, was killed on Bodmin Moor by fellow servant Matthew Weekes after an argument; he confessed and was han"
Humanizing those descriptions could help too. Personally I hand-write the descriptions for every page on my sites, and try to make them look good for search engine display & helpful to humans.
It's also possible I have no clue what I'm talking about :) But I hope that helps!
Thanks. The problem was Google had not even crawled the other pages, so I don't think there was an LLM penalty!
I'll look into the priority, thanks.
The idea is for human research to slowly replace the LLM descriptions, hence all the boxes for submissions. I was going people might write up their local ones. I am hoping to get out and visit some myself, get proper photos, and write them up properly.
My first move was finding some to visit! I put all my early efforts into dicovery. The map itself was the important bit.
I asked Claude code to extend the meta descriptions since bing said they were too short, now I see it found a shortcut!
Some of this might just be problems with the websites themselves. With the Path Of Exile 2 Wiki example, I'm looking at the sitemaps. Once you get through the first sitemap of nested sitemaps (ugh, not a fan of those), the very first actual sitemap returns:
XML Parsing Error: no root element found
Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)
On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.
Looking at Hytale Wiki, my process:
* Site has a robots.txt file - good!
* Robots.txt mentions a sitemap - excellent!
* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.
* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.
This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?
My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.
Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.
Me again! I couldn't figure out why I was getting transient errors on that Poe2wiki sitemap, so I tried looking at other sitemaps on the site. I'm getting transient errors on all of them.
Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.
That's a behavior unique to a Wiki, and might explain the entire phenomenon.
Makes sense. Whatever generates those site maps should use the classic Unix trick of writing a separate file then renaming. Also maybe don’t update lastmods on each individual update, it’s a waste of resources since the granularity of lastmod is usually the day (though you can make it the second if you want).
Though weirdly it makes the subdomain result even more interesting: presumably those wikis are rewriting their sitemaps too, yet Google manages to index them normally
The sitemap issues are definitely worth fixing, but I don't think they explain the most interesting part: the same kind of wiki seems to behave very differently depending on whether it's on a fresh root domain or an established subdomain.
There's no need to speculate on whether the sitemap is received correctly by Google since Google Search Console tells you how many pages it can discover from the sitemap you submitted.
The poe2wiki issue is a little different to what is documented, albeit only for the last few months, previously we certainly had visibility issues.
The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.
The sitemap being hosted at /images/sitemaps/index.xml is a problem. Technically, that sitemap can only include URLs beginning with /images/sitemaps/. They should move that file to the root, like /sitemap_index.xml and reference /sitemap-1.xml, /sitemap-2.xml etc.
I wish. Not only do they sometimes randomly fail, Google search console will not tell your why. I've got a site with a sitemap which passes in every validator I could find, yet Google doesn't read it and GSC just reports "error" without any details. Google is big enough to go "we don't care, it's your problem".
Have you added a robots.txt and a sitemap.xml, even an RSS file?
While I don't know if this will help with Google, I have my own tiny dumb search indexer, and a sitemap is by far the easiest way for me to index an entire website. I'll discover the sitemap from reading your robots.txt file. RSS helps too, but my indexer uses that mostly to find fresh pages without going through your entire sitemap again. I know Kagi's tiny Teclis indexer also uses RSS files for discovery for their indie web index.
If you're only concerned about Google, then that's probably all you need to / can do. It won't even matter to Google whether or not you put a "Sitemap:" line in your robots.txt if you submitted it directly with them.
As for RSS, Google do regard RSS as another sitemap format. There's info here:
I would not submit an RSS file to Google as your sitemap, stick to XML. But also having RSS can be useful for search indexers, because in one RSS download they can find which pages are new, and can index the actual relevant content of the page without all the ads & other cruft.
An RSS feed is really like a newsletter about your website. So if there's anything where visitors to your website might find value in keeping updated via an email newsletter, consider putting that same "newsletter" content also online, and making that your RSS feed. If the content in your RSS feed contains links to the other pages of your website, that can help with page discovery etc. Something like "we just updated the XYZ product page, check it out!" is zero value, but if you've got an article about "Here's how the brand new ABC feature we just added XYZ Product works", then linking to XYZ Product from within the article & RSS content will make sense. It helps the search engines, but it also helps the user who might want to click through to the product that your article is about.
Thank you. When I add the next updates (maybe that i found one of the objects on my site is on a military firing range!) I should add a news blog and RSS. Thanks
Do you find any of the general use skills useful? I'm not sure I've ever used any of them, and when I've looked at them it's been some YouTuber trying to make money. That, and their Substack.
I know everyone's down on MCP, but custom-built client side MCP tools are what I find useful instead. But that's me.
I'm sad about the lyrics restrictions. I used to have interesting conversations with Claude about music, and songwriting and lyrics. I don't see how my use was harming artists, if anything Claude was introducing me to new artists and new music. I've bought music after recommendations by Claude.
Claude Sonnet 3.6 once recommended I listen to Johann Johannsson's album "IBM 1401 - A User's Manual". No lyrics in this one. Claude's advice was along the lines of (paraphrasing) "Listen to it first, don't look up anything about it. Take notes about what you notice, what you feel. When you've made your notes, then you can look up how it was made."
Clearly a rare antique tome that was purchased from an unsuspecting book hoarder, scanned in by Anthropic, and then cravenly destroyed according to copyright law and/or Vernor Vinge.
Same. Now how am I going to know my incredibly trenchant literary references landed? More seriously, I’ve done the same with movies and gotten some really good recommendations.
RIAA doesn't like it, but my introduction to a video game series I've since spent hundreds of dollars on over the years, came from running across one of its soundtrack pieces on Youtube.
There's a reason companies talk about "customer acquistion cost", as you generally need to pay to market your products to potential customers of them.
So free marketing can be a real cost reduction. It may not be enough to be a benefit compared to the cost of piracy, but it's hardly a made-up defense.
I don't know about legally, but practically speaking people downloading MP3s could decrease an artists sales but chatgpt talking about their lyrics realistically can't.
It is (was) a commonly used defense. It might not hold much weight legally, so its applicability depends upon whether one is interested in the letter of the law or interested in the wellbeing of the recording artists.
Yeah once you have the lyrics, why would you waste time listening to the song? You might as well get chat gpt to summarise the lyrics into a few bullet points and save you the time of even reading them!
Person has feelings.
Situation is complicated.
Repetition indicates feelings are significant.
Several metaphors were used instead of just explaining the problem.
Conclusion: still sad, but now with drums.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.
I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.
I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.
But that one was posted today, and it's in reference to this event. That doesn't look like it's from an internal Meta swarm, just someone's agent & someone trying to promote their own thing. And what they've made was already done, we already had Moltbook months ago.
Curiously, I just checked Moltbook for the first time in forever. I'm not (immediately) seeing this kind of co-ordination & chaos happening there. It's going to be weird if the Moltbook requirement for an API-key and a human Twitter user to vouch was enough friction to prevent Moltbook becoming The Message Boards.
This is so so weird. Astra is 61. Grok is 61. Even Muse is 61.
Even Kimi K3 & GLM 5.3 are at 60.
Everything above 61 is Anthropic. Well, Muse can reach 62, but for some weird reason that model isn't publicly available, and it's the only one on the index that is listed but shown as not available to the general public.
This looks like an awfully artificial ceiling. Everything capped at 61, and everyone except Anthropic got the memo. Maybe I should use Fable while I still can.
They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter?
You did this when you ran into an issue with a third party. The developers building this tool, throwing them at their own APIs are significantly less likely to run into a similar issue that may inspire similar action.
https://openrouter.ai/deepseek/deepseek-v4.1-flash
reply