I was kinda on the fence, but I was considering my target: Journalists. A journalist may not notice simple differences like an extra space here or there when reading, but probably wouldn't retype a double space. I agree that this should be automated in some way, but it's a bit of an arms race.
What kind of arms do you envision for normalizing text down to Latin-1, or even ASCII, while normalizing any whitespace to single space characters?
There is a known variant when every subscriber of a confidential text receives a slightly different copy with the same meaning. But it's much harder to implement, and does not scale.
Not that much harder to implement. You could automate that too with some basic text substitution. Eg replacing various instances of "and" with ampasand or plus sign. You could also vary joined words like "without" / "with out", and alternative between
the types of quotation marks used (as there are several in unicode).
And this is without breaking into more intelligent heuristics where you swap out synonyms ("more intelligent" because you'd need to be careful not to alter passages that need to be kept verbatim, like quoted text or where a synonym might alter the context of the sentence.but with a little care I think that is achievable as well)
I'm curious as to why you think a linter tool for sanitizing the text is less effective than the other measures you describe. As long as the target problem is zero-width characters, this seems like it should be more effective than all but #1. I would agree if you include the synonym or other lexical fingerprints.
Humans make fixes that are hard to codify in programming. I know it isn't perfect, but with my audience in mind (journalists) I thought it was probably safer than something automatic.
I was kinda on the fence, but I was considering my target: Journalists. A journalist may not notice simple differences like an extra space here or there when reading, but probably wouldn't retype a double space. I agree that this should be automated in some way, but it's a bit of an arms race.