It's annoying that most file formats don't checksum their own content.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.
Though fair enough, it could offer it as an opt-in thing.
Checksums use CPU cycles. SQLite will do checksums with an extension (https://sqlite.org/cksumvfs.html) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.
I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.
FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).
If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.
> SQLite is a Recommended Storage Format for datasets according to the US Library of Congress. Recommended storage formats are formats which, in the opinion of the preservationists at the Library of Congress, maximizes the chance of survival and continued accessibility of digital content.
> It's annoying that most file formats don't checksum their own content.
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.