A lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?
1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.
2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.
3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox.
There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof.
This should exist for good debugging anyway.
It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.
Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.
Build-related fixes are only treating the symptoms, not the disease. The real fix would be better sandboxing and capability-based security[1] built into major OSes which make backdoors a lot less useful. Why does a compression library have the ability to "install an audit hook into the dynamic linker" or anything else that isn't compressing data? No amount of SBOMs, reproducible builds, code signing, or banning binaries will change the fact that one mistake anywhere in the stack has a huge blast radius.
Note that the malicious binary is fairly long and complex.
This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.
You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.
autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.
This requires a more comprehensive redesign of the build process. Most Linux distributions also run the tests of the project they're building as part of the build process.
Profile guided optimization is, unfortunately, wildly powerful. And it has a hard requirement that a casual link exists from test data (or production data!) to the build process.
We should be able to produce a tar and a proof that tar was produced from a specific source code.
Quote from the article:
That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.
Zero Knowledge virtual machines, like cartesi.io, might help with this. Idea is to take the source, run a bunch of computational steps (compilation & archiving) and at the same time produce some kind of signature that certain steps were executed.
The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.
The advantage of Zero-Knowledge technology in this case is that one doesn't need to repeat the computational steps themselves nor rely on a trusted party to do it for them (like automated build - that can also be compromised by the state actors). Just having the proof solves this trust problem mathematically: if you have the proof & the tar, you can quickly check source code that produced the tar wasn't modified.
I don’t think zero knowledge systems are practical at the moment. It will take over around 8 orders of magnitude more compute and memory to produce a ZKP proof of generic computation like compilation. Even 2 orders of magnitude is barely acceptable.
I haven't looked at Guix but in the discussions around this exploit for NixOS they mentioned that regenerating autoshit for xz-utils would not be something they can/want to do because that would add a lot more dependencies to the bootstrap before other packages can be build. Kind of funny how a requirement for bootstrapped builds can add a requirement for trusting not-quite-binaries-but-also-not-really-source blobs.
More reproducible builds, maybe even across distributions? Builds based on specific commits (no tarballs like in this case), possibly signed (just for attribution, not for security per se)? Allow fewer unsafe/runtime modifications The way oss-fuzz ASAN was disabled should've been a warning on its own, if these issues weren't so common.
I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?
