Randomly looking at the "Trophy Case", it looks like most of these errors are run-time "panics". We can break Rust errors down into three main categories:
1. Compile-time errors. This includes most memory-related errors, which are mostly caught by the borrow checker. These are very serious errors, often with ugly security consequences. Rust's big selling point is that it can catch many different kinds of errors at compile time—but not all.
2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak.
3. Expected run-time errors. These are mostly reported using return values of type Error, which is the normal way to handle errors in Rust.
Most of the errors caught by AFL seem to be errors in group (2) that ought to be in group (3). In most cases, these errors couldn't be moved into group (1), because they're not the kind of thing that's easily caught at compile-time.
So this is a really cool tool for Rust developers, especially ones working on libraries that parse untrusted input. I was especially impressed by the fact that AFL could discover overflow errors, which Rust normally only protects against in Debug mode.
> 2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak.
I still argue, to every Rust user I meet, that they should turn on overflow checking in Release builds too.
I strongly disagree with the idea that it's a Debug feature only. If it's not always on, then it's not actually a language feature. The performance hit, if you selectively opt-out for profiled critical paths (and thoroughly review them), is usually very small.
In my opinion the default should switch to "opt-out" instead of "opt-in" for Release builds.
> I still argue, to every Rust user I meet, that they should turn on overflow checking in Release builds too.
Yeah, I remember when the debug-mode overflow checks were added right before Rust 1.0. There wasn't enough time remaining to thoroughly measure the performance impact of these checks in release mode or to implement compiler optimizations to reduce the cost. So IIRC, the decision was made to allow integer overflows to panic, and to turn on checks in debug mode so that the ecosystem wouldn't come to depend on the absence of checks.
But AFAICT, the debug-mode checks are basically still a placeholder for some indefinite future version of Rust that wants to fix this issue properly, if the performance cost can be made low enough. And in the meantime, you can still turn the checks on for release builds if the performance hit is acceptable for your application. I'll consider your recommendation when building binaries in the future!
It's a fine line to walk. Performance is a deal breaker for a lot of people, whether it rightly should be of not. It doesn't matter how much safer Rust is if it's not used much.
If your goal is purely to make Rust as safe as possible, then you should advocate the view in your comment.
If your goal is to increase Rust adoption, and/or decrease the use of less safe languages for development in the future (in whichever prioritization of the two you like), then IMO the route described in ekidd's comment seems the quickest route there, IMO.
> If your goal is purely to make Rust as safe as possible, then you should advocate the view in your comment.
> If your goal is to increase Rust adoption, and/or decrease the use of less safe languages for development in the future (in whichever prioritization of the two you like), then IMO the route described in ekidd's comment seems the quickest route there, IMO.
My view is this is a false dilemma.
The "performance" case is not performance, because the benchmark is broken code. It does not work, because it is not secure/safe/reliable. It is incorrect to compare the performance of code compiled without overflow checking against code which is compiled with it enabled. They simply aren't the same program.
A correct performance comparison would be a program with manually inserted overflow checks everywhere, against a program with the overflow checks done automatically (by compiler).
Or, a performance comparison where code is selectively opted-out, where a critical path bottleneck is identified.
The same can be said of the performance comparison of "array index bounds checking". One program works, the other does not, so it isn't a fair comparison.
> A correct performance comparison would be a program with manually inserted overflow checks everywhere, against a program with the overflow checks done automatically (by compiler).
Correct is relative to the goals of the person assessing the comparison. If that person values performance over safety to enough of a degree, them a small performance loss will outweigh safety (regardless of whether you or I think this is a useful way to compare). If your goal is get people using Rust, and you believe people value performance, you can't ignore this and expect things to turn out how you like. This is the balancing act of a community and of advocates, making hard choices about what they would like to do, and what they feel is needed due to the constraints of reality.
> The same can be said of the performance comparison of "array index bounds checking". One program works, the other does not, so it isn't a fair comparison.
No, for some subset of array bounds checking, one works and the other doesn't. For the rest, they both work, because it isn't out of bounds. Whether you think people should program defensively and assume their code could fail, some will instead choose to assume they can deal with that as the programmer and choose to do so because it is more performant. You can either write those people off, or meet them half-way, and hope to escort them the rest of the way. Rust apparently chose the latter.
You can do it with either -C debug-assertions (stable) or -Z force-overflow-checks=on (unstable). The difference is that the former will also enable debug_assert! in your own code.
> These are mostly reported using return values of type Error
However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>, e.g. writing to a file or a socket will get you Result<usize, io::Error>, values of which can be Ok(usize) (the number of bytes written) or Err(io::Error) (an I/O error which you can further inspect).
Although if you want to, you can use Result<T, Box<Error>> where Error is a trait implemented by types representing some error information.
You then lose the ability to inspect errors other than for reporting them wholesale, but the APIs are slightly more uniform, type-wise.
> However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>
Yes, this is a more complete and precise explanation. :-) I rather like Rust's error handling in practice—it adds only a small amount of syntactic noise, and even the annoying bits (such as converting and wrapping error types) can mostly be isolated in a single file per project.
If you're parsing untrusted input, you should try to systemically avoid panics. Easier said than done, but I have started a small framework for protocol parsers called untrusted.rs that helps: https://github.com/briansmith/untrusted.
The main idea is that most panics (in parsers) are going to be caused by using the indexing operator on slices containing the untrusted input, so we wrap the input in a type that hides the slicing functionality, giving us only methods that return 'Result's.
1. Compile-time errors. This includes most memory-related errors, which are mostly caught by the borrow checker. These are very serious errors, often with ugly security consequences. Rust's big selling point is that it can catch many different kinds of errors at compile time—but not all.
2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak.
3. Expected run-time errors. These are mostly reported using return values of type Error, which is the normal way to handle errors in Rust.
Most of the errors caught by AFL seem to be errors in group (2) that ought to be in group (3). In most cases, these errors couldn't be moved into group (1), because they're not the kind of thing that's easily caught at compile-time.
So this is a really cool tool for Rust developers, especially ones working on libraries that parse untrusted input. I was especially impressed by the fact that AFL could discover overflow errors, which Rust normally only protects against in Debug mode.