I feel it's slightly concerning that the same text contains:
"There’s no room for error."
and
"Someone on the product team will usually play with the feature on our staging environment trying their best to break things."
The first sentence seems to imply that they try to maintain the highest quality possible. Yet the second sentence does not really fit that idea. The article pretty much only lists the tools they use.
But on a scale from Random PHP site to NASA space shuttle software, where is zenpayroll located?
To be fair, the "play" sentence comes after a whole bunch of description of spec writing, test coverage, code reviews, and automated test suites including using anonymized real data. Doing a final human "check it out" step doesn't invalidate all that, and could be helpful.
If you truly have "no room for error" (i.e. you are shipping code to a hundred-billion-dollar space vehicle), you might have:
- Very strict coding guidelines
- Rigorous code review
- An extremely thorough test suite
- Multi-stage approval through a QA process involving many pairs of eyes and thorough, formally defined and scientifically rigorous test procedures
- Static analysis and possibly even algorithms proved correct in Coq
Simply playing with it in staging for a while and determining that everything looks good is NOT the kind of testing that you do when there's "no room for error."
Webdevs (particularly on HN) are highly vocal about TDD/automated testing in general and there is a great deal of tooling around testing Rails and Django. Code review is also part of the fairly dated Joel Test and widely regarded on HN as a good idea.
These are substantially higher bars than "play with it for a while and see if you can break anything."
even if software is formally verified, and even assuming the specification is both totally accurate and complete, there is plenty of room for issues at software deployment in a multi-component system such as this
Even so, imho on a staging environment everything should be ready to deploy barring either functional acceptance or very, very unforeseen technical issues that didn't show up systems that mimic production just a tad less closely.
Staging = "ready to roll", not "still needs testing to see if things break". Things shouldn't break on staging.
Of course you can forgo manual QA if your automated testing is so perfect that staging is just a formality. I'm guessing ZenPayroll has that covered, given that they deploy to production "several times a week".
So I assume "trying their best to break things" is just a flippant description of final acceptance, not QA testing.
"There’s no room for error."
and
"Someone on the product team will usually play with the feature on our staging environment trying their best to break things."
The first sentence seems to imply that they try to maintain the highest quality possible. Yet the second sentence does not really fit that idea. The article pretty much only lists the tools they use.
But on a scale from Random PHP site to NASA space shuttle software, where is zenpayroll located?