Unique salts are absolutely the correct way to rainbow-proof a database of hashes[1]. But a novel hash function will still do the trick and isn't comparable to using a single salt across the entire database, as you originally claimed.
Reverse engineering a novel hash function, while theoretically possible, isn't a capability possessed by the folks dumping passwords from LinkedIn or Riot. And even if it were, doing so would cost more than would be gained by compromising those passwords[2].
Thus, in practical terms, a novel hash function is a great way to undermine attackers, not unlike using Linux or opening files in non-standard viewers (LibreOffice, Nitro PDF Reader, etc.).
1. And if you have access to a good cryptographer, you should absolutely roll a novel hash function as well. Security through obscurity is the proverbial cherry on top of your defensive sundae, i.e. it's only a bad thing if you rely on it exclusively.
2. Not to mention that the whole point of rainbow tables is that they've already been computed. If an attacker has to reverse engineer a hash function to create a rainbow table, that largely defeats the rainbow table's purpose.
1. No. A good cryptographer will tell you to not re-invent crypto and to use existing peer-reviewed hash functions. This is why the NIST is hosting an open competition for selecting the future SHA-3.
2. This is exactly what happened with the custom proprietary "obscure" GSM A5/1 encryption algorithm. First it was reverse engineered (based on a leak), then David Hulton and Steve Muller created rainbow tables for it. The fact that it was a custom algo did not defeat the rainbow table's purpose.
There are many ways a custom hash algo designed by a company could end up being used across multiple of its password databases, and could subsequently be leaked, making the generation of rainbow tables possible and valuable (by rogue employees or hackers), that relying on the obscurity of a custom algo is a bad idea. As you recognized, salting is the ultimate protection. So there is no point in talking about alternatives.
But a novel hash function will still do the trick and isn't comparable to using a single salt across the entire database, as you originally claimed.
I'm not sure I understand this point, what is the criteria for a function to be novel?
This function produces different hashes from most other hash functions, eg, it is new and possibly novel, but is equivalent to using a single salt across a database:
function novelHash(password) {
return md5("novelHash" + password);
}
Reverse engineering a novel hash function, while theoretically possible, isn't a capability possessed by the folks dumping passwords from LinkedIn or Riot. And even if it were, doing so would cost more than would be gained by compromising those passwords[2].
Thus, in practical terms, a novel hash function is a great way to undermine attackers, not unlike using Linux or opening files in non-standard viewers (LibreOffice, Nitro PDF Reader, etc.).
1. And if you have access to a good cryptographer, you should absolutely roll a novel hash function as well. Security through obscurity is the proverbial cherry on top of your defensive sundae, i.e. it's only a bad thing if you rely on it exclusively.
2. Not to mention that the whole point of rainbow tables is that they've already been computed. If an attacker has to reverse engineer a hash function to create a rainbow table, that largely defeats the rainbow table's purpose.