You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much.
To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.
You are absolutely correct that the LLM vs Regulation relationship would be adversarial.
This is why there is the saying: regulations are written in blood.
Mankind has been on a blast improving the understanding and engineering of stuff in our society for a while now. The regulations are already written in an adversarial sense.
Will LLMs take it up a notch? Yeah.
And then the people making the regulations in their respective expert field, will have to deal with it. What they do?! I'm not qualified to say. The whole point is that engineers are the people we deem qualified in their field.
To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.