This isn't backpressure, this is forcing synchronicity by using a queue depth of 1. For low throughput applications this is fine, but the latency will kill you on the long run and vastly underutilize available resources.
Backpressure should be proportional to the amount of work you want to prevent. And that backpressure should propagate up the event stream to limit the whole capacity of the subgraph. The slowest chain governs the speed of the whole system so that it can retain functionality instead of falling over. Which relates to other properties of the system
* work stealing, processes handle their own dynamic capacity
* prioritization, we can automatically guage what to NOT do
* feedback
You can think of synchronicity as the most basic form of backpressure. The goal is to avoid flooding the target when source sends at a too high of a rate.
But that was a quick fix. Eventually I implemented something else. Notice that in my case there was a also a bug, so it wasn't just due to normal hardware limits.
One other simplistic hack I found is to force the source to send every Nth message synchronously. Say it sends 10k messages as a cast, then if target is too slow and has queued them up, source sends 10001st as a call So it is forced to wait for target to process them before sending the next. (Sorry reverting to Erlang/OTP terminology here, cast mean send without acknowledgement, call means wait for a response).
Agree, but that synchronous kludge needs to be temporary. I too have used a similar hack, launching waves of tasks/messages and waiting with a timeout, possibly terminating the slow ones and then starting again. The ones that are slow multiple times can be put into a longer, bounded queue.
Backpressure should be proportional to the amount of work you want to prevent. And that backpressure should propagate up the event stream to limit the whole capacity of the subgraph. The slowest chain governs the speed of the whole system so that it can retain functionality instead of falling over. Which relates to other properties of the system