I'm personally more worried about flaws in the quantum-safe crypto schemes than in RSA, which is based on the age-old factorization problem of prime numbers.

They can harvest all they want, but if no one ever invents a usable, practical and scalable quantum processor all this will be moot.