Psychology is fundamentally not science. I don't mean to devalue it as a discipline, but it is incorrect to call it science for a pretty simple epistemological reason.
Science depends upon the reproducibility of experiments. This means if I perform an experiment twice, then the second experiment (up to relativity) is done in the context of the experiment having already been done once, so the outcome of these experiments can be thought of as a fixed point of performing applying the have-done-the-experiment context (often with respect to all other experiments). Science is made up of these fixed points, which we can think of as the "set" of properties of the capital-W World.
Fundamentally, however, what it means to observe the outcome of an experiment, is to be able to condition your behavior upon that outcome, so if your own behavior is the thing that you are observing, there aren't necessarily any fixed points of this process.
This isn't merely a technicality either. For example: I measure that people experiencing insecurity about their performance are mostly pessimistic, and dub this phenomenon "impostor syndrome". I publish my results and, as a result, people assume that the insecurity they feel about their performance is explained by impostor syndrome, rather than being an honest evaluation of their performance. The next time I try to measure this phenomenon in the population, I might indeed find that it's the other way around, that people are optimistic about their performance, as a result of the previous experiment.
I'm straying a bit off topic, but while agree on some of the premises, I reach a different conclusion.
I agree with you, and probably more broadly that studying systems which aren't fully isolatable is particularly challenging, but I think there is more than a single simple explanation for this. One we need to get out of the way is that there are things under the psychology umbrella that are science and ones that aren't and what is and isn't science isn't simply topic-based, it's approach-based.
For example, reaction-time is a psychological measurement. Is reaction time scientific? I think that's an ill-posed question. It's, like you mentioned, an epistomological question: specifically an ontological question[1]. In the sense that reaction-time is measurable and largely repeatable with respect to specific stimulus, yes, measuring it and analyzing the results can be a step in the scientific path toward knowledge. I'd find it surprising if most people disagreed with this.
Let's take a more difficult example, is behavioral psychology science? Again, an ill-posed question. Can we ask scientific question of behavioral psychology? Sure; Does the intervention of CBT in anxiety-diagnosed subjects (as opposed to a non-intervention control) result in lower post-treatment hospital admittance? That's a valid scientific question. Does it say anything about how CBT works? No. It relates a treatment to a result, and if we want to be more specific, a treatment at a point in time, for a specific group.
Anticipating the test-retest issue you mentioned above, it's being loose with assumptions and isolation. These things can either be controlled for, or assumed. We should be honest with ourselves that far too many times, they haven't been in practice, but let's not conclude from that that psychology has an essential characteristic of not being scientific. It has a social problem instead.
I think we can all be more transparent about assumptions, conditions, and generalization, but I think that's a benefit, not a disadvantage. It enables us to be more precise in our ideas, concepts, and language and that's a good thing.
Specifically regarding the fixed-point framing. It's an assumption of time-invariance. I'm not exactly an expert on the philosophy of science, but I'd be surprised not to find stronger and weaker versions of that assumption and that's a reasonable thing to be transparent about when communicating validity.
1. As an aside, it's funny that "Is reaction time scientific?" is itself not a scientific question.
I don't think there's no value in making observations of human psychology. I was arguing that RCT does not address the epstemological problem, since the subjects observe the outcomes.
I don't think that there are no properties to observe about humans, but simply doing science isn't sufficient. You need to add in some math. Game theory and computer science come to mind as disciplines which are also not science which can be used to reason that an observation might be invariant with respect to repeated observation. Experiment alone is not sufficient to establish this.
Science depends upon the reproducibility of experiments. This means if I perform an experiment twice, then the second experiment (up to relativity) is done in the context of the experiment having already been done once, so the outcome of these experiments can be thought of as a fixed point of performing applying the have-done-the-experiment context (often with respect to all other experiments). Science is made up of these fixed points, which we can think of as the "set" of properties of the capital-W World.
Fundamentally, however, what it means to observe the outcome of an experiment, is to be able to condition your behavior upon that outcome, so if your own behavior is the thing that you are observing, there aren't necessarily any fixed points of this process.
This isn't merely a technicality either. For example: I measure that people experiencing insecurity about their performance are mostly pessimistic, and dub this phenomenon "impostor syndrome". I publish my results and, as a result, people assume that the insecurity they feel about their performance is explained by impostor syndrome, rather than being an honest evaluation of their performance. The next time I try to measure this phenomenon in the population, I might indeed find that it's the other way around, that people are optimistic about their performance, as a result of the previous experiment.