There's a longitudinal problem as well, in that most of the interesting aspects of productivity improvements come over longer time scales, with large projects and people who have already gotten up to speed on them. Those are extremely difficult/expensive to do controlled studies with. Medicine does them, but 5-year studies in medicine are quite expensive, logistically complex, and have a lot of institutional support because it's considered so important to run them. In HCI, user studies are typically of a shorter length, like A/B-testing one UI paradigm versus another in 30-minute user sessions. Applied to PLs, it's feasible to do user studies looking at learning curves, but much less feasible to answer how Haskell compares to C++ on a large project. Instead, like in economics, the best you can do often ends up being to look for "natural experiments" where almost-comparable things happened in different languages, and try to compare them.

A few software-engineering researchers have told me that that's one major reason that recent "tools" type SE-research happens outside academia: if someone in academia had invented git, it's not clear how they would design a user study to evaluate it, especially within the constraints of, say, a PhD thesis timescale/budget. The typical/simple study design is you recruit N participants, randomly assign N/2 to your tool and N/2 to the control tool, have them perform a task, and then try to show with p<0.05 that the group using your tool did better than control. But in this case, the "perform a task" step has to be non-trivial, and it tends not to be feasible to recruit people to participate in a random study that involves them developing serious software over several years, which would be the equivalent of the kinds of randomized studies that are done with medical devices.

I don't actually find the case-study-based approach particularly bad. Start from cases that are awkward or error-prone to handle in a language (either constructed or derived from data about real-world errors), and propose a solution that captures the underlying computation more directly, or in a more checkable way, etc. There are other areas that make progress in that manner; for example, symbolic logic develops with a case-study and counter-example-driven methodology, where someone will propose a case that either can't be represented in Logic X, or at least can't easily be represented, or maybe produces incorrect inferences when encoded in the obvious way, and this will drive development of a Logic X'.