I never said that there isn't need of any further study into LLM's. What I did say is that doing a study in which in which the results are skewed by avoiding using one of the most fundamental best practices for interacting with LLM's, as easily derived from a surface level understanding of one of the most basic principles of LLM's, well that is just irresponsible.

The author clearly had some understanding that context windows could influence their results, but they still decided to release an analysis that does not separate one data gathering technique from the other, allowing them to cherrypick LLM answers from either technique as needed, depending on whether they want to show more or less hallucinations.

It's not that we don't need a study, it's that we don't need bad studies.

From Study: "The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency".

So both ways.

Are you saying they took both methods and intermixed results to skew a narrative? That might be a bit of a leap, but I didn't go find the raw data to disprove that.

It looks like they asked questions within a context window, and also isolated in separate context windows. And compared results.

It seems like this was actually part of the study. How much does the context window skew results, versus if questions were independent? How is that a bad study?

You are saying the study is bad for doing what the study said it was doing. How can using the same context window be bad if studying the context window is what they were looking at. It sounds like you wanted a different study done where the data gathered would be different.

"context windows could influence their results" How much and in what way is useful to study. And, as windows get longer, many common users are just going with one long context and not starting a new window.