For the purposes of this discussion, Norvig has defined a statistical model as:
"a mathematical model which is modified or trained by the input of data points."
He then illustrates how what would be considered a "scientific" model, Newton's law of gravitation, is a statistical model under his definition, but simple one with not many parameters. He contrasts this with a Markov model of a communication channel with a large vocabulary, which has many parameters. His argument is, then, that Chomsky dislikes statistical models with large numbers of parameters, as stated in the passage I quoted before.
My point was that Chomsky's concerns are, with reference to Norvig's argument, equivalent to concerns about model fitting, namely that to fit models with many more parameters than you have data, you require additional assumptions about your model structure. It is difficult (though not necessarily impossible) to learn about the system from your model, because in order to construct your model you have had to assume things about reality that you will not be verifying against observations.
In the opposite case, where you have many more data than parameters, you can fit your model with confidence, given only assumptions about your sampling (which you address by being a good experimentalist). This is what allows you to "learn about the underlying system" - you have a model that describes reality well by itself, without requiring additional assumptions about the nature of reality, so the structure of your model reflects something about the structure of reality, and you can explore your model as though you were exploring reality. Of course, sometimes it turns out the equivalence wasn't as good as we thought, but often it provides us with new directions of investigation.
Hopefully that clarifies the equivalence between the two statements - I apologise for not making it more obvious earlier.
It's more than just overfitting: e.g. if a neural net model with several hundreds or thousands of parameters can be trained on a dataset of billions of observations, and may even be OK at predicting behavior both in and out of sample, but it still remains a black box, since we generally do not know what phenomena particular parameters (or their combinations) represent.
On the other hand, IMO there is nothing wrong or unscientific with having empirically estimated relationships as part of the model -- I just see them as shortcuts whose purpose is to parcel the problem so as to allow other analysis, and as something to potentially investigate further to see why the relationship takes a particular form.
Some ML methods are more amenable to this type of analysis than others though.
To your first point - as you increase the number of parameters in a model you (quickly) begin to suffer from the curse of dimensionality. In a crude sense, the data requirement for a similar level of confidence is exponential in the number of parameters, so it can be difficult to use a high-dimensional model to understand a problem, even with a billion (or trillion...) observations. The best one can hope for is that, if there is a simple relationship hidden in the data, your high-dimensional model captures that relationship in a way that is amenable to extraction - I presume this is what you are saying in your third paragraph.
To your second point - I agree. I too do not reject that there is utility in constructing models that make no effort to match the form of the underlying reality. However, the fact remains that in such cases it is very difficult to use your model to gain deeper understanding, and as such these models simply aren't useful for a lot of science in their current form, precisely because they don't tell you anything about reality. Now if someone were to devise a way of extracting "intelligent", (meaning, sensible given existing understanding) simplified relationships from high-dimensional models, that might be a different matter...
The issue has nothing to do with model fitting or the number of parameters. The key question is whether the model is explanatory (of nature) rather than merely predictive. The case of a Markov model is a red herring because those Markov models are designs of human engineered communication systems rather than models of nature.
What is your definition for "explanatory", in this context?
I agree with your second sentence - the key question is indeed whether the model is explanatory rather than merely predictive. I offered a definition for explanatory as being when "the structure of your model reflects something about the structure of reality, and you can explore your model as though you were exploring reality", which isn't a terrible attempt, from my experience.
At the risk of repeating myself ad nauseam, the relationship to model fitting is found in the presence or absence of additional assumptions required for finding your fit. The difference is between having a very low-dimensional model that fits the data and requires few if any extra assumptions to fit (fitting the model being equivalent to "validating your theory", in this context), or a very high-dimensional model that fits the data (making no claim to "theory"), but by definition requires extra assumptions to get the fit.
In another sub-thread you said:
> If you model a system using the smallest possible mathematical model you don't, from that act alone, understand how the system works.
This is correct inasmuch as the understanding doesn't leap forth immediately, but if the model is a good representation of the data (an important if), then modelling a system in a parsimonious way possible does provide you with understanding, pretty much for free, by looking for systems with a similar structure and learning about their properties. As an example, if you have some random variable, and you realise that it might be modelled with a Poisson distribution, then (assuming you are correct) you immediately gain a lot of understanding, because there is an enormous amount of literature exploring the implications of such a model.
This is what substantiates the link between model fitting and the explanatory vs. predictive question. If you can successfully fit a small model to a problem, without adding assumptions, then that model gives you understanding, by virtue of being a good representation of the data, and having structure. That is simply not the case with the high-dimensional models used in machine learning.
I would be interested to see a counter example - a small model that fits a particular set of data well, but does not provide any explanatory power.
Model fitting is only a useful analogy only in the very abstract sense of "is there something missing in my model". But usually people mean model fitting in the sense of choosing parameters in some restrictive model schema (i.e. selecting from a set of machine learning models), but those typically don't provide any meaningful understanding of the system, unless you apply a much weaker definition of "explanatory" or "understanding" than is typical. From my perspective, when you fit a linear regression I don't think you have any understanding of why the parameters are what they are, in the same way that I don't think a coder who tweaks constants in a buggy program until it works has any understanding why those constants are what they are.
> I would be interested to see a counter example - a small model that fits a particular set of data well, but does not provide any explanatory power.
As I said above, a linear regression typically doesn't lend itself to understanding. In fact, I would be interested in an example of a statistical model that does provide any meaningful explanation. Most ML and AI researchers don't even seem to pursue scientific understanding as a goal; predictive power is their measure of success. That is Chomsky's criticism.
"a mathematical model which is modified or trained by the input of data points."
He then illustrates how what would be considered a "scientific" model, Newton's law of gravitation, is a statistical model under his definition, but simple one with not many parameters. He contrasts this with a Markov model of a communication channel with a large vocabulary, which has many parameters. His argument is, then, that Chomsky dislikes statistical models with large numbers of parameters, as stated in the passage I quoted before.
My point was that Chomsky's concerns are, with reference to Norvig's argument, equivalent to concerns about model fitting, namely that to fit models with many more parameters than you have data, you require additional assumptions about your model structure. It is difficult (though not necessarily impossible) to learn about the system from your model, because in order to construct your model you have had to assume things about reality that you will not be verifying against observations.
In the opposite case, where you have many more data than parameters, you can fit your model with confidence, given only assumptions about your sampling (which you address by being a good experimentalist). This is what allows you to "learn about the underlying system" - you have a model that describes reality well by itself, without requiring additional assumptions about the nature of reality, so the structure of your model reflects something about the structure of reality, and you can explore your model as though you were exploring reality. Of course, sometimes it turns out the equivalence wasn't as good as we thought, but often it provides us with new directions of investigation.
Hopefully that clarifies the equivalence between the two statements - I apologise for not making it more obvious earlier.