i feel like you're beating around the bush with a definition of reproducible that fits your preferences. and i have no idea what you mean by "artifacts".
here's a quote from the NSF: "reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results…. Reproducibility is a minimum necessary condition for a finding to be believable and informative".
let's say someone has some data and a method to process that data. if they implement it in mathematica, then someone else can do the same implementation in mathematica and attempt to reproduce the results. so what if they had to pay for it? that doesn't affect the reproducibility. it's a barrier maybe, and a small one, but it doesn't make it inherently less reproducible. someone else could take the data and method and implement it in something else, say matlab, sage, octave, or whatever, and attempt to reproduce the results. i know of a researcher who does this very thing. he likes mathematica, but his graduate students use a range of software to do their own work and produce results. he takes their methods and investigates and implements them in mathematica to see if they are reproducible. this is good mathematics and good science. just because mathematica is involved doesn't all of the sudden make everything un-reproducible and bad science.
if you're thinking, but oh, if it's open source, i could crawl through and understand everything the code is doing. but nobody does that. it isn't even feasible in anything but the simplest of simple cases.
> This doesn't mean we should be driving away from reproducibility just because it's lacking in some areas of science.
i didn't say we should. but in all the comments sections of big physics announcements, we don't see people whining about reproducibility. i can't go and reproduce the LIGO experiment in my backyard because it takes huge amounts of money, equipment, and engineering expertise. that's why many things don't compete that well with mathematica. it takes a lot of time, money, and expertise to develop such a software package. the opposing model (open source software) just barely drags along and could disappear at any time as maintainers come and go.
here's a quote from the NSF: "reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results…. Reproducibility is a minimum necessary condition for a finding to be believable and informative".
https://stm.sciencemag.org/content/8/341/341ps12.full
let's say someone has some data and a method to process that data. if they implement it in mathematica, then someone else can do the same implementation in mathematica and attempt to reproduce the results. so what if they had to pay for it? that doesn't affect the reproducibility. it's a barrier maybe, and a small one, but it doesn't make it inherently less reproducible. someone else could take the data and method and implement it in something else, say matlab, sage, octave, or whatever, and attempt to reproduce the results. i know of a researcher who does this very thing. he likes mathematica, but his graduate students use a range of software to do their own work and produce results. he takes their methods and investigates and implements them in mathematica to see if they are reproducible. this is good mathematics and good science. just because mathematica is involved doesn't all of the sudden make everything un-reproducible and bad science.
if you're thinking, but oh, if it's open source, i could crawl through and understand everything the code is doing. but nobody does that. it isn't even feasible in anything but the simplest of simple cases.
> This doesn't mean we should be driving away from reproducibility just because it's lacking in some areas of science.
i didn't say we should. but in all the comments sections of big physics announcements, we don't see people whining about reproducibility. i can't go and reproduce the LIGO experiment in my backyard because it takes huge amounts of money, equipment, and engineering expertise. that's why many things don't compete that well with mathematica. it takes a lot of time, money, and expertise to develop such a software package. the opposing model (open source software) just barely drags along and could disappear at any time as maintainers come and go.