Most of the Artificial Neural Net simulation research I have seen (say, at venues like NeurIPS) seems to take a very simple conceptual approach to analysis of simulation results - just treat everything as independent observations with fixed effects conditions, when it might be better conceptualised as random effects and repeated measures. Do other people think this? Does anyone have views on whether it would be worthwhile doing more complex analyses and whether the typical publication venues would accept those more complex analyses? Are there any guides to appropriate analyses for simulation results, e.g what to do with the results coming from multi-fold cross-validation (I presume the results are not independent across folds because they share cases).
@jonny@cogsci
Thanks for the confirmation of the observation. I am asking around elsewhere for an introduction/guide/tutorial on statistical analysis of computational simulation studies.
@jonny@cogsci I have just updated that post, clarifying that I am interested in "appropriate statistical methods for evaluating computational research studies" rather than "using simulation studies to evaluate statistical methods".
I would greatly appreciate pointers to any introduction/guide/tutorial on appropriate statistical methods for evaluating computational research studies.
Context: I am starting to do some work in a research field where the researchers are mostly computer scientists and most of the studies are computational experiments. The statistical analysis of the results generated by those studies is nonexistent to naive - often results consist only of a table of means, and you might get standard deviations (not standard errors). I would like to do better from a statistical point of view, but (a) it's a very long time since I had to think about analogous issues in non-computational disciplines, and (b) some statistical issues may be specific to computational experiments, e.g. re-using the same random number stream as input, and non-independence between the folds of a multi-fold validation. Also, I am concerned that trying to introduce some statistical sophistication will attract negative comments from reviewers - so I want to be able to cite something that points out what the statistical problems are and how to deal with them.
Example: Graph Neural Networks are machine learning models that operate on graphs as input. A typical task is to learn to label graphs (classification of graphs), for example, represent chemical molecular structures as graphs and classify them as mutagenic or not. Researchers develop new GNN algorithms and want to compare their performance to other GNN algorithms. There is an archive of graph datasets, which is typically used for this comparison. There are many datasets in the archive, but of course this archive is just a convenience sample of all possible graph datasets (if that even makes sense). Each dataset contains some number (not always large) of labelled graphs (cases). The cases are randomly partitioned into train/test sets, the GNN trained on the train set, then the trained GNN evaluated on the test set. The evaluation metric is usually a single number summary - accuracy (they really love accuracy as the metric). The random train/test partitioning is repeated some small number of times (k-fold validation) to get a distribution of evaluation metrics, and the mean value is reported. This is done independently for each of the GNNs to be compared.
I have not, so far, seen any discussion of: the convenience sample nature of the dataset archive, possible advantages of comparing GNNs on the same train/test partition, issues around accuracy as a metric, or statistical dependence between folds in k-fold validation because of sharing cases. So I am trying find resources identifying statistical issues in that kind of research and potential statistical approaches for analysing the results of the experiments.
@RossGayler
Aha, well yes it entirely depends on the question at hand and the experimental design. So eg. One major distinction is whether you are trying to say something about a model, a family of models, or the data. Parametric statistics is for inference over samples of a definable population, so eg. a point estimate of accuracy on held out data is fine if all youre trying to do is make a claim about a single model since there is no "population" you are sampling from. If youre trying to make a claim about a class of models then now you are sampling from the (usually) real valued, n-dimensional model space, so there the usual requirements for random sampling within parameter space would apply.
Making a claim about the data is much different, because now you have a joint analysis problem of "the effects of my model" and "the effects of the data" (neuroscientists love to treat the SVMs in their "decoding" analyses as neutral and skip that part, making claims about the data by comparing eg. Classification accuracies as if they were only dependent on the data. Even randomly sampling the subspace there doesnt get rid of that problem because different model architectures, training regimes, etc. Have different capacities for classifying different kinds of source data topologies, but I digress.)
For methods questions like this I try and steer clear of domain specific papers and go to the stats lit or even stats textbooks, because domain specific papers are translations of translations, and often have uh motivated reasoning. For example, the technique "representational similarity analysis" in neuro is wholly unfounded on any kind of mathematical or statistical proof or theory, and yet it flourishes because it sounds sorta ok and allows you to basically "choose your own adventure" to produce whatever result you want.
For k-fold, its a traditional repeated measures problem (depending on how you set it up). The benchmarking paradigm re: standard datasets and comparing accuracy is basically fine if the claim you are making is exactly "my model in particular is more accurate on this particular set of benchmarks." Youre right that even for that, to get some kind of aggregated accuracy you would want an MLM with dataset as random effect, but since the difference in datasets is often ill defined and as you say based in convenience im not sure how enlightening it would be.
Would need more information on the specific question you had in mind to recommend lit, and I am not a statistician I just get annoyed with lazy dogshit and think stats and topology (which is relevant bc many neuro problems devolve into estimating metric spaces) is interesting rather than a nuisance.
Add comment