#ML#Science#Transparency#Reproducibility: "Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility." https://www.science.org/doi/10.1126/sciadv.adk3452
During meeting no.9, we'll learn how to streamline your analysis using {targets} package, based on chapter 13 of “Building reproducible analytical pipelines with R” by Bruno Rodrigues (https://raps-with-r.dev/targets.html).
One important lesson from the #xz situation is that we should not allow binary blobs to enter the build process because they can't be audited. (In the case of xz-utils, most of the malicious code was hidden in a binary test archive.)
Some time ago, I have banned all binaries from the revision control system used for our papers. That means no PDFs, PNGs, etc. In our case, it's not malicious code but #reproducibility; nevertheless, the challenges are quite similar. (1/3)
April 4th we'll learn about how to package your code, based on chapter 11 of “Building reproducible analytical pipelines with R” by Bruno Rodrigues (https://raps-with-r.dev/packages.html)
Watching with interest: "Backed by 250,000 Swiss francs, or roughly $285,000,… from the University of Bern, [a new program] will pay reviewers to root out mistakes in influential papers, beginning with a handful in #psychology. The more errors found, and the more severe they are, the more the sleuths stand to make." https://www.chronicle.com/article/wanted-scientific-errors-cash-reward
(#paywalled)
I wish more of the #reproducibility discussions in #rstats focussed on ensuring that analysis could be repeated with current/maintained R packages (and system libraries) rather than just taking a snapshot of your current environment. Fixing your environment is just accruing technical/analytical debt so "production" use cases aside (interpret "production" how you wish), I don't think it is as useful for research/science as some make out. There is of course more nuance to this but still ... 1/1
Just learned that I needed to go back a version in #Rstats to keep using the groundhog 📦 and load versions of packages that I was using. Future me better appreciate this... #reproducibility
I love the renv #rstats package for #reproducibility.
But I often receive feedback from frustrated colleagues who get package installation errors.
I have written a blog post to dive more into the issues that can arise when using renv, where they come from (spoiler: it's not renv's fault), and possible solutions.
It allows you to predefine the environment (e.g., specific python or R version) and then boot up an entire terminal with those prepackaged. This ensures you run code with the appropriate versions preloaded!*
*still in the process of testing and caveats will surely follow
I make it part of my job to educate my graduate students about open access, open-source software, open data, practices for #reproducibility, and #OpenScience
SymfonyCon has been a fantastic experience, especially for the opportunity to meet so many incredible people for the first time. Our live discussions have already yielded tangible results. After PHPUnit, GrumPHP is now including a composer.lock for achieving total reproducibility.
This underscores the importance of the social aspect in our work, it's crucial and should never be underestimated.
I am super excited about this mini-conference on #reproducibility in #linguistics that I am organising this evening: Four of my M.A. students will be reporting on their attempts to reproduce the results of four published quantitative linguistics papers for which the data is available, but not the code!
Colleagues, they have a lot of things to report! So, if you're in the area (Cologne), do come along! There will be #ReproducibiliTea and Christmas biscuits! 🍵 🍪 #OpenScience
As promised, here are some of the (anonymised) highlights from my students' attempts to reproduce the results of four published #linguistics studies using the authors' original data, which my students brilliantly presented at our mini-conference on #reproducibility yesterday. #OpenScience
1/ One student perfectly replicated the statistics and plots for RQ1 of the paper she chose, but could not replicate RQ2 because some data for this was missing. She contacted the authors. They never replied.