The Reproducibility Crisis Workshop
The two Princeton researchers decided to hold an online workshop. Hosted by the Center for Statistics and Machine Learning at Princeton University, the online workshop aimed to highlight the size and scope of the crisis, identify the root causes of the observed reproducibility failures, and move toward solutions. They expected about 30 participants, but more than 1,500 people signed up, a surprise they say suggests that machine learning problems in science are widespread. During the event, guest speakers cited several examples of where AI is being misused in fields such as medical and social sciences: Michael Roberts, a senior research associate at the University of Cambridge, discussed the problems with dozens of papers claiming to use machine learning to combat COVID-19, particularly when the data was skewed because it had been exposed to different imaging machines. Jessica Hulman, an associate professor at Northwestern University, compared the problems with studies using machine learning to prove the unreplicable phenomenon of major findings in psychology. She said, " In both cases, researchers risk using too little data and misinterpreting the statistical significance of the results." Momin Malik, a data scientist at the Mayo Clinic, talked about his work tracking problematic uses of machine learning in science. He said, " In addition to common mistakes in implementing the technology, researchers sometimes apply machine learning when it's not the right tool for the job." In particular, he cited an example of machine learning that produced misleading results - Google Flu Trends, a tool developed in 2008 to more quickly identify flu outbreaks from logs of search queries typed by Internet users. To him, Google got positive publicity for the project but failed spectacularly to predict the course of the 2013 flu season. An independent study later concluded that the model was based on seasonal conditions that had nothing to do with the spread of the flu. "You can't put everything into one big machine learning model and see what comes out," Momin Malik explained. For some of the workshop participants, not all scientists can master machine learning, especially given the complexity of some of the problems. For example, Amy Winekoff, a data scientist at the Princeton Center for Information Technology Policy, says that while it's important for scientists to learn good software engineering principles, master statistical techniques and take the time to maintain data sets, it shouldn't come at the expense of the field. She stated:"For example, we don't want schizophrenia researchers to know too much about software engineering, but too little about the causes of the disorder."She suggests that greater collaboration between scientists and computer scientists could help strike the right balance. Momin Malik, meanwhile, assures:
"The overall lesson is that it's not right to do everything with machine learning. Despite the rhetoric, the hype, the successes and hopes, it remains a limited approach."For Sayash Kapoor, it's important that the scientific community start thinking about the issue. He states:
"Machine learning-based science is still in its infancy. But it is urgent - it can have very harmful consequences in the long run."Translated from Quelles sont les causes de la crise de reproductibilité dans le domaine de la science basée sur le ML ?
Our articles will then appear first in Google Top Stories.
