A problem family — different problems, same issues

the infrastructure that generates bad science

13 problems across 8 fields · newly named in the latest release

These 13 problems share a pattern that is hard to see from inside any one field. The basic inputs to research have documented quality problems: reagents, cell lines, software dependencies, analytical methods, benchmark datasets, experimental protocols, soil testing methods. The incentives discourage anyone from checking them. 15-20% of cell lines are misidentified. More than 50% of commercial antibodies don't work as labeled. Software dependencies break within months. Research built on unreliable inputs produces unreliable conclusions, and those inform policy, clinical practice, and engineering decisions.

The infrastructure that produces knowledge has a quality problem that nobody is rewarded for fixing, and mandates have not fixed it. How many research conclusions rest on inputs that were never validated?

Ways this family shows up

Biomedical reagents. Cell lines, antibodies, and protein assays with documented quality problems that persist
Methods not specified closely enough. Methods described too loosely to give the same result in different labs
Decaying digital infrastructure. Software dependencies and benchmark datasets that degrade over time
Measurement methods not validated. Measurement methods used in conditions they were never validated for

The 13 problems in this family

same soil, three hundred percent aparttested in the wrong worldhigh

the code rots before the paper is citedtested in the wrong worldhigh

the scoreboard is corruptedtested in the wrong worldhigh

new poisons, old test methodstested in the wrong worldhigh

also in it worked in the lab

wrong cells, published anywaybuilt without the people who'd live with itmedium

every lab, a different answerthe rules got in the wayhigh

also in the regulator demands evidence that cannot exist

twelve times too many patientstested in the wrong worldhigh

also in it worked in the lab

half the cancer studies don't repeatbuilt without the people who'd live with itmedium

the experiment worked oncetested in the wrong worldhigh

half the antibodies miss their targettested in the wrong worldmedium

the gold standard is guessingmeasured the wrong thingmedium

simulated decades, real surprisestested in the wrong worldhigh

also in it worked in the lab

identical instructions, incompatible crystalstested in the wrong worldmedium

also in it worked in the lab

Confidence: high = from a primary expert source (Tier 1) · medium = from an analyst, conference, or community source (Tier 2 or 3) · needs sourcing = flagged until we find a better source · the dot's size is its tier; a dashed dot needs sourcing

Loading…

the infrastructure that generates bad sciencethe infrastructure that generates bad science

Ways this family shows up

The 13 problems in this family

the infrastructure that generates bad sciencethe infrastructure that generates bad science

Ways this family shows up

The 13 problems in this family

the infrastructure that generates bad science

the infrastructure that generates bad science