digital · environment · health · family: the model was never trained on this
big ai, small data
AI for scientific discovery fails on small, fragmented, multi-modal datasets typical of most sciences
Problem statement
Modern AI — particularly large language models and generative AI — achieves its successes by training on large-scale, diverse datasets. Many scientific domains cannot produce such data. Scientific processes involve complex systems with numerous interdependent observations across different modalities, scales, and quality levels that serve as proxies for underlying phenomena. The resulting datasets are small, fragmented across disparate sources (simulations, lab experiments, sensor measurements, figures/tables extracted from literature), sometimes proprietary, and lack standardized formats. An NSF workshop convening 31 experts from AI and fields including computational biology, neuroscience, climate science, materials informatics, and physics reported that the scarcity of labeled or high-quality data is a persistent challenge in many scientific domains, and that publicly accessible AI-ready datasets in science remain scarce, although access to such datasets is one of the key enablers for progress in generative AI for science.
Why this matters
The workshop report says foundation models are now being used across nearly every scientific discipline, and it points to initiatives such as the National AI Research Resources program and RAISE (Research in AI for Science and Engineering) as opportunities to integrate AI into scientific discovery and engineering pipelines. But the data are uneven: abundant observational data exist for well-observed systems (the report's examples are well-monitored urban environments and heavily studied ecosystems) while many other systems remain undersampled because of cost, inaccessibility, or technical limitations. The workshop identified rare classes, infrequent extreme events such as extreme precipitation, and conditions outside the training data as cases where real-world and scientific applications need models to work, and where current generative models struggle. The report says addressing these challenges requires generative AI frameworks that not only excel at general-purpose tasks but can also address the specialized requirements of specific scientific domains using limited data.
What’s been tried and why it hasn’t worked
Transfer learning from data-rich to data-scarce systems is one of the approaches the workshop report puts forward (its example is a foundation model trained on hundreds of extensively monitored lakes that can be adapted to data-poor lakes using only a handful of measurements), but the report also notes that generative models may struggle to generalize when tested outside the temporal periods, spatial regions, scales, or operating regimes used during training, and that they are brittle on out-of-distribution inputs, producing incorrect, incoherent, and inconsistent outputs. Physics-guided loss functions and pre-training via synthetic models have demonstrated data and label efficiency and out-of-sample generalization, and knowledge-guided model architectures with compositional structure can reduce trainable components without losing performance; the report presents these as potential solutions, not as solved problems. Current generative models model the typical patterns seen across most samples but struggle with the atypical patterns of rare classes and heavy tails, a problem the report attributes to standard loss functions that are not sensitive to rare classes and to biases in data collection and annotation, made worse by noise and missing values in real-world data. Foundation models for science are being attempted in two ways — adapting pre-trained language and vision models where scientific data is naturally text or images, and training domain-specific foundation models from scratch for graphs, 3D structures, time series, spectral images, and spatio-temporal fields — but the report lists gaps that still hold: hallucination and poor out-of-distribution generalization, rare classes, black-box models that cannot explain their reasoning, poorly calibrated uncertainty, the computational and energy cost of training and multi-step inference, the number of candidates needed in experimental design, and weak symbolic and long-horizon reasoning.
What would unlock progress
The workshop identified several directions: models that serve what the report calls the end-goal of scientific applications, discovering and understanding the causal pathways relating inputs to outputs, rather than predictive accuracy alone — the report names causal modular structures as one form of scientific knowledge to build in, and compositional, knowledge-guided architectures that provide transparency for intermediate processes and reflect their causal dependencies; general uncertainty quantification methods for generative models so scientists know when to trust AI outputs; approaches that integrate domain knowledge (physical equations, causal structures, knowledge graphs) into pre-training, architecture, and adaptation; Bayesian experimental design principles to give generative models sample-efficient decision-making policies, and complementary AI paradigms such as reinforcement learning, planning, and symbolic reasoning; and coordinated efforts to build AI-ready benchmark datasets, common benchmarks, and standardized evaluation metrics and protocols for science. The report also describes building domain-specific foundation models from scratch for data richer than text and images, alongside adapting pre-trained language and vision models where scientific data is naturally text or images.
Entry points for student teams
A team could take a well-studied scientific system (e.g., a simple chemical reaction, ecological population dynamics, or material fatigue behavior) where both data and ground-truth models exist, then systematically benchmark how current AI methods (standard neural networks, PINNs, transfer learning, few-shot learning) degrade as dataset size shrinks and modality diversity increases. The deliverable would be a "data efficiency curve" showing where each method breaks down and what minimum data requirements look like for reliable prediction. This is feasible as a computational/data science project requiring programming skills and basic domain knowledge. A second starting point needs no computing beyond a laptop: pick one scientific field and audit which of its datasets are publicly accessible and AI-ready, classifying each obstacle found against the three reasons the workshop report gives (non-standardized interdependent observations; data scattered across simulations, experiments, sensors, and literature, some of it proprietary; and reluctance to share before publishing), producing a map of where the field's data gap actually sits.
Genes
Structural cousins: stuck the same way, in other fields
Sources
"AI-enabled scientific revolution in the age of generative AI: second NSF workshop report," published in npj Artificial Intelligence (Nature), 2025. Workshop held August 2024, University of Minnesota. (accessed 2026-02-12) go to source ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
- The workshop brought together 31 experts from AI and scientific and engineering fields including computational biology, health sciences, neuroscience, materials informatics, ecology, climate science, hydrology, limnology, and physics, from academia, industry, and federal agencies.
- Relates to existing briefs involving data-constrained predictive models (Cluster 2 in cross-domain analysis) but is more fundamental: those briefs describe models trained on unrepresentative data, while this brief describes the structural inability to create representative scientific training data at all.
- The `failure:ignored-context` tag reflects that AI methods designed for internet-scale data are being applied to scientific settings with fundamentally different data characteristics — analogous to deploying lab-designed solutions in field conditions.
- The causal reasoning gap is a distinct sub-problem: the report contrasts commercial applications, which often prioritize predictive accuracy, with scientific applications, whose end-goal is discovering and understanding the causal pathways relating inputs to outputs. The report says the black-box nature of current AI models limits their utility in developing explainable theories.
- Key adjacent communities: the report calls for sustained trans-disciplinary collaboration between AI researchers and domain scientists, across academia, industry, and government labs, and for support for the science of team science so that researchers think outside their disciplinary silos.
Reconciliation 2026-09-30: The single source was read in full on 2026-09-30 from the publisher page https://www.nature.com/articles/s44387-025-00018-6 (open access; the PDF at the same address and the participant list in Supplementary Appendix A were also fetched and read). A second reader went over the repairs against the same source on the same day; the list below gives the final wording. (1) The text said the workshop convened "30 experts"; the report says 31, and Appendix A lists 31 names; the Problem Statement and Source Notes now say 31. The Source Notes also said "10+ scientific disciplines" and "one of the broadest convenings on AI-for-science challenges"; the report names nine scientific and engineering fields plus AI and makes no claim about breadth relative to other convenings; the note now lists the fields the report names. (2) The phrase "big, labeled data corpora" was in quotation marks as the workshop's words; it does not appear in the report. The report says "the scarcity of labeled or high-quality data" is "a persistent challenge in many scientific domains" and that "publicly accessible AI-ready datasets in science remain scarce"; the sentence now paraphrases those statements and adds the report's point that access to AI-ready datasets is one of the key enablers for progress, and the words "fundamentally incompatible" and "structural bottleneck", which are not in the report, are gone. (3) "Most scientific domains" and "often proprietary" became "many scientific domains" and "sometimes proprietary", matching the report's wording ("many scientific domains", "sometimes also inaccessible due to proprietary restrictions"); "massive, curated datasets" became "large-scale, diverse datasets" for the same reason. (4) The text said federal agencies and universities are "investing billions" on the premise that AI will accelerate discovery across all scientific domains; the report gives no figure and does not state that premise. The sentence now says what the report says: foundation models are being used across nearly every scientific discipline, and the National AI Research Resources program and RAISE are named as opportunities to integrate AI into scientific discovery and engineering pipelines. (5) The text named genomics, astronomy, and materials screening as data-rich fields and ecology, hydrology, and clinical sciences as data-scarce ones, said the "most scientifically important problems" are where data is scarcest, and predicted AI would "miss the most important discoveries while creating an illusion of broad applicability". The report's examples are well-monitored urban environments and heavily studied ecosystems versus systems undersampled because of cost, inaccessibility, or technical limitations; it discusses rare classes, extreme precipitation, and out-of-distribution regimes; and it says addressing these challenges requires frameworks that can meet the specialized requirements of specific scientific domains using limited data. The paragraph now says that. (6) The "What's Been Tried" section said transfer learning "has shown limited success", that physics-informed neural networks "fall apart" for unknown governing equations, and that data augmentation, synthetic data, and federated learning fail for stated reasons. The report presents transfer learning via foundation models as "a powerful mechanism" for sparsely observed systems (its lake example), presents physics-guided losses and knowledge-guided architectures as potential solutions with demonstrated data and label efficiency, and does not mention PINNs, data augmentation, or federated learning at all. The section now describes what the report does say: the out-of-distribution and hallucination problem, the rare-class problem and the reasons the report gives for it, the two routes to foundation models for science, and the report's own list of gaps. (7) In "What Would Unlock Progress", "reason causally rather than just statistically — enabling extraction of mechanistic understanding from small datasets" and "the deeper shift needed is AI architectures designed from the ground up" went beyond the report, which names causal modular structures and compositional architectures reflecting causal dependencies, Bayesian experimental design, reinforcement learning, planning, symbolic reasoning, AI-ready benchmark datasets, and standardized metrics and protocols, and treats from-scratch domain foundation models and adapted pre-trained models as two applicable approaches rather than one as the deeper shift. The paragraph now follows the report's list. (8) The Source Notes bullet on the causal gap now attributes the prediction-versus-causation contrast to the report, which draws it, and its second sentence (that commercial AI improvements do not automatically benefit science, which the report does not say) is replaced with the report's statement that the black-box nature of current models limits their utility in developing explainable theories; the bullet naming NeurIPS, ICML, and SIAM and a missing "shared framework" is replaced with the report's call for trans-disciplinary collaboration and the science of team science, since the report does not discuss those communities (NeurIPS and ICML appear only as venues in its reference list, and SIAM not at all) or a missing shared framework. (9) A second entry point that needs no special facility, a data-availability audit built on the three reasons the report gives for scarce AI-ready datasets, was added so the write-up offers two distinct starting points. Confirmed in the source and left as written: the title, venue, and year on the Source line (npj Artificial Intelligence, volume 1, article 18, 2025; published 11 August 2025); the workshop date (6 August 2024) and place (University of Minnesota); the description of interdependent multi-modal, multi-scale observations serving as proxies; data sourced from simulations, laboratory experiments, sensor measurements, and figures and tables in literature; "large collections of small datasets, where data are fragmented and insufficiently standardized"; general uncertainty quantification as a solution direction; and the benchmark-dataset recommendation. The long title's words "fails" and "typical of most sciences" are stronger than the report, which says "many scientific domains"; the title is left as it was because it carries no number, and is reported for review. Genome tags untouched.