digital · space
ten million alerts before dawn
Astronomy's next survey will generate 10 million alerts per night and no system can classify them in time
Problem statement
The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will begin scanning the entire visible southern sky every three nights starting in 2025-2026, generating approximately 10 million transient alerts per night — objects that have changed brightness, position, or appeared for the first time. Each alert must be classified (supernova, asteroid, variable star, instrumental artifact, etc.) and distributed to follow-up telescopes within 60 seconds of image acquisition. Current alert broker systems (ANTARES, Fink, ALERCE, Lasair) have been tested on precursor surveys generating ~10,000-100,000 alerts/night and face three unsolved problems at LSST scale: (1) classification accuracy degrades when alert volume exceeds training set diversity, (2) real-time cross-matching against catalogs with billions of objects creates latency that exceeds the 60-second requirement, and (3) the science community lacks infrastructure to filter 10 million alerts/night to the ~1,000 that warrant immediate follow-up for any given science case.
Why this matters
Time-domain astronomy — the study of objects that change on timescales from seconds to years — is among the fastest-growing fields in astrophysics. Kilonovae (neutron star mergers producing heavy elements), gravitational-wave electromagnetic counterparts, tidal disruption events, and interstellar objects all require rapid identification and spectroscopic follow-up within hours to days. A delay of even one night can mean missing the critical early evolution of a supernova or losing a near-Earth asteroid. LSST will discover more transient objects in its first year than all previous surveys combined, but this scientific bounty is only realized if the alert stream can be processed, classified, and prioritized in real time.
What’s been tried and why it hasn’t worked
The Zwicky Transient Facility (ZTF), operating since 2018, generates ~100,000-300,000 alerts/night and serves as the primary testbed for LSST alert infrastructure. ZTF brokers use random forest and neural network classifiers trained on spectroscopically confirmed objects, but the training sets are biased toward bright, nearby, common transient types — they systematically misclassify rare events (the scientifically most interesting ones). Cross-matching ZTF alerts against Gaia, Pan-STARRS, and 2MASS catalogs takes milliseconds per alert, but LSST alerts must be cross-matched against the LSST source catalog itself (~37 billion detections), and the indexing and query infrastructure for sub-second cross-matching at this scale is unproven. Alert filtering (deciding which alerts a given science team should see) currently relies on simple cuts (sky region, brightness range, color), which cannot express the complex, multi-parameter selection criteria needed for scientifically motivated filtering. Community alert filtering services like the LSST Science Platform's alert filtering system are being designed but have not been tested at full volume.
What would unlock progress
Scalable, GPU-accelerated classification pipelines that can process 10 million alerts through multi-class probabilistic classifiers in under 60 seconds. Self-supervised or few-shot learning approaches that can identify anomalous/rare transients without requiring large labeled training sets for every class. Distributed database architectures with sub-millisecond spatial cross-matching against billion-row catalogs. Community-facing alert filtering interfaces that allow scientists to define complex selection functions (combining photometric, astrometric, and contextual features) without requiring database programming expertise.
Entry points for student teams
A student team could build an alert classification pipeline using public ZTF alert streams (available via the ZTF Alert Archive), implementing and benchmarking different ML classifiers (random forest, CNN, transformer) on the task of early-time transient classification (classification from the first 1-3 detections only, when follow-up decisions must be made). Alternatively, a team could develop an anomaly detection system designed to flag alerts that don't fit any known class, using autoencoders or isolation forests on ZTF photometric features. Relevant disciplines: machine learning, database engineering, distributed systems, astronomy.
Genes
Structural cousins: stuck the same way, in other fields
Sources
"Pathways to Discovery in Astronomy and Astrophysics for the 2020s" (Astro2020 Decadal Survey), National Academies of Sciences, Engineering, and Medicine, 2023. accessed 2026-02-16. Also: Vera C. Rubin Observatory LSST Science Book; LSST Alert Distribution white papers; Ivezić et al., ApJ 2019. go to source ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
- The `temporal:window` tag reflects that LSST first light is imminent (2025-2026). Alert infrastructure must be operational before the survey begins; retroactive processing misses the time-critical science.
- The `failure:unrepresentative-data` tag captures the core ML challenge: training sets from smaller surveys are biased toward bright/common transients and cannot represent the diversity of objects LSST will discover.
- The `failure:not-attempted` tag reflects that no system has ever processed 10 million astronomical alerts/night. The scale is qualitatively different from precursor surveys.
- Cross-domain connection: shares the data-scaling structure with digital-hllhc-collision-data-processing (both involve processing pipelines built for current-generation instruments that cannot scale to next-generation data rates) and the unrepresentative-training-data structure with ocean-dl-extreme-event-failure (models that work on common cases but fail on rare/extreme events).
- Public ZTF alert data, ZTF light curves, and Transient Name Server classifications provide excellent real-world datasets for student projects.
- Note reconciled 2026-08-20: a note above argues for `failure:not-attempted`; the genome now carries `failure:unrepresentative-data` after a taxonomy revision. The original note is kept verbatim as the tagging rationale of record.
Reconciliation 2026-09-29: Three of the sources named in the Source line were fetched and read in full on 2026-09-29. The Astro2020 report was read in the National Academies online reader (https://www.nationalacademies.org/read/26141, all 27 sections), which is where https://doi.org/10.17226/26141 leads. Ivezić et al., "LSST: from Science Drivers to Reference Design and Anticipated Data Products," was read in its arXiv copy (https://arxiv.org/abs/0805.2366, version 5). The "LSST Science Book, Version 2.0" was read in its arXiv copy (https://arxiv.org/abs/0912.0201). One change was made. (1) The Source line gave the year of the Astro2020 report as 2021. The copyright page and the suggested citation in the report both say 2023, and so does the Crossref record for the DOI. The Source line now says 2023. The report page mentions a public briefing on November 4, 2021, which may be where the earlier year came from. Confirmed against these sources and left as written: 10 million alerts per night (Astro2020 says the observatory "will generate 10 million time-domain alerts per night"; Ivezić et al. say "several million"); alerts sent out within 60 seconds of an observation (Ivezić et al. and the Science Book); the whole visible sky imaged about every three nights (Science Book); ANTARES as alert-broker software (Astro2020); the Zwicky Transient Facility as the forerunner of the Rubin alert stream, at about one tenth of its rate (Ivezić et al.); kilonovae, tidal disruption events and interstellar objects as targets of this kind of survey. The ZTF public alert archive named in the entry points was fetched at https://ztf.uw.edu/alerts/public/ and is open to anyone; that page also names Lasair, ANTARES and ALeRCE as brokers. NOT checked: the Source line also names "LSST Alert Distribution white papers" without a title or a link, so those documents could not be found and read. The following claims are not in the three sources that were read, may rest on those papers, and are left as written: the 2025-2026 start date (Astro2020 expected science to start in late 2023 or early 2024, and Ivezić et al. expected 2022; both were written before this write-up); the figure of about 37 billion detections (Ivezić et al. give 40 billion objects and 32 trillion observations); that classification, as well as sending, must happen within 60 seconds; the broker Fink; the range of 10,000 to 100,000 alerts per night for earlier tests and 100,000 to 300,000 for ZTF; ZTF operating since 2018; the figure of about 1,000 alerts worth immediate follow-up; that the survey will find more transients in its first year than all earlier surveys combined; the statements about biased training sets, cross-matching delays and simple filter cuts; and the statement in the title that no system can classify the alerts in time (Astro2020 says that sifting the stream "will be a major challenge"). Genes untouched.