digital · humanitarian · infrastructure · family: the model was never trained on this
the satellite can't see the settlements that need it
Satellite-based building footprint extraction fails in informal settlements where it is needed most
Problem statement
Automated building footprint extraction from overhead imagery has large open benchmarks (the SpaceNet building-detection challenges, SpaceNet 1, 2 and 4, cover satellite imagery of Rio de Janeiro, Las Vegas, Paris, Shanghai, Khartoum, and Atlanta), but Persello and Stein call the detection of slums in very high resolution images a difficult task: they describe slums as densely built-up areas with an irregular layout of buildings and almost no vegetation, and find that spectral information alone cannot separate them from other built-up areas. DrivenData reports that, before the Open Cities AI Challenge (2020), relatively low-effort models would have reached Jaccard scores of 0.5915 and 0.6235 on its drone imagery of African cities; the winning model reached 0.8598. These are areas where building maps are needed for disaster risk management and for planning urban upgrading. The sources name some of what makes them hard: the dense, irregular layout of informal settlements; labels that vary in quality (the first-place winner describes the tier 1 training labels as "noisy labels"); and imagery from different locations and sensors whose spatial resolution varies from region to region.
Why this matters
According to UN statistics cited by Persello and Stein, around one-quarter of the world's urban population lives in slums, a proportion that rises to 56% in sub-Saharan Africa, and informal settlements are proliferating in developing countries. Building maps are key to disaster risk management — knowing where assets are vulnerable to damage or disruption by natural hazards — and detecting informal settlements provides information for planning urban upgrading. The Open Cities labels were annotated with the help of local OpenStreetMap communities, and DrivenData describes machine learning as becoming critical to scaling such mapping efforts, which only helps if the models work in the contexts that matter. The Open Cities AI Challenge (DrivenData, 2020) targeted African cities directly, with high-resolution drone imagery from 12 African cities and regions covering more than 700,000 buildings; its winning model reached a 0.8598 Jaccard score (92% precision, 93% recall) on the private test set, against 0.5915 and 0.6235 for two pre-challenge baselines, and DrivenData judged the datasets diverse enough in locations and sensors to make the winning classifier useful for a range of urban mapping projects in Africa. The sources named here report no test of transfer to satellite imagery or to cities outside the challenge set; Persello and Stein, working on a single Quickbird image of Dar es Salaam, name transferability of their model to different cities as future work.
What’s been tried and why it hasn’t worked
Standard semantic segmentation models are what the Open Cities winners built (two of the three segmentation winners used U-Net or U-Net-like models; the third used a feature pyramid network); the SpaceNet building-detection datasets (SpaceNet 1, 2 and 4: satellite imagery of Rio de Janeiro, Las Vegas, Paris, Shanghai, Khartoum, and Atlanta) are the large open benchmarks for the task, and the third-place Open Cities entrant mixed some SpaceNet data into training, but the sources named here do not measure how well models trained on them transfer to informal settlements. What the Open Cities winners report helping was pseudo-labeling the test imagery with a first-stage model and retraining ("This allowed significantly improved generalization of model for test data"), strong image augmentation, oversampling rare tiles, and predicting on large mosaics rather than small chips to avoid errors at borders; the third-place entrant cautioned that there was "no clear evidence" his self-training gains were not simply a stronger backbone. The challenges the sources describe are: (1) informal settlements are densely built-up with an irregular layout of buildings and almost no vegetation, and spectral information alone cannot separate them from other built-up areas (Persello and Stein, on 60 cm Quickbird imagery of Dar es Salaam); (2) labels vary in quality (the competition page says so, and the first-place winner trained on "noisy labels"); and (3) imagery comes from different sensors at different scales (the competition page says spatial resolution varies from region to region, and one winner added a scale-regression head because "the samples in dataset are in different scales").
What would unlock progress
Two directions are promising: (1) foundation models pre-trained on globally diverse building imagery — analogous to how large language models generalize across languages — that learn building-ness independent of roof material, shape, or regional style; (2) active learning systems that efficiently select the most informative images for human labeling in each new target area, minimizing annotation effort while maximizing transfer. The Open Cities challenge itself ran on high-resolution drone imagery; whether its results carry over to satellite imagery, which Persello and Stein used at 60 cm resolution, is a comparison none of the sources named here makes, and Persello and Stein name transferability of their model to different cities as future work.
Entry points for student teams
A team could benchmark existing building detection models on a curated set of informal settlement images from multiple regions and quantify the specific failure modes (false negatives from unusual roofing, false positives from terrain features). Alternatively, a team could develop a minimal annotation pipeline that uses a pre-trained model's uncertainty to prioritize human labeling, testing how few labeled examples are needed to achieve acceptable accuracy in a new city. Skills: computer vision, geospatial data, machine learning, GIS.
Genes
Structural cousins: stuck the same way, in other fields
Sources
DrivenData "Open Cities AI Challenge: Segmenting Buildings for Disaster Resilience" (2020), and the SpaceNet building detection challenge datasets, Persello & Stein, "Deep Fully Convolutional Networks for the Detection of Informal Settlements in VHR Images," IEEE Geoscience and Remote Sensing Letters 14(12):2325–2329, 2017, DrivenData, "Meet the winners of the Open Cities AI Challenge" (11 May 2020), go to source 1 ↗ go to source 2 ↗ go to source 3 ↗ go to source 4 ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
Tier 3 pilot brief sourced from DrivenData/SpaceNet competition post-mortems and associated research. The Open Cities AI Challenge winners' post reports the opposite of a failure to generalize across the challenge's African cities (0.8598 Jaccard; datasets judged diverse enough in locations and sensors); the sources named here report no test of transfer to satellite imagery or to cities outside the challenge set. The temporal:worsening tag reflects the growing number of people in informal settlements combined with climate-related disaster risk increase. Cross-references: health-diabetic-retinopathy-screening-deployment-gap (competition-to-deployment gap pattern), environment-icimod-glacial-lake-outburst-warning-gap (satellite imagery limitations in challenging terrain).
Reconciliation 2026-09-30: Fact repair after reading the whole write-up against every source it names, by a first reader and then a second reader on the same day. Read in full on 2026-09-30: the DrivenData winners' post "Meet the winners of the Open Cities AI Challenge" (https://drivendata.co/blog/open-cities-disaster-winners/, fetched directly); the Open Cities AI Challenge competition page and its problem-description page (https://www.drivendata.org/competitions/60/building-segmentation-disaster-resilience/, fetched directly); the Persello and Stein letter (full text from an Internet Archive copy of the University of Twente repository PDF, web.archive.org/web/20241116195017/ris.utwente.nl/ws/portalfiles/portal/30048876/Persello2017deep.pdf; the repository's own file link returned 403, and the citation record was taken from https://api.crossref.org/works/10.1109/lgrs.2017.2763738); and the SpaceNet dataset pages (https://spacenet.ai/datasets/ and the SpaceNet 1, 2, 4, 6 and 7 pages, fetched directly). (1) The Source line gave the Persello and Stein citation as "...in VHR Satellite Images," IEEE TGRS 55(5), 2017. The Crossref record and the paper itself give the title as "Deep Fully Convolutional Networks for the Detection of Informal Settlements in VHR Images," IEEE Geoscience and Remote Sensing Letters, volume 14, issue 12, pages 2325–2329, December 2017; the Source line now carries that string and the DOI. The Source line also named a "Toolkit for Open Cities post-competition analysis" at the blog address; the page at that address is the winners' post, and it is now cited by its own title and date. Addresses for the competition page and the SpaceNet dataset pages were added for the two sources the line already named. (2) The Problem Statement said footprint extraction reaches ">85% F1" in North American and European cities and "40–60% F1" in informal settlements in Sub-Saharan Africa, South Asia and Latin America. No source gives either figure, or any F1 score, or a formal-versus-informal comparison. The paragraph now gives what the sources do: the cities of the SpaceNet building-detection challenges (SpaceNet 1, 2 and 4; the SpaceNet 6 and 7 datasets also label buildings, in Rotterdam and about 100 locations, so the list is not given as the whole of SpaceNet), Persello and Stein's description of slums (densely built-up, irregular layout of buildings, almost no vegetation, not separable by spectral information alone; detection "a difficult task"), and the Open Cities figures as DrivenData reports them (relatively low-effort models "would have produced" 0.5915 and 0.6235 Jaccard before the challenge, against 0.8598 for the winning model, on drone imagery of African cities). (3) The three stated causes — training-data bias from Western datasets, non-standard roofing, and "dense vegetation, narrow alleys, sheet-metal glare" — are not in any source, and Persello and Stein describe slums as having almost no vegetation. The sentence now names only what the sources name: dense irregular layout; labels that vary in quality (the competition's problem-description page says so, and the first-place winner describes the tier 1 training labels as "noisy labels"); and imagery from different locations and sensors whose spatial resolution, in the competition page's words, varies from region to region. (4) "Over 1 billion people live in informal settlements" is not in any source; the sentence now gives the UN statistics Persello and Stein cite (around one-quarter of the world's urban population, 56% in sub-Saharan Africa, and proliferating). (5) Removed because no source contains them: disease vector mapping and census enumeration as uses; Missing Maps; "thousands of volunteer hours" for a single city; and the "100×" acceleration. The uses now given are the ones the sources give (disaster risk management; planning urban upgrading), and the labeling sentence now says what the competition page says about OpenStreetMap communities and scaling. (6) The text said the Open Cities AI Challenge "found that top competition solutions still struggled with generalization across African cities." The winners' post says the opposite: the top model scored 0.8598 Jaccard (92% precision, 93% recall) on the private test set, a "dramatic improvement" over the baselines, and the datasets had enough diversity in locations and sensors to make the winning classifier useful for a range of urban mapping projects in Africa. The sentence now reports those figures; the untested step is stated as what the sources named here do not report (a test of transfer to satellite imagery or to cities outside the challenge set), with Persello and Stein's own line that transferability to different cities is future work. (7) "SpaceNet datasets (primarily US cities)" was wrong: the SpaceNet building-detection challenges cover Rio de Janeiro, Las Vegas, Paris, Shanghai, Khartoum and Atlanta. "Transfer poorly to informal settlement imagery," fine-tuning on local data, domain adaptation (style transfer, adversarial training) introducing artifacts, and "Mask R-CNN" are not in any source and were replaced by what the winners report helping (pseudo-labeling, strong augmentation, oversampling rare tiles, mosaic prediction), with the third-place entrant's caution quoted; the sentence does not call SpaceNet-trained models the "usual starting point", which no source says. (8) The three "fundamental challenges" — tin, thatched and concrete roofs in Nairobi, Dar es Salaam and Dhaka; buildings obscured by vegetation; "50 cm commercial vs. 30 cm drone" — are not in any source; they were replaced by the challenges the sources describe, with Persello and Stein's 60 cm Quickbird resolution. (9) "Drone imagery ... could provide the fine detail that satellite imagery lacks" was softened: the Open Cities challenge itself ran on drone imagery, and the drone-to-satellite comparison is one none of the sources named here makes. (10) The Source Notes sentence calling the challenge "a well-documented case of competition solutions failing to generalize across deployment contexts" is contradicted by the winners' post and now says so. The quoted phrases now in the text ("noisy labels"; "This allowed significantly improved generalization of model for test data"; "no clear evidence"; "the samples in dataset are in different scales") were each confirmed verbatim in the winners' post by both readers. Confirmed in the sources and left as written: the challenge year (2020) and sponsor (DrivenData); Persello and Stein's Dar es Salaam case; the entry points, which name no dataset and need no special facility. Noted, not changed: the competition's summary pages say 12 African cities and regions while its problem-description page says 10; the write-up follows the summary pages and the winners' post. Not checked: whether the Open Cities imagery and labels remain publicly downloadable (the entry points do not depend on it). Titles unchanged; note for the editor that neither title's claim — that satellite-based extraction fails in informal settlements — is made by any source named here, and that the challenge the write-up rests on used drone, not satellite, imagery. Genome tags untouched.