Single-cell RNA sequencing has transformed biology by letting scientists examine gene activity one cell at a time. A key challenge in this process is correctly identifying what type of cell each sequenced cell actually is. This step, known as cell type annotation, is essential because all downstream biological insights depend on getting these labels right.

Researchers at the Broad Institute of MIT and Harvard and colleagues tackled a common limitation in current computational approaches. While biologists naturally organize cell types in hierarchies, for example immune cells branching into T cells, B cells, and further subtypes, most machine learning models ignore this structure. Instead, they treat each cell type as if it were completely unrelated to the others.

The researchers introduced a simple but powerful idea called hierarchical cross-entropy loss. Rather than penalizing a model equally for every mistake, this approach takes biological relationships into account. Mislabeling a memory T cell as a naive T cell is treated as less severe than confusing it with a neuron, which better reflects real biology.

Evaluating model generalization in continuously updated single-cell atlases reveals
sharp OOD performance drops for the annotation task

a, A curated subset of the CELLxGENE census (May 2023 release) consisting of 22.2 million human cells annotated with 164 curated cell types, spanning 5,052 donors, 56 tissues and 249 studies. All cells were profiled using 10x Genomics platforms. b, OOD test set comprised of 2.6 million newly added cells from 21 studies in the December 2023 release. These cells span 470 donors and 16 tissues, and are annotated with 80 of the 164 original training cell types. All cells are also profiled using 10x Genomics platforms. c, We train three models (linear classifier, MLP and TabNet) on a donor-partitioned training set comprised of 15.2 million cells from the May 2023 CELLxGENE census. d, ID test set comprised of 3.4 million cells from the May 2023 release of the CELLxGENE census, held out by donor. The linear model, MLP and TabNet achieve 80%, 82% and 84% macro F₁ scores, respectively (mean taken over four independent runs). e, All models exhibit substantial OOD performance drops: macro F₁ scores decrease to 55%, 57% and 52% for the linear model, MLP and TabNet, respectively. The dashed red bars indicate the ID performances for comparison.

What makes this especially compelling is that the method works across many model types, from straightforward linear classifiers to more complex transformer-based systems. Even better, it improves performance on unfamiliar or out-of-distribution data by roughly 12 to 15 percent, without increasing computational cost. This is important because models often struggle when applied to new datasets generated by different labs or technologies.

The researchers also make an important point that goes beyond algorithms. They argue that generating better training data, especially data that fills gaps between related cell types, may matter more than simply building ever more complex models. Improving how cell types connect within reference atlases could lead to annotation tools that generalize more reliably across experiments.

Overall, this work highlights how aligning computational methods more closely with biological knowledge can yield practical gains, and it reinforces the idea that smarter data design and biological context are just as important as machine learning sophistication.