Understanding how individual cells behave is essential for advancing modern biology and medicine. Scientists often turn to single-cell transcriptomics, a technique that measures gene activity one cell at a time, to uncover hidden differences between cells that look similar on the surface. However, many existing computational tools only examine gene expression levels and miss deeper connections among genes.

A new model, called scKGBERT, takes a major step forward. The project was led by researchers from Northeast Forestry University and it combines an enormous amount of data to learn how genes and cells relate to each other in much more meaningful ways. Instead of studying expression patterns alone, scKGBERT integrates 41 million single-cell RNA sequencing profiles with 8.9 million known protein–protein interactions. Protein interactions reveal how gene products work together inside cells, so combining both kinds of information results in a richer, more realistic understanding of cell behavior.

An overview of the scKGBERT framework

a Pre-training of the biological knowledge graph. A TransR-based knowledge graph embedding approach is employed to model protein-protein interactions (PPIs), capturing both structural and semantic relationships among genes. The procedure encodes rich biological context into the initial gene embeddings, providing a foundation for modeling gene regulatory mechanisms and functional associations. b Pre-training phase of scKGBERT. scKGBERT is pre-trained using a masked language modeling (MLM) objective, where single-cell RNA-seq expression profiles are treated as sentences in a biological language—each gene as a word, and expression levels as contextual semantics. Unlike conventional models, scKGBERT introduces a dual-encoder architecture: a sequence encoder and a knowledge encoder, which are integrated via a Gaussian cross-attention mechanism. This mechanism highlights biologically meaningful expression patterns while incorporating regulatory priors into the learned representations. c Fine-tuning phase of scKGBERT. The model is adapted to downstream tasks via supervised fine-tuning, allowing the transfer of generalized knowledge to data-scarce scenarios. We benchmarked scKGBERT across 12 downstream tasks spanning gene-level and cell-level predictions, demonstrating its robust generalizability and biological interpretability across diverse single-cell applications

One of the most interesting features of scKGBERT is its use of Gaussian attention, a method that highlights the most important genes in each cell. This helps researchers find strong biomarkers, which are genes that can signal disease, predict how a cell will respond to a drug, or identify which cell type is present.

The model performs impressively across several tasks, including drug response prediction, disease classification, and gene annotation. Even more importantly, it improves biological interpretability, meaning that its insights are easier for researchers to understand and apply. This could support future discoveries in precision medicine, where treatments are tailored to individual patients based on their unique cellular and genetic profiles.

Overall, scKGBERT represents a powerful new resource that blends massive-scale data with advanced machine learning to help scientists uncover disease mechanisms and identify better therapeutic strategies.

Availability – All relevant data and source codes including the pre-trained model, can be downloaded from https://huggingface.co/catly/scKGBERT