Researchers at the University of North Carolina at Chapel Hill have developed a novel strategy called crafted experiments to improve the evaluation of single-cell RNA sequencing (scRNA-seq) analysis methods. While many computational tools exist for processing scRNA-seq data, comparing their accuracy and effectiveness has been difficult due to the lack of real-world reference datasets.

Crafted experiments work by introducing controlled modifications to real scRNA-seq data, allowing researchers to test and compare different feature selection and clustering methods under standardized conditions. The study specifically assessed GOF, a suite of gene selection methods, and found that GOF reliably identifies important features in both synthetic and real datasets. Additionally, the study determined which variations of GOF perform best in different contexts, providing valuable insights for researchers developing new RNA sequencing tools.

Illustration of crafted experiments

(A) A crafted experiment is based upon a real dataset with known cell labels (Fibroblast, Luminal, and Basal). (B) Gene perturbation strategies. To create a crafted group, the crafted genes can be selected from different gene regions (represented in boxes) based on the relationship between the average gene expression across cells and the proportion of zeros per gene across cells. Each dot is a gene, and the color of the dot represents the two-dimensional kernel density estimate. Sparse region is defined as the proportion of zeros per gene across the crafted cells between 0.9 and 1; Medium region is defined as the proportion of zeros per gene across the crafted cells between 0.45 and 0.55; Dense region is defined as the proportion of zeros per gene across the crafted cells between 0 and 0.1. (C) A deliberate cell group (Crafted luminal) is created by perturbing a set of genes (small box) for a set of cells.

This approach offers a practical way to benchmark analysis methods, ensuring that the most effective tools are used for studying gene expression in diverse biological systems.

Availability – Source code, including all functions for constructing crafted experiments and benchmarking feature selection methods, are publicly available at https://github.com/siyao-liu/CraftedExperiment.