Researchers at the University of Bern explored a common issue in RNA sequencing (RNA-Seq) research—how well results hold up when studies have only a few samples. RNA-Seq is a powerful tool for studying gene expression, but because it can be expensive and logistically difficult to collect many biological samples, many experiments are performed with small cohort sizes.

The researchers ran over 18,000 simulated RNA-Seq experiments using real gene expression data from 18 different datasets. Their findings highlight a key problem: when researchers analyze small groups of samples, the chances of getting results that can be reliably reproduced in other experiments are quite low. This lack of “replicability” doesn’t always mean the results are inaccurate—but it does make them less trustworthy as a basis for further studies or clinical decisions.

Flowchart of the study design

Top panel: A large RNA-Seq data set obtained from a public repository is subsampled to yield 100 small cohorts of size N, which are analyzed separately and compared pairwise to measure the level of agreement between the results. Additionally, the same analysis steps are run on the entire source data set to define the ground truth, from which precision and recall of results can be computed. This procedure is repeated for 18 different data sets and ten different cohort sizes ranging from 3 to 15 (18’000 cohorts in total). Bottom panel: For a given small cohort, the replicates are resampled with replacement to yield 25 bootstrapped cohorts. The log fold change distributions of the bootstrapped cohorts are compared against the original cohort using Spearman’s rank correlation, which is used as a predictor in a linear regression model to predict the performance metrics from the top panel.

Interestingly, the authors point out that some datasets still achieved good precision—meaning many of their reported findings were actually correct—even if they didn’t catch all the possible differences in gene expression (low recall). This suggests that small studies can sometimes still produce useful data, but with important limitations.

To help researchers evaluate their own RNA-Seq experiments, especially when dealing with small sample sizes, the researchers provide a simple bootstrapping method. This method can give scientists a better sense of how much they can trust their results before investing time or money into follow-up work.

Ultimately, the study offers practical advice for researchers working under sample size constraints and emphasizes the importance of being cautious when interpreting RNA-Seq results from small cohorts.