Geneformer

Geneformer is a transformer trained on single-cell gene expression that encodes each cell as a ranked list of its genes and learns by masked prediction, capturing gene-gene context without labelled data. The Nature 2023 paper pretrained it on about 30M cells (later 95M) and showed that in silico perturbation, deleting a gene in the model and watching the cell representation move, identified therapeutic targets in cardiomyopathy; the approach has since been applied to tumours. It was the first widely used single-cell foundation model and is valued for transfer learning to tasks with little labelled data. Its rank encoding loses expression magnitude, and benchmarks have found only modest gains over linear baselines on some tasks, so its added value is debated. For a newcomer: Geneformer learned the grammar of genes from millions of cells and can be asked which genes matter in a disease.