Real-world datasets characterized by discrete features are ubiquitous: from categorical surveys to clinical questionnaires, from unweighted networks to DNA sequences. Nevertheless, the most common unsupervised dimensional reduction methods are designed for continuous spaces, and their use for discrete spaces can lead to errors and biases. In this Letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting, finding a surprisingly small ID, of order 2. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high dimensionality of sequences' space.
Intrinsic Dimension Estimation for Discrete Metrics / Macocco, Iuri; Glielmo, Aldo; Grilli, Jacopo; Laio, Alessandro. - In: PHYSICAL REVIEW LETTERS. - ISSN 0031-9007. - 130:6(2023), pp. 1-6. [10.1103/PhysRevLett.130.067401]
Intrinsic Dimension Estimation for Discrete Metrics
Macocco, Iuri;Glielmo, Aldo;Laio, Alessandro
2023-01-01
Abstract
Real-world datasets characterized by discrete features are ubiquitous: from categorical surveys to clinical questionnaires, from unweighted networks to DNA sequences. Nevertheless, the most common unsupervised dimensional reduction methods are designed for continuous spaces, and their use for discrete spaces can lead to errors and biases. In this Letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting, finding a surprisingly small ID, of order 2. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high dimensionality of sequences' space.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.