One of the founding paradigms of machine learning is that a small number of variables is often sufficient to describe high-dimensional data. The minimum number of variables required is called the intrinsic dimension (ID) of the data. Contrary to common intuition, there are cases where the ID varies within the same data set. This fact has been highlighted in technical discussions, but seldom exploited to analyze large data sets and obtain insight into their structure. Here we develop a robust approach to discriminate regions with different local IDs and segment the points accordingly. Our approach is computationally efficient and can be proficiently used even on large data sets. We find that many real-world data sets contain regions with widely heterogeneous dimensions. These regions host points differing in core properties: folded versus unfolded configurations in a protein molecular dynamics trajectory, active versus non-active regions in brain imaging data, and firms with different financial risk in company balance sheets. A simple topological feature, the local ID, is thus sufficient to achieve an unsupervised segmentation of high-dimensional data, complementary to the one given by clustering algorithms.

Data segmentation based on the local intrinsic dimension / Allegra, M.; Facco, E.; Denti, F.; Laio, A.; Mira, A.. - In: SCIENTIFIC REPORTS. - ISSN 2045-2322. - 10:1(2020), pp. 1-12. [10.1038/s41598-020-72222-0]

Data segmentation based on the local intrinsic dimension

Allegra M.;Facco E.;Denti F.;Laio A.
;
Mira A.
2020-01-01

Abstract

One of the founding paradigms of machine learning is that a small number of variables is often sufficient to describe high-dimensional data. The minimum number of variables required is called the intrinsic dimension (ID) of the data. Contrary to common intuition, there are cases where the ID varies within the same data set. This fact has been highlighted in technical discussions, but seldom exploited to analyze large data sets and obtain insight into their structure. Here we develop a robust approach to discriminate regions with different local IDs and segment the points accordingly. Our approach is computationally efficient and can be proficiently used even on large data sets. We find that many real-world data sets contain regions with widely heterogeneous dimensions. These regions host points differing in core properties: folded versus unfolded configurations in a protein molecular dynamics trajectory, active versus non-active regions in brain imaging data, and firms with different financial risk in company balance sheets. A simple topological feature, the local ID, is thus sufficient to achieve an unsupervised segmentation of high-dimensional data, complementary to the one given by clustering algorithms.
2020
10
1
1
12
16449
Allegra, M.; Facco, E.; Denti, F.; Laio, A.; Mira, A.
File in questo prodotto:
File Dimensione Formato  
s41598-020-72222-0.pdf

accesso aperto

Descrizione: DOAJ Open Access
Tipologia: Versione Editoriale (PDF)
Licenza: Creative commons
Dimensione 2.2 MB
Formato Adobe PDF
2.2 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.11767/116131
Citazioni
  • ???jsp.display-item.citation.pmc??? 6
  • Scopus 16
  • ???jsp.display-item.citation.isi??? 14
social impact