1. Background

The HVT package offers a suite of R functions designed to construct topology preserving maps for in-depth analysis of multivariate data. It is particularly well-suited for datasets with numerous records. The package organizes the typical workflow into several key stages:

1. Data Compression: Long datasets are compressed using Hierarchical Vector Quantization (HVQ) to achieve the desired level of data reduction.

2. Data Projection: Compressed cells are projected into one and two dimensions using dimensionality reduction algorithms, producing embeddings that preserve the original topology. This allows for intuitive visualization of complex data structures.

3. Tessellation: Voronoi tessellation partitions the projected space into distinct cells, supporting hierarchical visualizations. Heatmaps and interactive plots facilitate exploration and insights into the underlying data patterns.

4. Scoring: Test dataset is evaluated against previously generated maps, enabling their placement within the existing structure. Sequential application across multiple maps is supported if required.

5. Temporal Analysis and Visualization: Functions in this stage examine time-series data to identify patterns, estimate transition probabilities, and visualize data flow over time.

What’s New?

This notebook showcases the enhancement made to the trainHVT function through the integration of dimensionality reduction techniques and comprehensive evaluation metrics. These advancements aim to enhance the visualization, analysis, and interpretability of high-dimensional data within the HVT framework.

1. Integration of Advanced Dimensionality Reduction Techniques:

The trainHVT function now includes dimensionality reduction techniques like t-SNE and UMAP, alongside the previously implemented Sammon’s method. This integration enhances the function’s capacity to explore and apply various dimensionality reduction approaches.

• t-Distributed Stochastic Neighbor Embedding (t-SNE): Integrating t-SNE into the trainHVT function facilitates non-linear dimensionality reduction, particularly by preserving local structures and visualization of intricate data structures. It efficiently processes large datasets with minimal computational overhead.

• Uniform Manifold Approximation and Projection (UMAP): Integrating UMAP into the trainHVT function used to preserve both local and global data structures. UMAP excels at maintaining local relationships between data points while also preserving the broader global structure, which helps in revealing more meaningful clusters and patterns in complex datasets.

2. Integration of Evaluation Metrics:

Dimensionality reduction evaluation metrics help to determine the quality and effectiveness of the dimensionality reduction process by evaluating aspects such as data point proximity, cluster separation, and overall fidelity of the reduced representation.

• Integrated key metrics such as Trustworthiness, Continuity, RMSE, Silhouette Score, KNN Retention Score, Sammon’s Stress and Likert Scale which address the five major area of dimensionality reduction, which are Structure Preservation Metrics, Distance Preservation Metrics, Interpretive Quality Metrics, Computational Efficiency Metrics, Human Centered Metrics.

2. t-Distributed Stochastic Neighbor Embedding

t-SNE is a widely recognized technique for visualizing high-dimensional data in a low-dimensional space, typically two or three dimensions. Developed by Laurens van der Maaten and Geoffrey Hinton, t-SNE is particularly effective at preserving the local structure of the data, ensuring that similar data points are positioned close to one another in the reduced dimensional space.

Advantages of t-SNE

• The key advantage of using t-SNE for dimensionality reduction lies in its probabilistic approach to measuring the similarities between data points.

• t-SNE focuses on preserving the relative distances between data points, rather than just the absolute distances. This results in visually intuitive maps where similar data points form dense clusters, while dissimilar points are more spread out.

• This property makes the resulting visualizations not only accurate in terms of capturing the underlying data structure, but also highly interpretable, even for users without extensive statistical expertise.

3. Uniform Manifold Approximation and Projection

UMAP is a cutting-edge technique for dimension reduction and data visualization, known for its speed, scalability, and ability to maintain both global and local data structure. Developed by Leland McInnes, John Healy, and James Melville, UMAP has quickly become a favorite among data scientists for its versatility and robust performance across a wide range of applications.

Advantages of UMAP

• The key advantage of using UMAP as a dimensionality reduction technique is its ability to simultaneously preserve the global structure of the data while also highlighting the local relationships between data points.

• This dual focus results in visualizations that accurately represent the underlying clusters and patterns within the dataset, providing insights that may be missed by other dimensionality reduction methods.

• UMAP delivers high-quality, interpretable visualizations that significantly enhance our understanding of complex, multi-dimensional datasets.

4. Dimensionality reduction evaluation metrics

Dimensionality reduction evaluation metrics are measures used to assess the effectiveness of dimensionality reduction techniques. They help evaluate how well these techniques preserve the structure, relationships, and quality of the data when reducing its dimensions.

These six metrics are organized into three main categories. Below is a brief overview of each metric included in the trainHVT function:

Structure Preservation Metrics

1. Trustworthiness:

• Range: 0 to 1
• Description: Trustworthiness measures how well the local structure of the high-dimensional data is preserved in the low-dimensional embedding. A score close to 1 indicates that the nearest neighbors in the high-dimensional space are well represented in the lower-dimensional space.

2. Continuity:

• Range: 0 to 1
• Description: Continuity assesses how well the low-dimensional embedding captures the global structure of the high-dimensional data. Like trustworthiness, a higher score closer to 1 indicates better preservation of the data’s global structure during dimensionality reduction.

3. Sammon’s Stress:

• Range: 0 to 1 (though typically not bounded)
• Description: Sammon’s Stress evaluates how faithfully the distances in the high-dimensional space are represented in the low-dimensional embedding. Lower values indicate a better representation of the original data structure.

Distance Preservation Metrics

4. RMSE(Root Mean Square Error):

• Range: 0 to ∞
• Description: Root Mean Square Error (RMSE) measures the average magnitude of the difference between the distances in the high-dimensional space and those in the low-dimensional space. It reflects the overall error in the distance preservation after dimensionality reduction. A lower RMSE value indicates that the distances in the reduced-dimensional space are closer to those in the original high-dimensional space, implying better preservation of the overall structure.

Human Centered Metrics

5. Likert Scale:

• Range: 1, 2 or 3
• Description: Ideally, the projection would separate clusters or highlight features of the data that a human could easily identify. In this case, it should ideally resemble an annulus. It is a subjective score where
  • 1 indicates a poor projection - significant distortion or overlap; unclear data structure.
  • 2 represents an average projection - adequate representation with minor distortions.
  • 3 signifies a good projection - accurate, distinct, and insightful representation.

Ground Truth: We have performed dimensionality reduction techniques on torus data. The underlying structure of this data is a torus, a surface shaped like a doughnut. The true shape of the data in its original high-dimensional space must resemble an annulus(two concentric circles) when properly reduced to two or three dimensions.

6. Spatial Orientation:

• Range: 0° to 360°
• Description: Spatial orientation describes how an object or point is positioned and directed in space relative to a reference frame. This includes both the object’s location and its rotation or angular orientation. In mathematical or data contexts, spatial orientation is often represented using vectors, matrices, or angles (e.g., Euler angles or quaternions in 3D space).

Interpretive Quality Metrics

7. Silhouette Score:

• Range: -1 to 1
• Description: The Silhouette Score measures the quality of clustering within the low-dimensional space. A higher score closer to 1 suggests that the data points are well clustered, with distinct separation between different clusters.

8. KNN Retention Score:

• Range: 0 to 1
• Description: The K-Nearest Neighbors (KNN) Retention Score measures the proportion of K-nearest neighbors that are preserved from the high-dimensional space to the low-dimensional embedding. A score of 1 indicates perfect retention, meaning all nearest neighbors are preserved.

Computational Efficiency Metrics

9. Execution Duration:

• Range: 0 to ∞
• Description: Execution duration refers to the amount of time it takes for an algorithm to complete its task from start to finish. In the context of dimensionality reduction, it measures how long it takes for methods like t-SNE, UMAP, or Sammon projection to process the data and produce a reduced-dimensionality output. Lower execution duration indicates faster performance, which is particularly important when working with large datasets or in time-sensitive applications.

5. Notebook Requirements

This chunk verifies the installation of all the necessary packages to successfully run this vignette, if not, installs them and attach all the packages in the session environment.

list.of.packages <- c("DT","plotly", "magrittr", "data.table", "tidyverse", "crosstalk",
                      "kableExtra", "cowplot","gdata","tidyverse", "ggplot2", "gridExtra","tibble","HVT")

new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[, "Package"])]
if (length(new.packages)){install.packages(new.packages, repos='https://cloud.r-project.org/')}
invisible(lapply(list.of.packages, library, character.only = TRUE))

6. Data Understanding

First, let us see how to generate data for torus. We are using a library geozoo for this purpose. Geo Zoo (stands for Geometric Zoo) is a compilation of geometric objects ranging from three to 10 dimensions. Geo Zoo contains regular or well-known objects, e.g., cube and sphere, and some abstract objects, e.g., Boy’s surface, Torus and Hyper-Torus.

Here, we will generate a 3D torus (a torus is a surface of revolution generated by revolving a circle in three-dimensional space one full revolution about an axis that is coplanar with the circle) with 12000 points.

The torus dataset includes the following columns:

• x: This column represents the X-coordinate of each point in the torus.
• y: This column represents the Y-coordinate of each point in the torus.
• z: This column represents the Z-coordinate of each point in the torus.

Lets, explore the raw torus dataset containing 12000 points. For the sake of brevity, we are displaying the 20 rows.

set.seed(124)
torus <- geozoo::torus(p = 3,n = 12000)
torus_df <- data.frame(torus$points)
colnames(torus_df) <- c("x","y","z")

torus_df1 <- torus_df %>% round(4)
colnames(torus_df1) <- c("x","y","z")
torus_df1$Row.No <- as.numeric(row.names(torus_df))
torus_df1 <- torus_df1 %>% dplyr::select(x,y,z)
displayTable(torus_df1)

Now, let’s try to visualize the torus dataset in 3D.

plot_ly(x = torus_df1$x, y = torus_df1$y, z = torus_df1$z, type = 'scatter3d',mode = 'markers',
marker = list(color = torus_df1$z,colorscale = c('#F50000', '#000FFF'),showscale = TRUE,size = 3,colorbar = list(title = 'z'))) %>%
layout(scene = list(xaxis = list(title = 'x'),yaxis = list(title = 'y'),zaxis = list(title = 'z'),
aspectratio = list(x = 1, y = 1, z = 0.5)))