1. Background

The HVT package offers a suite of R functions designed to construct topology preserving maps for in-depth analysis of multivariate data. It is particularly well-suited for datasets with numerous records. The package organizes the typical workflow into several key stages:

1. Data Compression: Long datasets are compressed using Hierarchical Vector Quantization (HVQ) to achieve the desired level of data reduction.

2. Data Projection: Compressed cells are projected into one and two dimensions using dimensionality reduction algorithms, producing embeddings that preserve the original topology. This allows for intuitive visualization of complex data structures.

3. Tessellation: Voronoi tessellation partitions the projected space into distinct cells, supporting hierarchical visualizations. Heatmaps and interactive plots facilitate exploration and insights into the underlying data patterns.

4. Scoring: Test dataset is evaluated against previously generated maps, enabling their placement within the existing structure. Sequential application across multiple maps is supported if required.

5. Temporal Analysis and Visualization: Functions in this stage examine time-series data to identify patterns, estimate transition probabilities, and visualize data flow over time.

What’s New? - v24.9.1

• We are introducing a function named clustHVT, designed for hierarchical cluster analysis. This function integrates with the NbClust and stats packages to determine the optimal number of clusters (k). It performs AGNES clustering and generates an interactive 2D cluster plot by clustering cells at level/depth 1.

Flow of clustHVT

• The function is applied to determine the optimal number of clusters for a dataset by evaluating 20 different default indices. It identifies the most appropriate k (optimal number of clusters) by exploring various combinations of cluster counts, distance measures, and clustering methods.

• Based on user input, the function then performs hierarchical clustering using the AGNES (Agglomerative Nesting) technique, specifically employing the default ward.D2 method. The output includes a dendrogram for the specified number of clusters and a 2D cluster HVT plot which is interactive, allowing users to see the context of each cell when hovered over.

• This function is explained in section 8 of this notebook.

Limitation of clustHVT

• This function is only compatible with models trained at a depth of 1; it does not support cases where the hierarchy exceeds depth 1.

Abstract

• This notebook showcases the analysis and evaluation of token embeddings generated by OpenAI using the HVT package. It imports 1536-dimensional embedding data produced by OpenAI’s text-embedding-3-small model from textual content.

• The HVT package is used to compress this embeddings data by constructing a model with trainHVT, enabling the projection of the compressed data in 2D using plotHVT and scoring each data point in the dataset with scoreHVT based on the trained model.

• A key visualization is the compressed 2D Heatmap generated from scoreHVT. This visualization helps to analyze the distribution and placement of cells, visually representing the clustering of textual data based on similarity, with contextually alike data grouped in the same cell.

• This notebook also includes two sections on Hierarchical Clustering. The former section analyze the Sammon’s coordinates data (2D - x and y coordinates) of the centroids and the later section analyze Sammon’s coordinates data (2D - x and y coordinates) with the embeddings (1536D) of the centroids.

• The optimal number of clusters (k) is determined using dendrogram plots from the clustHVT function. The clustered data are then sent to OpenAI’s gpt-4o via API call to generate domain names for each cluster. This domain name allocation process is also done manually.

• The 2D HVT plot is visualized with overlaid domain names of both OpenAI-generated and manually assigned for comparison. Conclusions are drawn regarding the HVT package’s efficiency, the performance of hierarchical clustering method on two different datasets and insights are provided for domain names of both OpenAI generated and manually allotted.

2. Experimental Setup

Objective: To evaluate the effectiveness of the HVT package in compressing and visualizing high-dimensional LLM token embeddings while preserving the underlying structure and relationships between the embeddings.

Hypothesis:

Null Hypothesis (H0):

• The HVT package cannot effectively compress high-dimensional LLM token embedding data and the projected token embeddings do not meaningfully represent contextual similarities and differences between tweets.

Alternate Hypothesis (H1):

• The HVT package can effectively compress high-dimensional LLM token embeddings data and the projected token embeddings meaningfully represent similar tweets converging into the same cells, and cells that are closer together representing similar tweet domains.

Test Metrics:

Compression Ratio:

• A higher compression ratio indicates that the method can significantly reduce the dimensionality of the data while potentially retaining important information. A compression percentage of 80 and above is preferred.

Visual Similarity Evaluation:

• This metric evaluates the HVT package’s ability to visually represent contextual similarities and differences between tweets through the generated heatmap visualization, where compressed and projected token embeddings are displayed as cells with overlaid tweet content.

3. Notebook Requirements

This chunk verifies the installation of all the necessary packages to successfully run this vignette, if not, installs them and attach all the packages in the session environment.

list.of.packages <- c("dplyr","magrittr", "ggplot2", "kableExtra", "httr", "stats", "HVT", 
                       "cluster", "rmarkdown", "NbClust","plotly","dendextend","jsonlite","knitr")

new.packages <-
  list.of.packages[!(list.of.packages %in% installed.packages()[, "Package"])]
if (length(new.packages))
  install.packages(new.packages, dependencies = TRUE, verbose = FALSE, repos='https://cloud.r-project.org/')
invisible(lapply(list.of.packages, library, character.only = TRUE))

plot_columns <- function(dataset, no_of_cols, hvt.results) {
  cols_to_plot <- colnames(dataset)[no_of_cols]
  plots <- lapply(cols_to_plot, function(col_name) {
    print(plotHVT(hvt.results, hmap.cols = col_name, child.level = 1, 
                  plot.type = '2Dheatmap', centroid.size = 0.2))
  })
  invisible(plots)}

create_and_write_clusters <- function(cluster_numbers, cluster_method) {
  cluster_list <- lapply(cluster_numbers, function(i) {
    subset(cluster_method$names.column, cluster_method$clusters == i)
  })
  invisible(list2env(setNames(cluster_list, paste0("clust_", cluster_numbers)), envir = .GlobalEnv))
  write_cluster_to_file <- function(cluster, cluster_name) {
    processed_text <- paste0("'", 
                             gsub("[[:punct:]]", "", paste(cluster, collapse = " ")),
                             "'")
    filename <- file.path("./twitter_data/",paste0(cluster_name, ".txt"))
    writeLines(processed_text, filename)
  }
  cluster_names <- paste0("clust_", cluster_numbers)
  invisible(lapply(cluster_names, function(name) {
    cluster_data <- get(name)
    write_cluster_to_file(cluster_data, name)
  }))}

get_mode <- function(v) {
  uniqv <- unique(v)
  uniqv[which.max(tabulate(match(v, uniqv)))]}


get_cluster_domains <- function(input_dir = "./twitter_data",output_csv) {
  ITERATIONS <- 10
  LITELLM_BASE_URL <- Sys.getenv("LITELLM_BASE_URL")
  LITELLM_API_KEY  <- Sys.getenv("LITELLM_API_KEY")
  url <- paste0(LITELLM_BASE_URL, "/v1/chat/completions")

  cluster_files <- list.files(
    input_dir,
    pattern = "^clust_[0-9]+\\.txt$",
    full.names = TRUE)

  if (length(cluster_files) == 0) {
    stop("No clust_*.txt files found")}

  cluster_ids <- as.integer(gsub("clust_|\\.txt", "", basename(cluster_files)))
  cluster_files <- cluster_files[order(cluster_ids)]
  k <- length(cluster_files)

  content_list <- lapply(cluster_files, function(f) {
    readLines(f, warn = FALSE)})

  prompt <- "You are a deterministic domain classification engine. Read the entire text and assign exactly ONE broad, high-level domain that best represents the dominant theme based on overall meaning, not isolated words. Output must be EXACTLY one English word containing only letters A–Z or a–z. Do NOT include numbers, labels, prefixes, cluster identifiers, punctuation, quotes, formatting, explanations, or multiple words. Do NOT output anything except the domain word. The domain MUST NOT repeat, rephrase, or be semantically similar to any domain in this list: {used_domains}. If a candidate domain overlaps conceptually with any used domain, it is INVALID and must not be selected. Prefer the most general valid domain and prioritize stability over creativity. If multiple domains apply, choose the dominant one. Valid outputs are single words like: Environment, Culture, Politics, Sports, Religion, Technology. Return ONLY the domain word."

  send_request <- function(content, used_domains) {

    full_prompt <- gsub(
      "\\{used_domains\\}",
      paste(used_domains, collapse = ", "),
      prompt)

    request_body <- list(
      model = "gpt-4o",
      temperature = 0,
      messages = list(
        list(role = "user", content = paste(full_prompt, content))))

    response <- POST(
      url,
      add_headers(
        `Authorization` = paste("Bearer", LITELLM_API_KEY),
        `Content-Type`  = "application/json"
      ),
      body = jsonlite::toJSON(request_body, auto_unbox = TRUE))

    if (httr::status_code(response) == 200) {
      httr::content(response, as = "parsed")$choices[[1]]$message$content
    } else {
      stop(paste("LLM error:", httr::status_code(response)))}}

  raw_results <- t(replicate(ITERATIONS, {
    used_domains <- character(0)
    sapply(seq_len(k), function(j) {
      domain <- send_request(content_list[[j]], used_domains)
      used_domains <<- c(used_domains, domain)
      domain
    }) }))

  raw_results <- as.data.frame(raw_results)
  colnames(raw_results) <- paste0("Cluster_", seq_len(k))

  final_domains <- sapply(raw_results, get_mode)

  final_df <- data.frame(
    cluster = paste0("Cluster_", seq_len(k)),
    selected_domain = final_domains,
    stringsAsFactors = FALSE)

  write.csv(final_df, output_csv, row.names = FALSE)
  invisible(final_df)}

4. Dataset Understanding

dataset_raw <- read.csv("./twitter_data/final_embeddings.csv")
dataset_raw <- dataset_raw %>% select(-c(1, 2)) %>% rename(username = source_file)
displayTable(dataset_raw[1:10,])