# ====================================================================
# Developer Utility: Module Auto-Reloading
# --------------------------------------------------------------------
# Uncomment the lines below if you are actively modifying the underlying
# pi-metaboqc source code. It ensures that changes in .py files are
# dynamically reloaded without restarting the Jupyter kernel.
# ====================================================================

# %load_ext autoreload
# %autoreload 2

${\pi}$-metaboqc: Interactive Analytical Workflow

Run the cells in order to inspect the data, filter low-quality measurements, correct signal drift, impute missing values, normalize intensities, and compare quality across stages. The notebook follows the same order as pimqc.run_pipeline, but exposes every native object so that its data product, typed audit, diagnostic payload, and exported artifacts can be inspected independently.

In pi-metaboqc 1.5.0, processing stages consume MetaboDataset and return StageResult. result.data carries the next scientific data product, while result.audit records resolved parameters, decisions, metrics, tracking tables, warnings, and plot-ready payloads. These explicit boundaries also make future workflow-plugin and LLM-skill adapters possible without adding framework dependencies to the scientific core.

Step 00: Environment Initialization

Load pi-metaboqc, display the installed package version, initialize logging and progress behavior, locate the packaged demonstration data, validate the TOML configuration, read the metadata and intensity tables, and create the output directory.

Configuration precedence

Settings are resolved in this order: non-None keyword overrides supplied to a stage run method > constructor overrides > validated TOML/JSON configuration > built-in defaults. Unknown fields and invalid values are rejected rather than silently ignored. The preferred imputation options are r_route_method and s_route_method; their historical aliases, mar_method and mnar_method, remain accepted and are retained in normalized configuration dictionaries for compatibility. Supplying output_dir enables the stage's export and rendering phases; omitting it performs the same calculation in memory and returns the same structured result without exported stage files.

The notebook uses the loaded configuration without overriding worker counts or scientific parameters. In the bundled configuration, n_jobs=-1 uses the available CPU workers; edit your configuration if you need fewer workers. Each result variable is retained explicitly so the next action receives the appropriate MetaboDataset, Audit, or tracking table.

Saved execution

The displayed outputs are generated with the bundled configuration, including global_seed=123, shared with the supplementary three-batch case study. Run All in a fresh kernel and save the notebook to retain a complete execution.

import os
# Disable terminal/widget progress; retain loguru messages and warnings.
os.environ["TQDM_DISABLE"] = "1"
from datetime import datetime
from importlib.resources import files
import re

import pandas as pd
from loguru import logger

import pimqc
from pimqc import (
    DataNormalizer,
    FeatureMissingValueFilter,
    FeatureQualityFilter,
    MetaboDatasetBuilder,
    MissingValueImputer,
    QualityAssessor,
    SampleMissingValueFilter,
    SignalCorrector,
)
from pimqc.io import ensure_directory, load_pipeline_config
from pimqc.reporting import (
    NarrativeStatsReporter,
    ReportInput,
    VisualAssetReporter,
)

pimqc.init(check_hardware=False, log_level="INFO", show_progress=False)
logger.info(f"pimqc.__version__: {pimqc.__version__}")
logger.info("Step 00: Environment Initialization...")

# Define standard directories. Convert the package resource root once,
# then use os.path.join consistently for filesystem paths.
DATA_DIR = str(files("pimqc") / "resources" / "demo")
repo_layout = os.path.isdir("examples")
OUTPUT_DIR = os.path.join(
    "examples" if repo_layout else ".", "tutorial_output"
)
ensure_directory(OUTPUT_DIR)

# Load and validate the packaged TOML configuration.
PARAMS_PATH = os.path.join(DATA_DIR, "pipeline_parameters.toml")
params = load_pipeline_config(PARAMS_PATH)

# Load the sample metadata and feature-by-sample intensity matrix.
meta_df = pd.read_csv(os.path.join(DATA_DIR, "project_meta.csv"))
int_df = pd.read_csv(
    os.path.join(DATA_DIR, "project_intensity.csv"), index_col=0
)

# Set True to request the optional final PDF after Markdown generation.
EXPORT_PDF = True

2026-10-08 15:41:10.211 | INFO     | __main__:<module>:30 - pimqc.__version__: 1.5.0
2026-10-08 15:41:10.212 | INFO     | __main__:<module>:31 - Step 00: Environment Initialization...
2026-10-08 15:41:10.216 | SUCCESS  | config:load_pipeline_config:70 - Pipeline configuration loaded and validated.

Step 01: Raw Dataset Construction

Validate feature and sample identifiers, required acquisition metadata, injection order, sample roles, and table alignment. Exact zero intensities are converted to missing values before downstream calculations. The resulting MetaboDataset stores the feature × sample intensity matrix separately from sample metadata, feature metadata, DatasetSchema, and ProcessingContext.

MetaboDatasetBuilder.run_build() returns a StageResult: raw_result.data is the aligned dataset, and raw_result.audit retains dataset composition metrics and a detached plot payload. When an output directory is supplied, the builder also writes the raw matrix and renders the Global Acquisition Overview. The report consumes raw_result.audit.metrics, using the same result boundary as every downstream processing stage.

# Construct the dataset and retain its typed Audit for final reporting.
step1_dir = os.path.join(OUTPUT_DIR, "01_Raw_Data")
dataset_builder = MetaboDatasetBuilder(
    meta_info=meta_df,
    int_df=int_df,
    pipeline_params=params,
)
raw_result = dataset_builder.run_build(output_dir=step1_dir)
raw_dataset = raw_result.data

2026-10-08 15:41:10.250 | INFO     | builder:execute_build:465 - MetaboDataset built: 376 metabolites, 466 samples.
2026-10-08 15:41:10.251 | INFO     | builder:_audit_dataset_health:270 - Executing dataset health audit...
2026-10-08 15:41:10.253 | INFO     | builder:_audit_dataset_health:297 - [Audit] No Outlier Reference Features (ORF) detected. This is normal for untargeted datasets; ORF diagnostics will be skipped.
2026-10-08 15:41:10.335 | INFO     | builder:_export_dataset_table:503 - Raw dataset saved as: tutorial_output\01_Raw_Data\Raw_Data_Intensity.csv

2026-10-08 15:41:11.849 | INFO     | builder:_render_dataset_dashboard:535 - Global acquisition overview plot saved as: tutorial_output\01_Raw_Data\Global_Acquisition_Overview.svg
2026-10-08 15:41:11.850 | SUCCESS  | timing:wrapped:41 - Execution time of "run_build": 1.607 seconds.

QA-Step 01: Quality Assessment of Raw Data

Establish the unprocessed reference using QC-sample correlation, batch-level QC correlation where applicable, QC and study-sample RSD distributions, PCA structure, and multivariate outlier diagnostics. Internal standards and outlier-reference features are included only when they are present and valid in the dataset context.

qa_raw_result.data contains the assessment diagnostics, while qa_raw_result.audit contains serializable metrics, tables, flags, and the AssessmentPlotPayload used for later cross-stage rendering. Retaining the Audit avoids recalculating PCA, distances, or outlier statistics during report compilation.

# Retain the baseline Audit for the final cross-stage QA comparison.
qa_raw_engine = QualityAssessor(raw_dataset, pipeline_params=params)
qa_raw_result = qa_raw_engine.run_assessment(
    output_dir=os.path.join(OUTPUT_DIR, "QA_01_Raw_Data")
)

2026-10-08 15:41:20.294 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_01_Raw_Data\QA_Summary_Dashboard.svg
2026-10-08 15:41:20.295 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:41:20.296 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.433 seconds.

Step 02: Missing-Value Filtering

Run two explicit but dependent filtering actions. First, screen QC and Actual samples by their missing-value proportion while preserving Blank and other non-target sample roles. Second, use detection patterns, QC intensity, and biological-group information to assign features to the reconstruction route (R-route), special-handling route (S-route), or exclusion. These processing routes do not infer statistical missingness mechanisms. The R in R-route means reconstruction, not the R programming language.

The sample and feature actions return independent StageResult and Audit objects even though their CSV tables and combined dashboard share 02_MV_Filtered. mv_filter_result.audit.feature_tracking records retained, rescued, and excluded features. The dataset's imputation_route column drives downstream processing; the legacy missingness_type column remains available for compatibility. Quality filtering receives the tracking table explicitly.

# The two actions share an artifact directory but return separate Audits.
missingness_output_dir = os.path.join(OUTPUT_DIR, "02_MV_Filtered")

sample_mv_engine = SampleMissingValueFilter(
    raw_dataset,
    pipeline_params=params,
)
sample_mv_result = sample_mv_engine.run_filter_samples_by_missingness(
    output_dir=missingness_output_dir
)

feature_mv_engine = FeatureMissingValueFilter(
    sample_mv_result.data,
    pipeline_params=params,
    sample_result=sample_mv_result,
)
mv_filter_result = feature_mv_engine.run_filter_features_by_missingness(
    output_dir=missingness_output_dir
)
mv_filter_dataset = mv_filter_result.data

2026-10-08 15:41:23.200 | INFO     | runner:render:147 - MV classification dashboard saved as: tutorial_output\02_MV_Filtered\MV_Classification_Dashboard.svg
2026-10-08 15:41:23.201 | SUCCESS  | runner:render:148 - Feature missing-value filtering completed.
2026-10-08 15:41:23.201 | SUCCESS  | timing:wrapped:41 - Execution time of "run_filter_features_by_missingness": 2.812 seconds.

QA-Step 02: Quality Assessment after Missing-Value Filtering

Recompute the fixed QA suite for the high-missingness-filtered matrix and compare it with the raw baseline. Changes in QC consistency, RSD distributions, PCA geometry, outliers, and reference-feature behavior should be interpreted together with the preceding sample/feature tracking tables; the QA Audit observes the matrix but does not replace the filtering Audit.

# QA observes the retained matrix; filtering decisions stay in its Audit.
qa_mv_engine = QualityAssessor(mv_filter_dataset, pipeline_params=params)
qa_mv_result = qa_mv_engine.run_assessment(
    output_dir=os.path.join(OUTPUT_DIR, "QA_02_MV_Filtered")
)

2026-10-08 15:41:31.728 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_02_MV_Filtered\QA_Summary_Dashboard.svg
2026-10-08 15:41:31.729 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:41:31.729 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.517 seconds.

Step 03: Signal Drift and Batch-Effect Correction

Correct injection-order drift and batch effects using pooled QCs and the selected correction family. With base_est="Auto", the core combines QC precision (median and feature-wise QC-RSD), D-ratio improvement, and study-sample structure preservation to select a supported candidate. Check the validation audit for the actual evaluation basis: held-out QC evidence where available, descriptive full-model evidence for jointly fitted methods, or unavailable evaluation. The native API documents method-specific OOF requirements and fallback behavior.

correction_result.data is an ordered mapping because some selected methods produce both intra-batch and inter-batch datasets. Only those scientifically propagated stages cross the data boundary; candidate scores and selected predictions are retained once in correction_result.audit.plot_payload. The final mapping value becomes the input to quality filtering.

The saved candidate dashboard shows one final correction-stage result per candidate using the recorded scoring basis; the OOF/Full legend identifies the QC evaluation mode. The two panels use identical candidate positions. This saved figure was rearranged from its existing SVG box summaries (quartiles, median, and whiskers), cross-checked against the saved report medians; correction models and feature statistics were not recomputed. The selected method is marked with an asterisk; both vertical axes are autoscaled to the displayed distributions.

# AUTO evidence and the selected prediction are stored in the Audit.
correction_engine = SignalCorrector(
    mv_filter_dataset,
    pipeline_params=params,
)
correction_result = correction_engine.run_signal_correction(
    output_dir=os.path.join(OUTPUT_DIR, "03_Corrected_Data")
)
corrected_stages = correction_result.data
corrected_dataset = list(corrected_stages.values())[-1]

2026-10-08 15:41:31.746 | INFO     | analysis:transform_correction:1392 - [PYTHON] Blank policy: 24 Blank samples follow each candidate's declared fitting and output policy.
2026-10-08 15:41:31.748 | INFO     | analysis:transform_correction:1458 - [PYTHON] AUTO mode enabled. Evaluating multiple methods.
2026-10-08 15:41:31.910 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: SERRF ---
2026-10-08 15:41:31.910 | INFO     | serrf:fit_transform:384 - [PYTHON] Fitting batch-wise native SERRF with refitted QC folds...
2026-10-08 15:42:03.320 | WARNING  | serrf:fit_transform:549 - [PYTHON] SERRF correction is degraded: 827 sample and 0 QC values have invalid baselines and become missing; 4 unsupported and 2 location-only feature/batch fits; eligible held-out QC coverage is 99.9%.
2026-10-08 15:42:03.660 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] SERRF Eval QC RSD: 11.69%
2026-10-08 15:42:03.815 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: RUV-III ---
2026-10-08 15:42:03.820 | INFO     | analysis:_prepare_ruv_control_features:639 - [PYTHON] RUV-III Control Features: 20 total (5 predefined, 17 empirical).
2026-10-08 15:42:03.821 | INFO     | ruv:fit_transform:65 - [PYTHON] Executing RUV-III (k=5)...
2026-10-08 15:42:03.828 | WARNING  | ruv:fit_transform:93 - [PYTHON] RUV temporarily median-filled 3748 missing fit cells in log space; original missing positions will be restored.
2026-10-08 15:42:04.356 | WARNING  | ruv:fit_transform:197 - [PYTHON] Native RUV returned invalid raw intensities; the correction stage maps these to missing. Counts remain in diagnostics.
2026-10-08 15:42:04.533 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] RUV-III Eval QC RSD: 11.09%
2026-10-08 15:42:04.687 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: WaveICA 2.0 ---
2026-10-08 15:42:04.687 | INFO     | waveica:fit_transform:484 - [PYTHON] Executing WaveICA 2.0 correction...
2026-10-08 15:42:04.711 | WARNING  | waveica:fit_transform:529 - [PYTHON] WaveICA temporarily median-filled 3748 missing cells; original missing positions will be restored.
2026-10-08 15:42:05.112 | INFO     | waveica:fit_transform:555 - [PYTHON] WaveICA 2.0 Blank policy: 24 Blank samples received frozen artifact predictions (4172 proxy fallback cells; 204 source boundary fallbacks).
2026-10-08 15:42:05.350 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] WaveICA 2.0 Eval QC RSD: 18.55%
2026-10-08 15:42:05.514 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: QC-RLSC ---
2026-10-08 15:42:05.515 | INFO     | regression:fit_transform:124 - [PYTHON] Phase 1: Executing Intra-batch drift correction with QC-RLSC...
2026-10-08 15:42:11.020 | INFO     | regression:fit_transform:333 - [PYTHON] Phase 2: Executing Inter-batch median alignment...
2026-10-08 15:42:11.457 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] QC-RLSC Eval QC RSD: 13.58%
2026-10-08 15:42:11.615 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: robust QC-RLSC ---
2026-10-08 15:42:37.087 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] robust QC-RLSC Eval QC RSD: 12.87%
2026-10-08 15:42:37.232 | INFO     | analysis:_evaluate_correction_candidates:702 - [PYTHON] --- Evaluating Method: QC-SVR ---
2026-10-08 15:42:37.233 | INFO     | regression:fit_transform:124 - [PYTHON] Phase 1: Executing Intra-batch drift correction with QC-SVR...
2026-10-08 15:42:40.242 | INFO     | regression:fit_transform:333 - [PYTHON] Phase 2: Executing Inter-batch median alignment...
2026-10-08 15:42:40.527 | INFO     | analysis:_evaluate_correction_candidates:1288 - [PYTHON] QC-SVR Eval QC RSD: 14.32%
2026-10-08 15:42:40.529 | SUCCESS  | analysis:transform_correction:1526 - [PYTHON] Auto selection: RUV-III is optimal (score = 0.625, Eval QC RSD = 11.09%).
2026-10-08 15:42:41.100 | INFO     | runner:render:76 - [PYTHON] Assembling correction diagnostic dashboard...

2026-10-08 15:42:55.961 | INFO     | runner:render:108 - [PYTHON] Correction candidate dashboard saved as: tutorial_output\03_Corrected_Data\Correction_Candidate_Dashboard_RUV-III.svg
2026-10-08 15:42:55.966 | INFO     | runner:_render_internal_standards:147 - [PYTHON] Generating IS plots for RUV-III...
2026-10-08 15:42:59.025 | INFO     | runner:_render_internal_standards:168 - [PYTHON] Bypassing IS baseline prediction for RUV-III.
2026-10-08 15:42:59.026 | SUCCESS  | runner:render:112 - [PYTHON] Signal drift correction (RUV-III) completed.
2026-10-08 15:42:59.027 | SUCCESS  | timing:wrapped:41 - [PYTHON] Execution time of "run_signal_correction": 87.281 seconds.

QA-Step 03: Quality Assessment of Corrected Stages

Assess every correction stage selected for propagation, rather than every candidate evaluated internally. For a multi-batch QC-anchored run this may include intra-batch and inter-batch corrected datasets; a global method may return one final stage. Each Audit is retained under its stage name so the final report can compare the same RSD, PCA, correlation, and outlier diagnostic across stages.

# Some methods propagate more than one scientifically meaningful stage.
qa_correction_results = {}
for stage_name, stage_dataset in corrected_stages.items():
    safe_name = re.sub(r"[^A-Za-z0-9_-]+", "_", stage_name).strip("_")
    assessor = QualityAssessor(stage_dataset, pipeline_params=params)
    qa_correction_results[stage_name] = assessor.run_assessment(
        output_dir=os.path.join(
            OUTPUT_DIR, "QA_03_Corrected_Data", safe_name
        )
    )

2026-10-08 15:43:07.068 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_03_Corrected_Data\RUV-III\QA_Summary_Dashboard.svg
2026-10-08 15:43:07.069 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:43:07.069 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.032 seconds.

Step 04: Low-Quality Feature Filtering

Remove background-dominated features whose mean Blank/QC ratio exceeds the configured tolerance, then apply pooled-QC RSD filtering to R-route features. S-route features retained through group/QC rescue remain exempt from this RSD check under the existing retention policy; that exemption does not establish biological sparsity or a particular missingness mechanism.

The complete Step 02 tracking table is supplied explicitly so QualityFilterAuditPayload can preserve the original feature denominator, cumulative retention counts, and final removal reasons. The filtered dataset alone does not contain enough information to reconstruct that history.

# Supply the complete upstream tracking table to preserve provenance.
quality_filter_engine = FeatureQualityFilter(
    corrected_dataset,
    pipeline_params=params,
    missingness_metadata=mv_filter_result.audit.feature_tracking,
)
quality_filter_result = quality_filter_engine.run_filter_features_by_quality(
    output_dir=os.path.join(OUTPUT_DIR, "04_Quality_Filtered")
)
quality_filtered_dataset = quality_filter_result.data

2026-10-08 15:43:07.082 | INFO     | analysis:filter_features_by_quality:965 - Features before filtering: 347
2026-10-08 15:43:07.084 | INFO     | analysis:filter_features_by_quality:983 - Features after Blank/QC check: 203
2026-10-08 15:43:07.086 | INFO     | analysis:filter_features_by_quality:1010 - Features after QC RSD check: 165
2026-10-08 15:43:07.135 | INFO     | runner:export:211 - Data after low-quality features filtering saved as: tutorial_output\04_Quality_Filtered\Filtered_Data_Low-quality_Features.csv

2026-10-08 15:43:13.581 | INFO     | runner:render:230 - Low-quality Filter summary dashboard saved as: tutorial_output\04_Quality_Filtered\Low-quality_Filtering_Dashboard.svg
2026-10-08 15:43:13.581 | SUCCESS  | runner:render:238 - Low-quality features filtering completed.
2026-10-08 15:43:13.582 | SUCCESS  | timing:wrapped:41 - Execution time of "run_filter_features_by_quality": 6.501 seconds.

QA-Step 04: Quality Assessment after Low-Quality Filtering

Check whether Blank/QC and pooled-QC-RSD removal improved analytical consistency without distorting sample-level PCA geometry or unexpectedly changing outlier and reference-feature behavior. Feature decisions remain in the quality-filter Audit; this QA checkpoint records only the condition of the retained matrix.

# Keep processing decisions separate from observational QA diagnostics.
qa_quality_filter_engine = QualityAssessor(
    quality_filtered_dataset,
    pipeline_params=params,
)
qa_quality_filter_result = qa_quality_filter_engine.run_assessment(
    output_dir=os.path.join(OUTPUT_DIR, "QA_04_Quality_Filtered")
)

2026-10-08 15:43:21.874 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_04_Quality_Filtered\QA_Summary_Dashboard.svg
2026-10-08 15:43:21.875 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:43:21.876 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.282 seconds.

Step 05: Missing-Value Imputation

Use imputation_route to apply separate processing routes to non-Blank samples. S-route targets use QRILC or the configured minimum-fraction strategy (s_route_method); R-route targets use a fixed reconstruction method or metric-guided selection (r_route_method). Automatic selection compares KNN, MinProb, QRILC, Median, LLS, and BPCA using masked reconstruction, distribution fidelity, and sample-structure preservation. The GMM-based, low-intensity-weighted mask is a reconstruction stress test, not evidence for a MAR/MNAR mechanism. Correction-induced missing cells enter reconstruction separately from original S-route targets.

Blank columns are preserved without imputation. If target samples contain no missing values, the stage returns a typed skipped Audit and propagates the matrix unchanged. The final-data structure diagnostics are separate from the masked candidate benchmark stored in the imputation Audit.

# The stage may return a typed skipped Audit when no values need filling.
# Preferred optional overrides: r_route_method="Auto", s_route_method="QRILC".
# Here both routes follow the loaded configuration without overrides.
imputation_engine = MissingValueImputer(
    quality_filtered_dataset,
    pipeline_params=params,
)
imputation_result = imputation_engine.run_imputation(
    output_dir=os.path.join(OUTPUT_DIR, "05_Imputation"),
)
imputed_dataset = imputation_result.data

2026-10-08 15:43:21.892 | INFO     | analysis:transform_imputation:1498 - [PYTHON] Hybrid Imputation Engine Initialized. R-route: Auto (Evaluating KNN=5, LLS (K=15), BPCA (PCs=2), MinProb, Median) | S-route: QRILC | Sim_Mask: 0.05
2026-10-08 15:43:21.902 | INFO     | analysis:transform_imputation:1524 - [PYTHON] Applying QRILC to 3 S-route features.
2026-10-08 15:43:22.983 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "KNN" on the R-route subset...
2026-10-08 15:43:23.200 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "MinProb" on the R-route subset...
2026-10-08 15:43:23.511 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "QRILC" on the R-route subset...
2026-10-08 15:43:23.847 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "Median" on the R-route subset...
2026-10-08 15:43:24.052 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "LLS" on the R-route subset...
2026-10-08 15:43:24.238 | INFO     | analysis:_select_best_imputation_method:1109 - [PYTHON] Simulating "BPCA" on the R-route subset...
2026-10-08 15:43:24.925 | INFO     | analysis:_select_best_imputation_method:1190 - [PYTHON] R-route algorithm selected: LLS (score=0.902)
2026-10-08 15:43:24.926 | INFO     | analysis:transform_imputation:1612 - [PYTHON] Executing isolated 'LLS' on R-route features.
2026-10-08 15:43:25.009 | INFO     | analysis:transform_imputation:1762 - [PYTHON] Calculating imputation-related metrics...
2026-10-08 15:43:25.253 | INFO     | runner:render:42 - [PYTHON] Generating diagnostic plots for imputation...

2026-10-08 15:43:39.263 | INFO     | runner:render:85 - [PYTHON] Imputation dashboard saved as: tutorial_output\05_Imputation\Imputation_Dashboard_LLS.svg

2026-10-08 15:43:42.186 | INFO     | runner:render:104 - [PYTHON] Imputer candidate NRMSE grid saved as: tutorial_output\05_Imputation\Imputation_Candidate_Dashboard_LLS.svg
2026-10-08 15:43:42.187 | SUCCESS  | analysis:run_imputation:1837 - [PYTHON] Missing value imputation completed successfully.
2026-10-08 15:43:42.187 | SUCCESS  | timing:wrapped:41 - [PYTHON] Execution time of "run_imputation": 20.299 seconds.

QA-Step 05: Quality Assessment of Imputed Data

Re-evaluate QC consistency, RSD, PCA geometry, multivariate outliers, and reference-feature behavior after missing values are completed. The ImputationAuditPayload remains the authoritative record of the selected method and masked reconstruction evidence; the assessment Audit supports comparison with other pipeline stages.

# Assess the final imputed matrix, not the masked candidate benchmark.
qa_imputation_engine = QualityAssessor(
    imputed_dataset,
    pipeline_params=params,
)
qa_imputation_result = qa_imputation_engine.run_assessment(
    output_dir=os.path.join(OUTPUT_DIR, "QA_05_Imputed_Data")
)

2026-10-08 15:43:50.621 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_05_Imputed_Data\QA_Summary_Dashboard.svg
2026-10-08 15:43:50.622 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:43:50.623 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.423 seconds.

Step 06: Data Normalization

Normalize the imputed intensities using robust-log-only, TIC, Median, PQN, MDFC, Quantile, VSN, or the AUTO strategy configured by norm_method="Auto". AUTO compares QC relative-log-expression alignment, variance stabilization, QC structure, and study-sample structure preservation before selecting a conservative candidate.

Blank samples are permanently removed from the normalized output. NormalizationAuditPayload stores the selected-method passport, candidate summaries, before/after QC diagnostics, and final sample-structure diagnostics. The returned MetaboDataset.context explicitly records the normalization stage and whether the matrix is already log-like.

# The output context records the selected transformation and log state.
normalization_engine = DataNormalizer(
    imputed_dataset,
    pipeline_params=params,
)
normalization_result = normalization_engine.run_normalization(
    output_dir=os.path.join(OUTPUT_DIR, "06_Normalized_Data"),
)
normalized_dataset = normalization_result.data

2026-10-08 15:43:50.631 | INFO     | runner:compute:35 - [PYTHON] Permanently dropping 24 Blank samples.
2026-10-08 15:43:50.632 | INFO     | runner:compute:36 - [PYTHON] Applying Normalization | Method: Auto
2026-10-08 15:43:50.635 | INFO     | analysis:_select_auto_normalization:1270 - [PYTHON] Auto normalization evaluates fixed strategies on a common log-like view using QC RLE, QC variance stabilization, QC structure, and sample structure criteria.
2026-10-08 15:43:50.635 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: ROBUST_LOG_ONLY (log_transform=True).
2026-10-08 15:43:50.800 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: TIC (log_transform=True).
2026-10-08 15:43:50.962 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: MEDIAN (log_transform=True).
2026-10-08 15:43:51.120 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: PQN (log_transform=True).
2026-10-08 15:43:51.279 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: MDFC (log_transform=True).
2026-10-08 15:43:51.281 | INFO     | analysis:calc_mdfc_normalization:1074 - [PYTHON] Executing chunked parallel MDFC (backend='loky', cores=8, chunks=32)...
2026-10-08 15:43:51.618 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: QUANTILE (log_transform=True).
2026-10-08 15:43:51.810 | INFO     | analysis:_select_auto_normalization:1283 - [PYTHON] Evaluating Auto normalization candidate: VSN (log_transform=False).
2026-10-08 15:43:52.215 | INFO     | analysis:_select_auto_normalization:1363 - [PYTHON] Auto normalization candidate scores: ROBUST_LOG_ONLY=0.500, TIC=0.596, MEDIAN=0.653, PQN=0.657, MDFC=0.567, QUANTILE=0.349, VSN=0.412
2026-10-08 15:43:52.216 | INFO     | analysis:_select_auto_normalization:1373 - [PYTHON] Auto normalization selected PQN (score=0.657, margin=0.005).
2026-10-08 15:43:52.437 | INFO     | runner:export:157 - [PYTHON] Auto normalization summary saved as: tutorial_output\06_Normalized_Data\Normalization_Auto_Summary.csv
2026-10-08 15:43:52.437 | INFO     | runner:render:161 - [PYTHON] Generating diagnostic plots for normalization...

2026-10-08 15:43:56.579 | INFO     | runner:render:171 - [PYTHON] Normalization summary dashboard saved as: tutorial_output\06_Normalized_Data\Normalization_Dashboard_PQN_Log2.svg
2026-10-08 15:43:56.580 | SUCCESS  | analysis:run_normalization:1544 - [PYTHON] Data normalization completed successfully.
2026-10-08 15:43:56.580 | SUCCESS  | timing:wrapped:41 - [PYTHON] Execution time of "run_normalization": 5.949 seconds.

QA-Step 06: Quality Assessment of Normalized Data

Create the final QA checkpoint using the normalized dataset's explicit ProcessingContext. Scale-sensitive QA therefore knows whether the matrix is already log-like and avoids applying an unintended second transformation. Interpret improvements in RSD and QC/batch correlation together with PCA geometry, outlier status, and the normalization Audit's preservation diagnostics.

# QualityAssessor reads the explicit normalized-data context.
qa_normalization_engine = QualityAssessor(
    normalized_dataset,
    pipeline_params=params,
)
qa_normalization_result = qa_normalization_engine.run_assessment(
    output_dir=os.path.join(OUTPUT_DIR, "QA_06_Normalized_Data")
)

2026-10-08 15:44:04.998 | INFO     | rendering:render_assessment:259 - Assessor summary dashboard saved as: tutorial_output\QA_06_Normalized_Data\QA_Summary_Dashboard.svg
2026-10-08 15:44:04.999 | SUCCESS  | rendering:render_assessment:260 - Data quality assessment completed.
2026-10-08 15:44:05.000 | SUCCESS  | timing:wrapped:41 - Execution time of "run_assessment": 8.410 seconds.

Step 07: Sequential Audit Report Compilation

Collect processing metrics, correction-stage metrics, and saved assessment Audits in the same scientific order used above. The visual reporter draws four cross-stage patchworklib dashboards—RSD, PCA, correlation, and outliers—directly from Audit payloads. Each dashboard uses one shared legend brick, consistent batch styles, and the complete applicable IS/ORF legend without reading or stitching intermediate SVG panels.

ReportInput records the package version, acquisition mode, batch design, validated configuration, metrics, and asset manifest used for the narrative. The reporter writes comprehensive and brief Markdown reports plus a JSON snapshot. With EXPORT_PDF = True, export tries WeasyPrint, then XeLaTeX, then HTML. Pandoc is required for conversion; inspect the log and generated file to distinguish PDF output from successful HTML fallback.

REPORT_DIR = "07_Report_Summary"

stage_results = {
    "raw_dataset": raw_result,
    "sample_missing_value_filtering": sample_mv_result,
    "high_mv_feature_filtering": mv_filter_result,
    "signal_correction": correction_result,
    "low_quality_feature_filtering": quality_filter_result,
    "missing_value_imputation": imputation_result,
    "normalization": normalization_result,
}
pipeline_metrics = {
    name: result.audit.metrics for name, result in stage_results.items()
}

qa_metrics = {
    "raw_dataset": qa_raw_result.audit.metrics,
    "high_mv_feature_filtering": qa_mv_result.audit.metrics,
    "low_quality_feature_filtering": qa_quality_filter_result.audit.metrics,
    "missing_value_imputation": qa_imputation_result.audit.metrics,
    "normalization": qa_normalization_result.audit.metrics,
}
correction_metric_keys = {
    "Intra-batch corrected": "intra_batch_correction",
    "Inter-batch corrected": "inter_batch_correction",
    "SERRF": "global_correction",
    "RUV-III": "global_correction",
    "WaveICA 2.0": "global_correction",
}
for stage_name, result in qa_correction_results.items():
    key = correction_metric_keys.get(stage_name, "unknown_correction")
    qa_metrics[key] = result.audit.metrics

# Keep QA checkpoints in processing order, including each correction stage.
qa_audits = {
    "Raw data": qa_raw_result.audit,
    "Missing-value filtering": qa_mv_result.audit,
    **{name: result.audit for name, result in qa_correction_results.items()},
    "Low-quality filtering": qa_quality_filter_result.audit,
    "Imputation": qa_imputation_result.audit,
    "Normalization": qa_normalization_result.audit,
}

asset_manifest = VisualAssetReporter(OUTPUT_DIR).compile_assessor_report(
    audits=qa_audits,
    is_multi_batch=raw_dataset.is_multi_batch,
    report_folder=REPORT_DIR,
)
report_input = ReportInput(
    pipeline_metrics=pipeline_metrics,
    qa_metrics=qa_metrics,
    metadata={
        "date": datetime.now().strftime("%Y-%m-%d %H:%M"),
        "version": pimqc.__version__,
        "mode": raw_dataset.context.acquisition_mode,
        "is_multi_batch": raw_dataset.is_multi_batch,
    },
    resolved_config=params,
    asset_manifest=asset_manifest,
)
reporter = NarrativeStatsReporter(OUTPUT_DIR)
reporter.generate_markdown(report_input, report_folder=REPORT_DIR)
if EXPORT_PDF:
    report_generated = reporter.export_report(pdf_engine="weasyprint")
    if not report_generated:
        raise RuntimeError(
            "Processing completed, but report export failed. "
            "Check the converter logs and installed dependencies."
        )

2026-10-08 15:44:15.546 | INFO     | utils:generate_markdown:1218 - Generating COMPREHENSIVE narrative report...
2026-10-08 15:44:15.604 | SUCCESS  | utils:generate_markdown:1228 - Comprehensive report generated: tutorial_output\07_Report_Summary\Report_Comprehensive.md
2026-10-08 15:44:15.605 | INFO     | utils:generate_markdown:1218 - Generating BRIEF narrative report...
2026-10-08 15:44:15.621 | SUCCESS  | utils:generate_markdown:1228 - Brief report generated: tutorial_output\07_Report_Summary\Report_Brief.md
2026-10-08 15:44:17.589 | INFO     | utils:export_report:1401 - --- Exporting PDF for: Report_Comprehensive.md ---
2026-10-08 15:44:17.590 | INFO     | utils:_render_weasyprint:1318 - Attempting PDF export via WeasyPrint...
2026-10-08 15:44:24.913 | SUCCESS  | utils:_render_weasyprint:1344 - PDF generated: tutorial_output\07_Report_Summary\Report_Comprehensive.pdf
2026-10-08 15:44:24.914 | SUCCESS  | utils:export_report:1429 - [Report_Comprehensive.md] completed using WeasyPrint.
2026-10-08 15:44:24.915 | INFO     | utils:export_report:1401 - --- Exporting PDF for: Report_Brief.md ---
2026-10-08 15:44:24.915 | INFO     | utils:_render_weasyprint:1318 - Attempting PDF export via WeasyPrint...
2026-10-08 15:44:27.068 | SUCCESS  | utils:_render_weasyprint:1344 - PDF generated: tutorial_output\07_Report_Summary\Report_Brief.pdf
2026-10-08 15:44:27.069 | SUCCESS  | utils:export_report:1429 - [Report_Brief.md] completed using WeasyPrint.

Result

Use normalized_dataset.intensity, normalized_dataset.sample_metadata, and normalized_dataset.feature_metadata for downstream analysis. Inspect each processing result.audit for method selection, tracking, thresholds, warnings, and plot payloads; inspect the named QA Audits for cross-stage quality evidence. Persist MetaboDataset and typed Audit objects separately through pimqc.serialization when a workflow plugin, LLM skill, or later Python session needs a stable hand-off. See the native API reference for the complete contracts and additional examples.