Data Platform Briefing · June 2025

Data Intelligence
Platform

Azure Databricks · Medallion Architecture · ML & Analytics

500KCustomers
R2.3TAUM
200Branches
12ML Models

Confidential · Internal Use Only · Prime Capital Bank Limited · © 2025

The Problem

Why We Had to Act

Legacy infrastructure was blocking growth, creating regulatory risk, and costing the business millions annually.

Fragmented Data Silos

14 isolated source systems with no shared data model. Customer data existed in 7 different systems with no single view. Analysts spent 70% of time on data wrangling, not analysis.

Regulatory Pressure

SARB BA700 preparation took 14 person-days monthly. 3 SARB Directive Letters in 24 months for data quality failures. IFRS 9 ECL calculated on spreadsheets — model risk violation.

Fraud & Risk Blind Spots

Fraud detection ran T+1 batch — too late to stop transactions. Credit scoring took 48 hours — unacceptable for instant loan products. No real-time AML transaction monitoring.

Technology Debt

On-premises Oracle DW: end-of-life in 2024. ETL pipelines: 200+ SSIS packages, zero documentation. Data quality: no automated testing, no lineage, no governance framework.

The Solution

Medallion Architecture

A single governed, cloud-native data platform replacing every legacy data system in the bank.

RAW Source data landed as-is from 8 systems via ADF, Auto Loader & Event Hub · No transformation · Full fidelity preservation ~50 TB
BRONZE Immutable Delta Lake tables · Audit metadata appended · Streaming + batch ingest · Schema inference & evolution ~38 TB · 14 tables
SILVER Cleansed · Conformed · PII-masked · SCD2 history · Great Expectations quality gates · dbt transformations ~22 TB · 42 tables
GOLD Star schema · Analytics-ready · Power BI · Azure SQL · ML Feature Store · Regulatory reports · Sub-2s queries ~8 TB · 54 models
✔ Unity Catalog governance ✔ Azure Private Endpoints ✔ Column-level PII masking ✔ Full data lineage ✔ Automated regulatory reports
Technology

The Azure Stack

Best-in-class cloud services, fully integrated on Microsoft Azure.

🗄️
ADLS Gen2
Data Lake Storage
~78TB managed
⚡
Azure Databricks
Compute + ML
Photon + MLflow
🔷
Delta Lake
ACID transactions
Time travel · MERGE
📋
Unity Catalog
Governance + Lineage
247 tables catalogued
📊
Power BI Premium
8 dashboards
500 active users
🔧
dbt Core
54 models · 1,847 tests
100% test pass rate
🔒
Key Vault + AAD
Zero plaintext secrets
MFA + PIM + CMK
🤖
MLflow Registry
12 models in prod
Champion-challenger
Data Model

Gold Star Schema

Kimball star schema design — 3 fact tables, 11 dimension tables, optimised for sub-second analytics.

FACT TABLES fact_transaction (2.1B rows) fact_loan_portfolio (85M rows) fact_account_balance (730M rows) dim_customer SCD2 · 500K rows dim_account SCD2 · 1.8M rows dim_product SCD1 · 850 rows dim_date Static · 10,957 rows dim_branch SCD2 · 200 rows dim_channel Static · 18 rows dim_currency Daily · 45 rows dim_merchant Monthly · 250K rows dim_risk_category SCD1 · 120 rows dim_geography Static · 4,200 rows dim_employee SCD2 · 12K rows
Bronze Layer

Ingestion at Scale

Auto Loader, Event Hub streaming, and ADF batch pipelines landing data into immutable Delta tables.

14Bronze TablesAcross 7 source domains
~38TBCompressed StorageDelta Lake format
2.1BTransaction RecordsStreaming + batch
<10minMax Ingest LatencyCard transactions
100%Schema EvolutionAuto Loader auto-detect
30 daysDelta HistoryPoint-in-time recovery
Auto Loader Advantage

Databricks Auto Loader (cloudFiles) provides exactly-once ingestion semantics via checkpoint state in ADLS — surviving cluster restarts without data duplication or loss. Schema evolution is handled automatically via column merging, eliminating the need for manual pipeline changes when source systems add columns.

Silver Layer

Data Quality & Cleansing

Great Expectations quality gates, dbt transformations, PII masking, and SCD2 history preservation.

42Silver Tables6 domain schemas
99.94%Quality Pass RateMonthly average
7DQ Check TypesGreat Expectations
SCD2History ModelFull audit trail
  • Null handling — configurable per column (reject / default / flag)
  • Format standardisation — ISO 8601 dates, ZAR base currency
  • Deduplication — row_number() over partition by business key
  • PII masking — HMAC-SHA256 tokenisation via Key Vault CMK
  • Referential integrity — FK validation against reference tables
  • Freshness SLA — source staleness alerts within 2 hours
  • SCD2 MERGE — full Delta Lake MERGE for dimension history
Gold Layer

Analytics Ready

Star schema optimised for Power BI DirectQuery, regulatory reporting, and ML feature serving.

<2sAvg Query TimePower BI DirectQuery
3–8×Photon Speedupvs standard Spark SQL
~8TBGold StorageOptimised + Z-ordered
54dbt Models100% tested + documented
1,847dbt Tests100% pass rate enforced
247Catalogued TablesUnity Catalog lineage
Liquid Clustering Delta OPTIMIZE Z-Order on join keys Databricks SQL Serverless Photon Engine 5-year Hot tier retention
Machine Learning

12 Production Models

Credit risk, fraud detection, AML scoring, and customer intelligence — all governed by MRM policy.

PD Model (Basel IRB)

0.94
Gini Coefficient

XGBoost · 180 features · 12-month PD · Monthly batch scoring · Full loan book

LGD Model (IFRS 9)

0.89
R² Score

GBM Regressor · Collateral + recovery features · ECL provisioning dependency

Fraud Detection

0.97
AUC-ROC

Real-time <50ms · GBM · 3yr fraud labels · R45M annual savings

AML Scoring

0.91
AUC-ROC

Graph network analysis · STR automation · Mule account detection

MLflow Model Registry
340 Feature Store features
Champion-challenger live in prod
Quarterly backtesting
Azure API Management serving
Fraud Detection

Real-Time Protection

Sub-50ms fraud scoring on every card transaction — saving R45M per year.

Annual Fraud Savings

R45.2M
Prevented fraud losses · FY2024
85% reduction vs FY2022 baseline

Model Performance

0.97
AUC-ROC · 3-year training set
False positive rate: 0.8% (industry: 1.5%)

Inference Latency

<50ms
P99 · Azure API Management
Well within card scheme 100ms limit

Transactions Scored

1M+
Events per second peak capacity
Current load: ~180K TPS average

Pipeline: Card transaction → Event Hub → Databricks Structured Streaming → Feature Store enrichment → GBM model score → API response (card hold / pass) → Fraud ops dashboard alert → ML retraining feedback loop

Compliance

Regulatory Automation

Zero manual data preparation for SARB, IFRS 9, Basel III, POPIA, and FICA compliance.

FrameworkRequirementPlatform ImplementationStatus
SARB BA700Monthly prudential returnAutomated XML from rpt_sarb_ba700 dbt mart · T+2 submission✔ Automated
IFRS 9 ECLMonthly provisioningPD × LGD × EAD pipeline · Stage 1/2/3 classification · GL journal output✔ Automated
Basel III LCRDaily liquidity ratioGold treasury data · Regulatory weight lookup · Daily scheduled job✔ Automated
Basel III NSFRQuarterly stable fundingBalance sheet positions from Gold layer · SARB-approved formula✔ Automated
POPIAData privacy & PII protectionUnity Catalog column masking · DSAR pipeline · Breach notification workflow✔ Embedded
FICA / AMLSTR submission within 15 daysAML model → STR draft generation → Compliance review → FIC submission✔ Automated
BCBS 239Risk data aggregationUnity Catalog end-to-end lineage from Bronze source to Gold metric✔ Compliant
Results

Business Outcomes

Measurable, sustained improvements across efficiency, risk, compliance, and customer intelligence.

99.8%Pipeline SLA47 pipelines · trailing 12 months
<2sQuery PerformancePower BI DirectQuery avg
40%Less Manual ReportingAnalyst time saved monthly
R45MFraud PreventedAnnual savings · FY2024
6hrsMonthly Close CycleDown from 5 business days
0SARB DirectivesSince go-live (was 3 in prior 24m)
ECL Coverage
R4.8B provision
1.26% coverage ratio
Capital Adequacy
CET1: 13.2%
vs 10% min requirement
User Adoption
500 BI users
8 dashboards · NPS 72
What's Next

Platform Roadmap

Three-phase evolution from foundational analytics to AI-native banking intelligence.

✔ Phase 1 — Complete
Foundation
  • Medallion architecture on Azure Databricks
  • Bronze / Silver / Gold layers live
  • dbt Core transformation pipeline
  • Unity Catalog governance framework
  • 8 Power BI dashboards deployed
  • Credit scoring + fraud models in prod
  • SARB BA700 + IFRS 9 automated
→ Phase 2 — In Progress (Q3 2025)
Real-Time CDC
  • Debezium CDC from core banking T24
  • Sub-minute data freshness on Silver layer
  • Streaming Silver transformations
  • Real-time Customer 360 dashboard
  • Event-driven AML alert pipeline
  • Open Banking API (PSD2-aligned)
◌ Phase 3 — Planned (Q1 2026)
AI Assistant
  • Natural language query on Gold layer
  • Databricks AI/BI Genie integration
  • Automated regulatory narrative generation
  • Predictive branch performance AI
  • Customer churn intervention engine
  • Real-time credit limit optimisation
Governance

Team & Data Ownership

A federated data governance model with clear ownership, stewardship, and support accountability.

👤
Chief Data Officer
Data Strategy
Platform vision, enterprise data policy, board-level reporting on data maturity
👤
Head of Data Engineering
Engineering Owner
Architecture decisions, pipeline SLA accountability, Databricks workspace governance
👤
Lead Data Scientist
ML & Analytics
Model development, MRM policy compliance, Feature Store ownership, MLflow registry
👤
Data Governance Manager
Governance & POPIA
Unity Catalog policies, PII classification, POPIA compliance, DSAR management
👤
Platform SRE Lead
Reliability
99.8% SLA delivery, incident management, DR testing, monitoring & alerting
👤
Regulatory Reporting Lead
Compliance
SARB/IFRS/Basel III return ownership, dbt regulatory mart review & sign-off

Support model: L1 — Data Users (self-service)  |  L2 — BI & Analytics team  |  L3 — Data Engineering & SRE  |  SLA: P1 <1hr · P2 <4hr · P3 <2 business days

Next Steps

Ready to Explore?

Three Actions to Take Now

  • Access the dashboards — Log into Power BI with your Azure AD credentials. Your role determines your dashboard scope.
  • Review the Data Dictionary — docs/data_dictionary/data_dictionary.md documents all 247 tables.
  • Request data access — Submit a Unity Catalog access request through the ServiceNow Data Access portal. Approved within 1 business day.
Questions? Contact the Data Engineering team · data.engineering@primecapital.co.za

Prime Capital Bank Limited · Registration No. 1968/000123/06 · Authorised FSP (FSP 12345)
Confidential — Internal Use Only · © 2025 Prime Capital Bank Limited

1 / 15