Building a Netflix-Scale Streaming Intelligence Platform: Architecture, ML, and the Data Engineering Behind It

When most people think about analytics, they think dashboards. When Netflix thinks about analytics, they think about a system that processes millions of streaming sessions, predicts which users will cancel next month, measures the exact revenue impact of a content thumbnail change, and does all of this across seven countries — simultaneously.

I built one. Not Netflix itself — but the full analytics architecture that replicates what a senior analytics engineering team at a company like Netflix would design, implement, and operate. The result is the Streaming Merchandising Intelligence Platform: a production-grade ecosystem simulating Netflix-scale operations across 7 APAC markets, live at analytics.kpintel.in.

The Vision: What Separates Analytics from Intelligence

There is a critical distinction between reporting and intelligence. Reporting tells you what happened yesterday. Intelligence tells you what is happening now, explains why, and predicts what will happen next. Most analytics projects stop at reporting. This platform is designed to demonstrate the full intelligence stack.

The platform answers four categories of questions that a real streaming business needs to operate:

• Merchandising effectiveness: Which content slots, thumbnails, and promotional placements are actually driving engagement — and which are wasting screen real estate?
• Experimentation rigour: Did the new homepage layout improve conversion? How confident are we, statistically?
• Retention intelligence: Which user cohorts are churning? Can we predict who will cancel before they do?
• Regional strategy: How does viewing behaviour in Japan differ from India, and how should content strategy reflect that?

Architecture: Demo Stack and the Ideal Production Path

The platform is built in two conceptual layers: what was actually built and deployed (the demo architecture), and how this system would be re-engineered for true Netflix-scale production. The diagram below shows both — and the equivalences between them.

Layer 1: Data Engineering Foundation

The base layer generates and processes synthetic but statistically realistic data representing user behaviour across all seven APAC markets. The ETL pipeline is written in Python 3.11 using Pandas and PyArrow for high-throughput columnar processing, with output stored in Parquet format — the same format used by Spark, Databricks, and every modern data lakehouse.

Why DuckDB alongside PostgreSQL?

Most analytics projects use PostgreSQL for everything. This platform uses PostgreSQL 15 as the operational store and DuckDB 1.1 as the OLAP query engine. DuckDB allows sub-second analytical queries on multi-million row datasets without a Spark cluster — it runs in-process, reads Parquet natively, and is ideal for the self-service analytics use case. In production, this role would be filled by Apache Druid or ClickHouse at petabyte scale.

Layer 2: The dbt Semantic Layer

Raw data without a semantic layer produces the most dangerous output in analytics: numbers that look right but mean different things to different people. I modelled the full semantic layer in dbt with 15 models across three tiers.

• 5 staging models — standardised, cleaned, typed source data. No business logic, just trust.
• 3 intermediate models — business logic transformations. Attribution rules, session stitching, campaign mapping.
• 7 mart models — final dimensional tables for each analytics domain. Star schema with SCD Type 2 for historical accuracy.

dbt's built-in testing framework enforces data quality contracts on every pipeline run: uniqueness, referential integrity, non-null constraints, accepted value ranges. In production, this would be complemented by Great Expectations for deeper statistical validation and DataHub for data lineage and discovery.

Layer 3: Pipeline Orchestration

Apache Airflow 2.9 orchestrates the full pipeline on a daily DAG firing at 06:00 UTC. The DAG runs in order: Extract → Transform → dbt run → ML scoring → data quality checks. One non-obvious design decision: SRM (Sample Ratio Mismatch) detection runs as a gate before any experiment results are trusted. If traffic allocation between experiment variants deviates beyond tolerance, the downstream experiment analysis is flagged automatically.

In production, the batch-only Airflow DAG would be complemented by Apache Flink for real-time stream processing — enabling the system to react to user behaviour within seconds rather than the next morning.

Layer 4: The FastAPI Analytics Backend

The dashboard and ML systems do not query the database directly. All data flows through a FastAPI 0.115 backend with five router modules, each serving a distinct analytics domain: regional KPIs, content performance, experiment results, merchandising analytics, and ML model outputs.

The API uses async SQLAlchemy for non-blocking database connections, API key authentication, and response caching. In production, this layer would expand to a GraphQL API gateway with service mesh, horizontal auto-scaling on Kubernetes, and a Redis cluster for sub-millisecond hot query caching.

Layer 5: Machine Learning Models

This is where the platform moves from "reporting what happened" to "predicting what will happen." Four models are in production:

Churn Prediction — LightGBM

Uses 30+ engineered behavioural features — recency of viewing, binge session frequency, genre diversity score, cross-device pattern, days since last session, completion rate trend — to predict which users will cancel within 30 days. The model achieves AUC-ROC of ~0.84 and outputs a three-tier risk score (High / Medium / Low) for each user, enabling targeted retention campaigns rather than blanket interventions.

Demand Forecasting — Prophet

Predicts title-level viewing demand across each APAC market. Prophet was chosen specifically for its handling of APAC market seasonality — Japanese Golden Week, Indian festivals, Korean Chuseok, Australian summer schedules — all encoded as custom regressors. The model achieves MAPE of approximately 3.1%, meaning content scheduling and merchandising decisions can be made with high confidence 14–30 days ahead.

Audience Segmentation — K-Means RFM

Groups 200,000 users into 5 behavioural clusters using Recency, Frequency, and Monetary (engagement value) scoring. The Silhouette score of ~0.39 confirms meaningful cluster separation. Each segment receives different content merchandising weights, recommendation biases, and campaign targeting — enabling personalisation without individual-level model complexity.

Experimentation Engine — Frequentist + Bayesian

Most analytics platforms stop at p-values. This engine implements both approaches:

• Frequentist: Z-test and Welch's t-test with confidence intervals, statistical power calculations, and minimum detectable effect sizing.
• Bayesian: Beta-Binomial conjugate model giving a direct probability that the variant beats the control — no binary p-value threshold, no "marginally significant" ambiguity.
• SRM Detection: Automatic flagging of traffic allocation errors before results are trusted, preventing false conclusions from biased experiment populations.

“The difference between a dashboard and an intelligence platform is not the technology. It is the architectural discipline applied to every layer — from data contracts at ingestion to statistical rigour at the output.”

Layer 6: The Self-Service Dashboard — 10 Pages

The dashboard is built in Streamlit 1.39 with Plotly 5.24 in Netflix's dark theme. Each of the 10 pages is designed around a specific business question, not a data schema — a distinction that matters enormously for adoption.

• Executive Overview — Platform health, regional comparison, content mix at a glance
• Title Performance — Funnel analysis, 90-day trend, benchmark scoring against portfolio
• Merchandising Analytics — Slot CTR heatmap, campaign ROI attribution, Sankey traffic flow
• APAC Regional Intelligence — Choropleth maps, market KPIs, genre preference matrices
• Experimentation Hub — A/B results, confidence intervals, forest plot, sample size calculator
• Engagement Funnel — Platform funnel drop-off, binge session analysis, completion by device
• Retention Analysis — Cohort heatmaps, retention curves, churn score distribution
• Content Discovery — Source attribution, recommendation algorithm performance scoring
• Campaign Effectiveness — Portfolio ROI, spend vs revenue trend analysis
• Operational Monitoring — Pipeline health, data freshness indicators, SRM alerts

The Technology Stack

• Database: PostgreSQL 15 (OLTP) + DuckDB 1.1 (in-process OLAP)
• ETL: Python 3.11, Pandas, PyArrow, columnar Parquet storage
• Semantic Model: dbt (15 models — staging, intermediate, marts), SCD Type 2
• API: FastAPI 0.115, Uvicorn, async SQLAlchemy, API key authentication
• Dashboard: Streamlit 1.39, Plotly 5.24, Netflix dark theme
• Machine Learning: LightGBM, Prophet, scikit-learn, SciPy
• Orchestration: Apache Airflow 2.9, daily DAG with data quality gates
• Infrastructure: Docker Compose, NGINX reverse proxy

What the Ideal Production Architecture Looks Like

The demo architecture makes deliberate trade-offs to run on a single server. A true Netflix-scale deployment would replace each layer with distributed, fault-tolerant, real-time equivalents — as shown in the lower half of the architecture diagram above.

The key architectural shifts from demo to production:

1. 1.Kafka replaces synthetic data generation — billions of real user events stream in continuously rather than being generated in batch.
2. 2.Apache Flink sits alongside Airflow — real-time stream processing enables second-level latency rather than daily batch cycles.
3. 3.Delta Lake / Apache Iceberg replaces raw Parquet — ACID transactions, time travel, and schema evolution at petabyte scale.
4. 4.Apache Druid or ClickHouse replaces DuckDB — distributed OLAP with real-time Kafka ingestion for sub-second queries across billions of rows.
5. 5.Feast Feature Store replaces ad-hoc feature engineering — versioned, reusable features shared across all ML models with point-in-time correctness.
6. 6.Kubernetes + MLflow replace local model execution — distributed training, A/B model experiments, canary deployments, automated retraining triggers.
7. 7.React + Apache Superset replaces Streamlit — production-grade BI with role-based access, embedded analytics, and custom UX.

Why I Built This

After 20+ years of building enterprise data platforms for broking firms, wealth management companies, and capital markets, I have consistently seen the same gap: organisations that understand the value of data intelligence but struggle to architect systems that deliver it at scale.

The engineering principles behind this platform — semantic modelling, statistical rigour in experimentation, feature-centric ML, business-question-led dashboard design — apply directly to enterprise BFSI analytics. The technology names change (Kafka instead of NSE feeds, Druid instead of SSAS cubes) but the architectural discipline is identical.

This project exists to demonstrate that discipline concretely, not just in theory.