CS 330 · Year 3 · Semester 2 · 3 credits · Computer Science

Distributed Systems, Cloud & Data Engineering

Architecting fault-tolerant, petabyte-scale data pipelines and lakehouses for mission-critical actuarial modeling and real-time risk intelligence.

Full practice exam · 30 Q · 120 min

Season 1 · 8 episodes

Pipeline to Solvency: Re-Engineering Apex Mutual

When legacy mainframe batch jobs crash 48 hours before a mandatory state insurance audit, a newly appointed data systems actuary must re-architect Apex Mutual's entire distributed data infrastructure from bare metal to a secure, cloud-native lakehouse.

Protagonist · Maya Lin, FSA and Lead Data Systems Actuary at Apex Mutual, caught between old-school reserving actuaries and modern cloud infrastructure demands.

Setting · Apex Mutual's actuarial computing headquarters in Hartford, Connecticut, amid an aggressive migration to a distributed cloud lakehouse.

Stakes · If the quarterly reserve valuation and regulatory claims triangle pipeline fails to complete within the statutory deadline, state regulators will issue a cease-and-desist order, freeze dividend payouts, and place Apex Mutual into supervisory oversight.

Pick an episode

Episodes

Season finale: Maya deploys an end-to-end governed claims lakehouse on Apache Spark, Kafka, and Airflow that shrinks a 36-hour reserving batch run to 42 minutes with differential privacy guarantees, ASOP 23 compliance, and a 60% cloud compute cost reduction.

From the lecturer: Welcome to CS 330. In modern actuarial practice, enterprise pricing engines, catastrophe simulations, and Solvency II / IFRS 17 reserving pipelines are only as dependable as the distributed infrastructure underneath them. When pricing dynamic commercial fleets or reconciling hundreds of millions of policy transactions, single-node scripts collapse. In this film series, we bridge foundational distributed systems theory—consensus, partitioning, and shuffle mechanics—with production cloud data engineering across Apache Spark, Delta Lake, Airflow, and secure cloud environments. To maximize your mastery, keep an engineering notebook beside you: sketch cluster network topologies, draw Spark Directed Acyclic Graphs (DAGs), trace physical memory plans, and compute partition and shuffle overheads by hand before we inspect the cluster UI. Every technical pattern is paired with enterprise actuarial data governance, encryption standards, and cloud cost control. Build alongside each lesson in your cloud lab; distributed computing cannot be learned through passive viewing. Treat every out-of-memory exception, partition skew, and IAM permission boundary as a first-principles challenge in latency, fault tolerance, and financial risk integrity.

Cold open

At 2:14 AM on the final Sunday of the quarterly close, a legacy 8-node valuation cluster stalls at 99% completion. Apex Mutual's chief actuary storms into Maya's office: the quarterly claims triangle job has run for 28 hours, and throwing four more servers at the cluster just increased execution time by twenty minutes.

preparing lecture…

Transcript

Differentiate vertical scaling from horizontal scaling and quantify speedup limits using Amdahl's and Gustafson's Laws.
Evaluate distributed consistency guarantees and latency trade-offs via the CAP and PACELC frameworks.
Diagnose communication overhead, network serialization costs, and data skew bottlenecks in partitioned actuarial workloads.