I automate things that shouldn't be manual.

Problem: 500+ Key Risk Indicator / Key Control Indicator metrics needed a single governed pipeline — not 500 one-off scripts — that could handle daily volumes ranging from thousands to millions of records per metric without breaking on outliers. Regulatory compliance meant every number had to be auditable, every schema change had to be tracked, and every late-arriving data point had to be handled without silently inflating yesterdays figures.

Architecture: I designed a medallion lakehouse on AWS with three explicit layers: Bronze (raw S3 landing zone with Auto Loader), Silver (Databricks Delta tables with SCD Type 2 history), and Gold (Redshift aggregates for Tableau). Auto Loader continuously monitors an S3 prefix and auto-detects new files without manual orchestration — critical when 20+ source systems push data at different times of day. Instead of a separate job per metric, one parameterised Glue PySpark job handles all 500+ by reading source config from a lightweight catalog table, dramatically reducing operational overhead.

Technical choices & tradeoffs: For SCD Type 2 (capturing historical changes), I chose MERGE INTO Delta instead of a separate upsert+insert pattern. Why? Merge is atomic — if a re-run happens or a job crashes mid-write, the entire transaction rolls back instead of leaving partial historical records. Partitioned by source + load_date in Bronze (not by id), which lets late-arriving data for yesterday get corrected without re-scanning the entire Bronze table. In Silver, version columns track effective_date + end_date per metric ID, so analysts can time-travel to any snapshot without rebuilding history. Glue bookmarks handle incremental processing — a rerun from the same source re-applies bookmarks to the last committed offset, so metrics never double-count.

Cost optimisation (35% total reduction): Broke down as: 15% via S3 Intelligent-Tiering (cold Bronze data auto-archived to Glacier after 30 days), 12% via Glue DPU right-sizing (profiled jobs, dropped from 10 DPUs to 6 where parallelism plateaus), 8% via scheduling non-urgent Gold aggregations during off-peak AWS hours using EventBridge. Redshift Workload Management (WLM) queues separate Tableau dashboard queries (low-latency, high-concurrency) from batch Gold layer inserts, preventing heavy analytical jobs from blocking dashboard refreshes.

Security & Compliance (regulatory requirement): KRI/KCI metrics feed compliance reports to regulators, so data governance is non-negotiable. S3 Bronze and Silver layers use encryption-at-rest (AES-256 default encryption), Redshift Gold layer enforces column-level encryption for sensitive metrics. Data in-flight is encrypted via TLS 1.2 between all AWS services. Access control is role-based: Analysts have read-only Redshift query access (via IAM roles), Data Engineers have full Glue job permissions, and only the compliance team can access historical audit tables. All data access is logged to CloudTrail and Redshift query logs, with retention policies enforcing 7-year storage for regulatory hold periods. PII (if any) in metric names is masked at the Bronze layer before Silver transformation.

Disaster Recovery & Backup: Redshift cluster maintains automated daily snapshots to S3 with point-in-time restore capability — can recover to any snapshot within the last 35 days. S3 Bronze data is versioned and cross-region replicated to a secondary AWS account for disaster recovery. If the primary Redshift cluster fails, failover to a standby cluster takes under 5 minutes via AWS RDS Proxy and automatic DNS failover. Delta Lake tables in Databricks use Unity Catalog replication for audit trail immutability — historical data is immutable after 30 days, preventing accidental or malicious overwrites.

Spark optimisation specifics: Used broadcast joins for small dimension tables (lookup tables under 100MB) against large fact tables, avoiding shuffle. Salting on high-cardinality join keys (metric category) that used to bottleneck single executors. Enabled Adaptive Query Execution (AQE) so Spark can re-optimize join strategy at runtime based on actual data size instead of static cardinality estimates — catches skewed partitions without manual intervention.

Monitoring & SLA enforcement: Every Glue job is instrumented with CloudWatch metrics — job runtime, input row count, error rate per metric source. If a job exceeds its baseline runtime by 20%, an alarm triggers for investigation (catches slow-downs before they become failures). Great Expectations runs schema and business logic validation on every Silver table load; if a validation fails (e.g., "metric value must be non-negative"), the entire job is marked as failed and an SNS notification pages the on-call engineer instead of silently publishing bad data to Redshift. Redshift query performance is monitored via WLM queues — if interactive dashboard queries start timing out, an alert fires so we can rebalance SLA priorities. This multi-layer alerting (job health + data quality + query performance) means compliance teams see a consistent, trustworthy platform.

Why it mattered: Replaced fragmented, per-metric scripts with one governed, horizontally-scalable pattern — new metric onboarding now takes config changes, not code changes. Historical data can be audited to source by following the MERGE lineage. Regulators can query both current and historical states directly, which is why the SCD Type 2 structure matters — it's not just data engineering, it's compliance evidence.

Problem: CISO, Intel, Hunt, and Detection teams each had their own reporting, with no unified, low-latency view of threat activity — and cyber operations dashboards can't tolerate the same staleness that's fine for a monthly compliance metric. An alert that's 30 minutes old is already stale in the security world.

Architecture: A dual-mode platform on AWS: Kinesis Data Streams for operational dashboards (15-min latency SLA), and batch Glue jobs for historical aggregates published once daily to Redshift. Lambda extracts Splunk saved-search results on a 5-minute cadence, partitioning alerts by threat category before landing in Kinesis. This way, high-volume categories don't starve low-volume ones (the single-partition anti-pattern).

Streaming specifics: Kinesis uses 8 shards partitioned by finding category (AppSec, Vulnerability, Secret, etc.) to spread load evenly — each shard processes up to 1000 records/sec. Consumer group logic (in Lambda) reads from the shard iterator, checkpoints offsets to DynamoDB every 100 records to handle failures gracefully. Out-of-order events (alerts arriving late) are tolerated up to a 1-hour lookback window before they're considered historical and relegated to the batch layer — this balances responsiveness with consistency. If Kinesis ingestion lags past 15 minutes, a CloudWatch alarm pages an oncall engineer instead of silently missing the SLA.

Batch complement: Redshift Workload Management (WLM) queues separate interactive dashboard queries (low-latency, high-concurrency) from batch Glue inserts, preventing heavy analytical jobs from blocking dashboard refreshes. That separation is why P95 latency stays under 2 seconds even while loading millions of historical rows nightly.

Why it mattered: Gave four previously siloed security teams one operational platform with explicit latency tiers — streaming for "right now" decisions, batch for audit trails. CISO now has a 15-min view of threat landscape instead of waiting for EOD reports. Not flashy, but security teams actually use it multiple times a day because the latency is trustworthy.

Problem: Tracking mandatory compliance training completion for 50,000+ employees needed to be zero-touch and audit-ready — regulators treat this as evidence, not just an internal report.

What I built: A fully serverless pipeline — EventBridge-scheduled Lambda ingests the SharePoint API, lands records in S3 with date partitioning, a Glue job deduplicates using a SHA-256 hash on employee ID and training code, and Great Expectations validates completeness before the Redshift load that feeds a Tableau dashboard used directly as regulatory audit evidence.

Engineering detail: Full observability with no human in the loop — CloudWatch alarms fire on Glue job failure, Lambda timeout, or a Great Expectations validation breach, so a broken run pages an engineer instead of silently publishing incomplete compliance data.

Why it mattered: Eliminated roughly 4 hours a week of manual tracking work and gave auditors a dashboard they could rely on directly, instead of a spreadsheet someone had to vouch for.

Problem: Without a shared standard, every engineer on the team was making the batch-vs-streaming call differently for new pipelines — some over-engineering low-stakes metrics with streaming infrastructure, others leaving genuinely time-sensitive data on slow batch schedules.

What I built: A documented decision framework: regulatory KRI/KCI metrics stay batch, because they need low frequency and high accuracy more than they need speed; cyber operations dashboards run on Kinesis structured streaming, because a 15-minute staleness window there is an operational risk, not an inconvenience. I documented the tradeoffs and thresholds explicitly rather than leaving it to instinct.

Why it mattered: Adopted by the whole 10+ engineer team — it turned an inconsistent, case-by-case judgment call into a standard anyone on the team can apply the same way I would.

Problem: Duplicate records were leaking in from more than 20 different source systems, each with its own quirks, and catching them after the fact in Tableau was too late — the bad number had already been seen.

What I built: SHA-256 hash deduplication on business keys at Bronze ingestion, MERGE INTO Delta Lake statements at the Silver layer so re-running a load never creates duplicates, and mandatory source-to-target row count reconciliation via Great Expectations before anything is considered "loaded."

Why it mattered: Took duplicate-record incidents across 20+ source systems down to zero — and it started, earlier in my career as an Analyst, with independently catching a duplication bug that was silently inflating a live financial metric before anyone else noticed.

Why: Stakeholders frequently ask what a chart is actually showing — which calculation it uses, which filters are applied, what a spike means — and that context usually lives only in the dashboard author's head.

What I'm building: A large language model connected to a semantic metadata layer describing every visualization in a Tableau workbook — its underlying calculation logic, applied filters, data source, and refresh cadence. A stakeholder can ask, in plain English, "what does this chart represent" or "why did this metric move this week," and the model answers by grounding its response in the actual dashboard metadata rather than guessing from the visual alone.

Why it's worth building: This is the direction I see BI heading — dashboards that can explain themselves instead of requiring a data engineer in the room to interpret them.

Why: The most common cause of a broken downstream pipeline isn't bad data — it's an upstream schema change nobody announced.

What I built: A lightweight data contract framework where each producing team declares an expected schema and set of quality guarantees; incoming data is validated against the contract at ingestion, and breaking changes are flagged and blocked before they propagate downstream, instead of being discovered when a report silently goes blank.

Why: Cloud data platforms make it easy to over-provision compute and never find out until the bill arrives — I wanted cost visibility at the same granularity as performance monitoring.

What I built: A tracking layer that captures cost-per-job and cost-per-terabyte-processed for Glue and Databricks workloads, correlates spend against cluster configuration (instance type, autoscaling range, spot versus on-demand), and surfaces right-sizing recommendations — the same FinOps thinking I applied to reduce our platform cost by 35%, generalized into a reusable tool.

Why: Data catalogs go stale the moment a human has to manually write column descriptions — documentation debt compounds faster than most teams can pay it down.

What I built: A generative AI layer that inspects table structure, column-level statistics, and upstream lineage, then drafts human-readable descriptions and lineage narratives for review rather than requiring an engineer to write them from scratch — cutting documentation time down to a review-and-approve step.

Why: Rule-based detection catches known attack signatures well but struggles with variations of a known technique — a natural fit for the security domain I already work in.

What I built: Security telemetry (log events, alert descriptions) embedded into a vector database, so that a new alert can be compared against historical incidents by semantic similarity rather than exact string or rule match — surfacing "this looks like a variant of an incident we saw before" even when the raw text doesn't match.

Why: The reconciliation pattern I use at Barclays is tied to our internal tooling — I wanted a portable, open-source version anyone could drop into their own pipeline.

What I built: A lightweight Python framework running null, uniqueness, freshness, and schema-drift checks at any pipeline stage, logging results to a queryable audit table and alerting on threshold breaches before bad data reaches a dashboard — the same discipline behind the zero-duplicate-incident work, generalized and open.

Why: My production streaming work runs on Kinesis — I wanted hands-on depth with Kafka's consumer group and schema evolution model too, since the two show up in different job descriptions.

What I built: Kafka producers/consumers streaming event data, aggregated in near-real-time and pushed to a live dashboard, with schema evolution handled explicitly instead of assumed away.

Why: Almost everything I work on professionally is structured/tabular. Image, video, and activity data break those assumptions, and I wanted to know exactly how.

What I built: Object storage lifecycle rules for large binary assets on S3, metadata extraction into a queryable index, and file-format/partitioning choices tuned specifically to stop read performance from degrading as volume grew.

Why: Control owners and compliance teams spend real time manually searching policy documents to answer audit questions — the same problem the Security Compliance Tracker solves for training records, applied to policy text.

What I built: A small retrieval-augmented assistant over internal policy and control documentation using a vector database, aimed at cutting manual lookup time — grounded and source-cited, not a free-form chatbot guessing at policy.