SD System Design In Depth
0%
lesson system-designsecurityabuse

Security and abuse prevention

Module 5.1: Production Reliability, SLOs & Resilience Updated Sep 18, 2026
📄 Cheat Sheet
Top 12 Tips For API Security
ByteByteGo Top 12 Tips For API Security Watch on YouTube ↗

Compact ByteByteGo overview of the layered controls in this unit: HTTPS, OAuth2/JWT, rate limiting, input validation, WAF and API gateway.

0:00 / --:--

Under the Hood: Defense-in-Depth Architecture

Modern system design operates under the Zero Trust security paradigm: assume the perimeter is already breached, network transport is inherently hostile, and every request must be authenticated, authorized, and continuously validated. Relying on a single firewall or border proxy creates a catastrophic single point of failure.

Layered Defense Perimeter

flowchart TD Internet["Public Internet Traffic"] --> Cloudflare["1. Edge CDN & DDoS Mitigation (Anycast BGP Scrubbing)"] Cloudflare --> WAF["2. Web Application Firewall (SQLi, XSS, OWASP Top 10)"] WAF --> BotMit["3. Bot & Abuse Detection (JA3 Fingerprinting, Turnstile)"] BotMit --> Gateway["4. API Gateway (OAuth2 / JWT Token Validation, Rate Limiting)"] Gateway --> Mesh["5. Internal Service Mesh (Mutual TLS 1.3 with SPIFFE / SPIRE)"] Mesh --> ServiceA["Microservice A (Least Privilege IAM)"] ServiceA --> DB[("Encrypted Datastore (Envelope Encryption with AWS KMS / HashiCorp Vault)")]

Production Abuse Mitigation Vectors

1. Credential Stuffing & Automated Bot Mitigation

Malicious actors weaponize billions of leaked password dumps to execute automated distributed brute-force attempts against login and authentication endpoints. Defenses include TLS Client Hello (JA3/JA4) fingerprinting to identify headless HTTP client scripts, IP reputation lookups, and progressive behavioral challenges (invisible CAPTCHA) triggered when request anomalies exceed threshold limits.

2. Mutual TLS (mTLS) & Workload Identity

Inside the cloud datacenter, plain unencrypted HTTP between microservices is unacceptable. Service meshes (Istio, Linkerd) enforce Mutual TLS (mTLS) with short-lived x509 certificates issued by the mesh CA or SPIRE and rotated automatically (Istio defaults to 24-hour workload certificates; SPIRE's default SVID TTL is 1 hour; both are configurable). Every service verifies the exact cryptographic identity of the calling service before granting RPC access.

3. Envelope Encryption at Rest

Data written to persistent storage is secured using Envelope Encryption: a unique local Data Encryption Key (DEK) encrypts the file or database row, and the DEK itself is encrypted by a root Key Encryption Key (KEK) locked inside a Hardware Security Module (HSM / AWS KMS). The plaintext DEK exists only in volatile memory during active encryption/decryption.

Threat-to-control map

ThreatLayerControlReal-world example
Volumetric DDoS (UDP/SYN floods, amplification)Network edgeAnycast absorption, scrubbing, SYN cookiesCloudflare, AWS Shield, Google Cloud Armor
HTTP floods at layer 7CDN/WAFPer-path rate limits, challenge pages, cachingCloudflare and Google publicly reported HTTP/2 Rapid Reset attacks peaking at hundreds of millions of requests per second in 2023
SQL injection, XSS, SSRFWAF + applicationParameterised queries, output encoding, egress allow-listsOWASP Top 10; the 2019 Capital One breach involved SSRF against cloud instance metadata
Credential stuffing, account takeoverAuth servicePer-account throttling, breached-password checks, MFA, risk scoringHave I Been Pwned k-anonymity range API
Lateral movement after a breachService meshmTLS workload identity, per-service authorisation policySPIFFE/SPIRE, Istio, Google BeyondProd
Stolen disks or snapshotsStorageEnvelope encryption with KMS/HSM, key access loggingAWS KMS, Google Cloud KMS, HashiCorp Vault

Worked example: credential stuffing against a login endpoint

An attacker has 5 million leaked email and password pairs and a pool of 40,000 residential proxy IPs. They send 1 million attempts per hour.

  • That is only 25 attempts per IP per hour, below a per-IP limit of 100 per hour, so IP limits alone do nothing.
  • A typical reuse success rate of about 0.1% to 2% means 1,000 to 20,000 accounts taken over per hour if nothing else stops the attack.
  • Controls that do work: count failures per target account (for example 5 per 15 minutes, then add friction). Watch the global login failure ratio, which jumps from about 5% to 95% during an attack, and trigger challenges for the whole endpoint when it does. Check submitted passwords against breach corpora. Fingerprint clients by TLS characteristics (JA3/JA4) and HTTP header order, and require MFA for risky logins.
  • Return the same error and roughly the same response time for "unknown user" and "wrong password", so the endpoint does not reveal which accounts exist.

Envelope encryption, step by step

sequenceDiagram participant App as Service participant KMS as KMS or HSM participant Store as Object store App->>KMS: GenerateDataKey for key alias orders KMS-->>App: Plaintext DEK and encrypted DEK App->>App: Encrypt payload with DEK using AES-GCM, then discard plaintext DEK App->>Store: Write ciphertext plus encrypted DEK Note over App,KMS: To read, send the encrypted DEK to KMS Decrypt, which checks access and logs the call

The root key never leaves the HSM, and every decrypt call is authorised and logged. Rotating the root key does not require re-encrypting the data, because only the data keys are wrapped with it.

Check yourself Question 1 of 3
Containing lateral movement staff
From: Threat-to-control map

An attacker gains code execution in a low-value internal reporting service and starts calling the payments service directly. Which control is designed to limit this?

Key Takeaways

Zero Trust architecture requires authenticating and encrypting every hop; never trust internal datacenter network traffic.
Protect authentication endpoints using behavioral fingerprinting (JA3 TLS signatures) and composite rate limiting.
Enforce Envelope Encryption with KMS-backed Hardware Security Modules to protect customer data at rest.
Was this lesson helpful? Spot an issue or have a suggestion? Let us know on GitHub.
Report an issue