On-the-Fly Image Processing vs. Batch Pre-Generation
Modern applications display images across hundreds of device resolutions, responsive breakpoints, and formats (WebP, AVIF, JPEG). Pre-generating and storing every dimension permutation multiplies storage costs (sizes x formats x densities). Modern media platforms generate resized variants on-demand at the CDN edge.
Dynamic Image Processing Pipeline
Edge Transformation Mechanics
- Clients request images with query parameters:
/images/profile.jpg?width=400&quality=80&format=webp. - The CDN edge checks if that exact transformed combination is cached. If yes, it returns it instantly from edge cache.
- On cache miss, a lightweight serverless worker (e.g., Cloudflare Workers, AWS Lambda@Edge) fetches the original master image from S3, applies resizing using libvips/Sharp, stores the result in CDN cache, and serves the client.
- Only accept an allowlist of widths and qualities (or signed transform URLs); otherwise arbitrary parameters let anyone force uncached, expensive resizes.
Worked example: pre-generate versus on demand
Ten million originals averaging 2 MB is 20 TB. Suppose the product uses 6 widths x 3 formats (JPEG, WebP, AVIF) x 2 pixel densities = 36 variants averaging 60 KB.
- Pre-generate all: 10M x 36 x 60 KB = about 21.6 TB of extra storage, plus 360 million resize jobs up front, most for images that will never be viewed at that size.
- On demand: if 80 percent of views hit 10 percent of images in 4 popular variants, the working set is about 10M x 0.1 x 4 x 60 KB = 240 GB, which fits comfortably in CDN caches. A resize with libvips of a 12 megapixel JPEG typically takes tens of milliseconds, paid once per variant per shield.
Cache key normalization
Turn /img/abc.jpg?width=611&q=83&fmt=auto into a canonical form: width snapped up to the nearest of 160, 320, 640, 960, 1280, 1920; quality to one of three presets; format resolved from Accept to avif, webp or jpeg; parameters sorted. The canonical string is the cache key, so equivalent requests share one cached object.
Trade-offs
| Approach | Storage | First-view latency | Operational risk |
|---|---|---|---|
| Pre-generate every variant | Highest | Lowest | Backfills needed when you add a size or format |
| On demand at the edge | Lowest | Resize cost on first miss | Cache-busting abuse if parameters are open |
| Hybrid: pre-generate the 2 or 3 hottest sizes, rest on demand | Moderate | Low for common views | Two code paths to keep consistent |
How real systems do it
- Cloudflare Image Resizing, imgix and Cloudinary expose URL-parameter transforms at the edge.
- Thumbor is a widely used open-source resize server, and AWS publishes a Serverless Image Handler reference architecture using CloudFront, Lambda and Sharp.
- Use Cache-Control with a long max-age on variants and version originals by key, so a replaced image gets a new URL instead of needing a purge.