Why Avoid Routing Uploads Through Application Servers?
In naive architectures, users upload photos or video files directly to an application server, which buffers the file in memory or temporary disk before uploading it to object storage (like Amazon S3 or Google Cloud Storage). This anti-pattern exhausts server network bandwidth, ties up worker threads during slow mobile uploads, and concentrates network, connection and memory pressure on the API tier.
The Pre-Signed URL Architecture
How Pre-Signed URLs Work
The application server uses AWS IAM credentials to generate a cryptographically signed HMAC-SHA256 signature containing:
- The exact S3 object key (e.g.,
uploads/users/42/avatar.jpg). - Allowed HTTP method (strictly
PUT). - Expiration timestamp (e.g., 15 minutes).
- Optional signed headers such as an exact
Content-Type. A pre-signed S3 PUT does not provide a size range (signing Content-Length pins one exact length, nothing more); use a pre-signed POST with acontent-length-rangecondition where appropriate, and verify the stored object's size before publishing it.
The client uses this URL to stream the binary payload directly to S3. The application server never touches the file bytes, scaling effortlessly to hundreds of thousands of concurrent uploads.
Worked example: why proxying bytes hurts
With 50,000 concurrent mobile uploads averaging 5 Mbit/s, proxying through your API means about 250 Gbit/s of ingress and egress on application servers. A 20 MB photo on a 2 Mbit/s uplink takes about 80 seconds, holding a worker, a socket and buffer memory the whole time. With direct upload, the API handles one small signing request (a few milliseconds of CPU) and one completion callback per file.
Upload flow in detail
- Client calls
POST /uploadswith intended type and size. The server checks quota and auth, creates a metadata row with state pending, and picks keypending/u42/9f1c...jpg. - The server signs a presigned POST policy with conditions: exact key, content-length-range from 1 byte to 25 MB, Content-Type starting with image/, expiry 15 minutes. Signing is a local HMAC computation with the server's credentials; no call to S3 is needed.
- The client uploads directly to the bucket.
- An object-created event goes to a queue; a worker checks the real size and magic bytes, scans it, strips EXIF location data, generates thumbnails, and copies it to its final key.
- The worker marks the metadata row ready; only now can the object be referenced by posts or profiles.
Trade-offs
| Method | Size enforcement | Resumable | Server load | Best for |
|---|---|---|---|---|
| Proxy through app server | Full control | Only if you build it | High: all bytes | Tiny files, strict inline validation |
| Presigned PUT | Exact length only if Content-Length is a signed header; otherwise verify after | No | Minimal | Simple uploads up to a few hundred MB |
| Presigned POST with policy | content-length-range condition enforced by S3 | No | Minimal | Browser form uploads with size limits |
| Multipart with presigned parts | Per part, verify total after | Yes, per part | Minimal; one signing call per part | Large files and flaky networks |
Equivalents in other clouds
- Google Cloud Storage offers signed URLs and a resumable upload protocol.
- Azure Blob Storage uses shared access signature (SAS) tokens scoped to a blob, permissions and expiry.
- If the URL is created with temporary credentials (for example an IAM role session), it stops working when those credentials expire, even if its own expiry is later.