The Problem with Monolithic Video Uploads
Uploading a 4GB video over a mobile or residential connection can take 30 minutes. If a standard single-part HTTP upload fails at 98%, the entire upload must restart from byte 0. S3 Multipart Upload solves this by breaking files into independent chunks.
S3 Multipart Upload Lifecycle
flowchart TD
Client["Client App"] -->|1. Initiate Multipart| API["App Server API"]
API -->|2. CreateMultipartUpload| S3[("Amazon S3")]
S3 -->|3. Return UploadId| API
API -->|4. Return UploadId + Part Size 10MB| Client
Client -->|5. Parallel Upload Part 1, 2, 3...| S3
S3 -->|6. Return ETag per Part| Client
Client -->|7. CompleteMultipartUpload with ETags| API
API -->|8. Finalize| S3
Resilience & Parallelism
- Parallel Uploads: Clients upload multiple 10MB parts simultaneously across parallel TCP streams, saturating available bandwidth.
- Resumable Retries: If part 42 fails, only part 42 is retried. Successfully uploaded parts remain in object storage (and are billed until the upload is completed or aborted; parts must be at least 5 MiB except the last, with at most 10,000 parts).
Worked example: a 4 GB upload over a flaky connection
- File size 4 GiB = 4,096 MiB. With 10 MiB parts that is 410 parts, well under the 10,000 limit. A rule of thumb: part size = max(8 MiB, ceil(file size / 10,000)).
- At 20 Mbit/s total uplink, the upload takes about 28 minutes. With 4 parallel streams, each part takes about 16 seconds.
- If each part has a 1 percent chance of failing, we expect about 4 part retries, costing about a minute, instead of restarting a 28 minute upload.
- If the app is killed at part 300, the client stores (UploadId, part size, completed part numbers and ETags) locally; on restart it calls ListParts to confirm what the server has and uploads only the remaining 110 parts.
Signed multipart flow
- Client asks the API to start an upload. The API calls CreateMultipartUpload with a server-chosen key and returns the UploadId.
- The client requests presigned UploadPart URLs in batches (each URL is bound to the UploadId and a part number and expires, for example, after an hour).
- The client PUTs parts in parallel and records each ETag; optionally it sends a checksum per part (S3 supports CRC32, CRC32C, SHA-1 and SHA-256 checksums) so corruption is caught per part.
- The client sends the ordered part list to the API, which calls CompleteMultipartUpload; S3 assembles the object atomically.
- An object-created event triggers validation and transcoding.
Options compared
| Protocol | Resume | Parallelism | Notes |
|---|---|---|---|
| Single presigned PUT | No; restart from byte 0 | No | Simple; S3 caps single PUTs at 5 GB |
| S3 multipart with presigned parts | Per part | Yes | Part limits: 5 MiB to 5 GiB, max 10,000 parts |
| GCS resumable upload | From last committed byte offset | Sequential per session (use composite objects for parallel) | One session URI per upload |
| tus open protocol | From offset, server-agnostic | Via concatenation extension | Good when you run your own upload servers |
Failure modes
- Presigned part URLs expiring mid-upload: request fresh URLs for remaining parts rather than restarting.
- Completing with a wrong or missing ETag fails the completion; keep the part list authoritative on the client and verify with ListParts.
- Duplicate completion calls are harmless if the API treats the upload record as idempotent by UploadId.