The Dropbox / Google Drive System Design
A cloud file synchronization service must synchronize files across multiple devices with minimal bandwidth usage, handle network drops gracefully, and resolve concurrent file modifications without data loss.
Client-Server Sync Architecture
flowchart TD
Client["Local Client Daemon"] --> Watcher["File Watcher: inotify / FSEvents"]
Watcher --> ChunkEngine["Chunking Engine: 4MB Chunks"]
ChunkEngine --> LocalDB[("SQLite Local Metadata Cache")]
ChunkEngine --> SyncService["Cloud Sync Gateway API"]
SyncService --> MetaDB[("Metadata Store")]
SyncService --> BlockStore[("Content-Addressed Chunk Store: S3")]
SyncService --> Notifier["Notification Service: Long Polling / WebSockets"]
Notifier --> OtherClients["Other Synchronized Devices"]
Delta Sync & Content Addressing
When a 500MB video or presentation changes by a few paragraphs, uploading the entire 500MB file is unacceptably slow. The client splits files into 4MB chunks and hashes each chunk with SHA-256. When a file is modified, the client re-chunks it, compares hashes with its local SQLite database, and only uploads the specific 4MB chunks that were added or changed. This works well for in-place edits and appends; an insertion near the start shifts every later fixed-size block boundary (the boundary-shift problem in the next unit).
Worked example: editing a 500 MB file
- The file is 500 MB, so with 4 MiB blocks it has about 120 blocks, each identified by its SHA-256.
- The user edits a slide in place; after re-chunking, 2 blocks have new hashes.
- The client asks the server which of the 120 hashes it lacks; the server answers with the 2 new ones. The client uploads 8 MB (under 2 percent of the file).
- The client commits: path /deck.pptx, base version 17, new blocklist. The server checks that the current version is still 17, writes version 18 to the namespace journal, and notifies watchers.
- A laptop long-polling on the namespace wakes up, lists changes since its cursor, sees version 18, downloads the 2 blocks it lacks, and rebuilds the file locally.
Conflict handling
| Strategy | Behaviour | Pros | Cons |
|---|---|---|---|
| Last writer wins | Newest commit replaces the file | Simple | Silent data loss |
| Conflicted copy | Second writer's version saved as a separate file | No data loss, easy to understand | User must merge manually |
| Lock or check-out | Only one editor at a time | No conflicts | Breaks offline editing |
| Operational merge (OT or CRDT) | Edits merged automatically | Best experience for documents | Only for structured formats, not arbitrary binaries |
Sync protocol
sequenceDiagram
participant C as "Client A"
participant M as "Metadata service"
participant B as "Block store"
participant D as "Client B"
C->>M: "which of these hashes are missing"
M-->>C: "hashes 57 and 58"
C->>B: "upload blocks 57 and 58"
C->>M: "commit path, base v17, blocklist"
M-->>C: "ok, v18"
M-->>D: "long poll returns, changes available"
D->>M: "list changes since cursor"
D->>B: "download missing blocks"
How real systems do it
- Dropbox splits files into 4 MB blocks hashed with SHA-256, stores blocks in its Magic Pocket system, and exposes a cursor-based list-changes API with a long-poll endpoint for notifications. On conflicting edits it creates a conflicted copy rather than overwriting.
- Dropbox rewrote its desktop sync engine (Nucleus) in Rust around 2020, with a strong emphasis on testing the sync state machine for correctness.