Design: a file sync service

Designing a File Sync Service (Dropbox/Google Drive)

A file-sync service carries changes between a remote folder and devices that may be offline. It has to move the right bytes, distinguish an edit from a retry, and preserve work when two devices change the same file.

The core challenge is minimizing bandwidth and ensuring consistency across multiple devices (laptops, phones) that might go offline and come back online at random times.

Block-Level Sync (Chunking)

If a user changes one line in a 100MB file, uploading the whole file may be inefficient. A client can hash chunks and upload only missing content. Fixed 4MB boundaries work well for aligned replacements, but an insertion near the beginning shifts later boundaries and can make most blocks appear new; content-defined chunking trades more CPU for boundary stability.

This is called block-level sync. The backend object storage doesn't store files as monolithic blobs; it stores these 4MB chunks, named by their cryptographic hash (Content-Addressable Storage). The metadata database maps a file version to an ordered list of chunk hashes.

Deduplication

Content hashes make deduplication possible, but global cross-user deduplication is a security and lifecycle choice rather than an automatic consequence. The service must authorize every reference, avoid revealing whether another account owns a guessed hash, count references safely, and delete a block only after no retained version is entitled to it. Per-tenant deduplication is a simpler isolation boundary.

The Architecture

A typical architecture splits metadata and block storage:

Synchronization Protocol

Clients need to know when files change on the server. Polling the server every second drains battery and wastes server resources. Instead, clients maintain a long-lived connection (like WebSockets or Long Polling) to a Notification Service.

When Laptop A uploads a change, it updates the Metadata DB. The Metadata Service publishes an event to the Notification Service, which instantly pings Laptop B: "There is a new version of 'document.docx'". Laptop B then requests the new metadata and downloads only the missing chunks from the Block Servers.

Handling Conflicts

If Laptop A and Laptop B both edit the same file while offline, and then both come online, we have a conflict. The system cannot merge the files itself (it doesn't know the business logic of a Word document). The standard approach is to accept both files, creating a "Conflicted Copy" of the second file to arrive, and relying on the user to resolve it.

Worked example: editing a 100 MB file

• The file is split into 25 chunks of 4 MB. Dropbox has used 4 MB blocks with SHA-256 hashes.
• The user appends 1 MB to the end. Only the last chunk changes (and possibly one new chunk is added), so the client uploads about 4 to 5 MB instead of 100 MB.
• The user inserts 1 KB near the start. With fixed chunks every boundary shifts, so all 25 hashes change and about 100 MB is uploaded again. With content-defined chunking (a rolling hash picks boundaries where the hash matches a pattern, averaging about 4 MB), only the chunk containing the insertion changes, so about 4 MB is uploaded.
• The metadata commit is a small request: file_id, base_rev 17, new block list [h1, h2, ...]. If the server's current revision is not 17, the commit is rejected and the client handles it as a conflict.

Chunking and sync choices

Sync protocol

Commit the metadata after the blocks are durable, so no device ever sees a version whose blocks are missing. Blocks that are uploaded but never committed are removed later by a garbage collector.