Realm Sync - The Details

Last updated 11 days ago

Sync Modes

When opening the realm, you can choose to only replicate out a subset of the objects specified with a set of queries (Query Based Sync), or you can use full sync which circumvents the overhead of doing queries and just replicates the entire realm (Full Sync).

It is no problem to open both query-based reference realms and fully-synced realms within the same app. Thus it is possible to mix and match the types of realms you use on any single device to adapt to the use-case. The only restriction is that each realm has to be designated as either a reference or full realm on creation and will stay as such from then on (a full realm can later be upgraded to a reference realm though).

Sync Details

Realm Sync is designed to empower offline-first use cases with shared data between users in disparate locations. Each user or application will have their own copy of the realm sent from the master copy on the Realm Object Server (self-hosted, or Realm Cloud). Because users can be editing the same data when they’re offline at the same or different times, Realm Sync is designed with a robust conflict resolution engine that is guaranteed to always deterministically converge to the same result, both on the server and all clients. In this way Realm Sync is strongly eventually consistent. In order to achieve this, Realm Sync works by keeping a running log of all transactions that occur on a realm and then sending these logs to the Realm Object Server. Once the logs are sent to the Realm Object Server, the Realm Sync merge algorithm is triggered, which uses vector clocks and timestamps to infer causality and place the master transaction log of that realm in the correct order. In the simplest example of the merge algorithm the sync-client sends a transaction that appends an item to the list, the sync-server receives the transaction and sees that the sync-clients list state is the same as its master list state - an append-only operation. In the most computationally expensive use case, multiple sync-clients have been working offline for a long time, all editing the same list of objects, deleting, inserting, and changing the order of the list. The sync-server must compare timestamps and causality of each transaction from each sync-client to determine the final state.

Transaction logs are created by interacting with objects within each Realm SDKs’ write transaction API. Realm Sync is always available and as such when a developer performs a write transaction the state of the realm is changed in an ACID compliant way and a transaction log entry is generated to account for this change. A single transaction can contain one or many different instructions to the realm database which are then translated to what we call changesets. The Adapter API emits instructions that correspond to these changeset log entries - you can see a list here: build-your-own-adapter

These changesets are then sent up to sync-server or down to the sync-client in a transactional manner. While the sync protocol may split up large transactions into chunks simply to make the network transfer of data more efficient, the receiving side will hold all the changesets in memory until the full transaction is received before committing to disk, in this way, atomicity is preserved. Once the transaction is received by the sync-server, the merge algorithm applied to create the master transaction log of the realm, and committed to disk - the sync-client will then receive acknowledgement that it can then proceed to the next transaction. The sync-client will then prune the instructions that made up the previous transaction and iterate its history version to match the version now present on the sync-server - this preserves disk space while still maintain the current state. On the other hand, the sync-server needs to preserve the master record of all instructions that have occurred since the realm was created in case an old offline client comes back online that needs to have its transactions merged. Over time your realm on the sync-server will grow as your app generates transactions. Keep in mind that Instructions are sent and received in the order they were emitted which can lead to slower performance for new clients that connect to an old realm, because they are downloading all of the instructions that may have occurred during the lifetime of that realm. This is why it is important to regularly prune your transaction logs with the sync-servers’ historyTTL setting which will trim superfluous transaction logs. For Realm Cloud the TTL is set to 30 days by default. Additionally, using query-based sync also generates a synthetic transaction to match the results of your query at that moment in time thus eliminating any extra changesets that occurred along the path of getting to that present state.