How efficient can Meteor be while sharing a huge collection among many clients ?
Here is an solution,
There are three important parts of the Meteor server that manage subscriptions: the publish function, which defines the logic for what data the subscription provides; the Mongo driver, which watches the database for changes; and the merge box, which combines all of a client’s active subscriptions and sends them out over the network to the client.
1. Publish functions
Most publish functions don’t have to muck around with the low-level added, changed and removed API, though. If a publish function returns a Mongo cursor, the Meteor server automatically connects the output of the Mongo driver (insert, update, and removed callbacks) to the input of the merge box (this.added, this.changed and this.removed). It’s pretty neat that you can do all the permission checks up front in a publish function and then directly connect the database driver to the merge box without any user code in the way. And when autopublish is turned on, even this little bit is hidden: the server automatically sets up a query for all documents in each collection and pushes them into the merge box.
On the other hand, you aren’t limited to publishing database queries. For example, you can write a publish function that reads a GPS position from a device inside a Meteor.setInterval, or polls a legacy REST API from another web service. In those cases, you’d emit changes to the merge box by calling the low-level added, changed and removed DDP API.
2. The Mongo driver
The Mongo driver’s is to watch the Mongo database for changes to live queries. These queries run continuously and return updates as the results change by calling added, removed, and changed callbacks.
Mongo is not a real time database. So the driver polls. It keeps an in-memory copy of the last query result for each active live query. On each polling cycle, it compares the new result with the previous saved result, computing the minimum set of added, removed, and changed events that describe the difference. If multiple callers register callbacks for the same live query, the driver only watches one copy of the query, calling each registered callback with the same result.
Each time the server updates a collection, the driver recalculates each live query on that collection Future versions of Meteor will expose a scaling API for limiting which live queries recalculate on update. The driver also polls each live query on a 10 second timer to catch out-of-band database updates that bypassed the Meteor server.
3. The merge box
The job of the merge box is to combine the results added, changed and removed calls of all of a client’s active publish functions into a single data stream. There is one merge box for each connected client. It holds a complete copy of the client’s minimongo cache.
In your example with just a single subscription, the merge box is essentially a pass-through. But a more complex app can have multiple subscriptions which might overlap. If two subscriptions both set the same attribute on the same document, the merge box decides which value takes priority and only sends that to the client. We haven’t exposed the API for setting subscription priority yet. For now, priority is determined by the order the client subscribes to data sets. The first subscription a client makes has the highest priority, the second subscription is next highest, and so on.
Because the merge box holds the client’s state, it can send the minimum amount of data to keep each client up to date, no matter what a publish function feeds it.
The meteor server has to keep a copy of the published data for each client in the merge box. This is what allows the Meteor magic to happen, but also results in any large shared databases being repeatedly kept in the memory of the node process. Even when using a possible optimization for static collections such as in there a way to tell meteor a collection is static experienced a huge problem with the CPU and Memory usage of the Node process.
In our case, publishing a collection of 15k documents to each client that was completely static. The problem is that copying these documents to a client’s merge box in memory upon connection basically brought the Node process to 100% CPU for almost a second, and resulted in a large additional usage of memory. This is inherently unscalable, because any connecting client will bring the server to its knees and memory usage will go up linearly in the number of clients. In our case, each client caused an additional ~60MB of memory usage, even though the raw data transferred was only about 5MB.
In our case, because the collection was static, we solved this problem by sending all the documents as a .json file, which was gzipped by nginx, and loading them into an anonymous collection, resulting in only a ~1MB transfer of data with no additional CPU or memory in the node process and a much faster load time. All operations over this collection were done by using _ids from much smaller publications on the server, allowing for retaining most of the benefits of Meteor. This allowed the app to scale to many more clients. In addition, because our app is mostly read-only, we further improved the scalability by running multiple Meteor instances behind nginx with load balancing though with a single Mongo, as each Node instance is single-threaded.
However, the issue of sharing large, writeable collections among multiple clients is an engineering problem that needs to be solved by Meteor. There is probably a better way than keeping a copy of everything for each client, but that requires some serious thought as a distributed systems problem. The current issues of massive CPU and memory usage just won’t scale.
Here are the few experimental solutions:
- Install a test meteor: meteor create –example todos.
- Run it under Webkit inspector (WKI).
- Examine the contents of the XHR messages moving across the wire.
- Observe that the entire collection is not moved across the wire.