Performance
Metarank is a secondary re-ranker: it's an extra non-free step in your retrieval process. In a typical scenario, you should expect the following extras:
Re-ranking latency: 10-30 ms
Redis memory usage: 1-10 GiB
Data import throughput: 1000-3000 events/second.
Response latency
On a RankLens dataset in a synthetic latency test, we observed the following distribution:
Each Metarank installation is unique, but there are common things affecting the overall latency:
State encoding format:
binary
is faster thanjson
due to its compact representation.Metarank-Redis network latency: Metarank pulls all features for all re-ranked items in a single large batch. There are no multiple network calls and only a constant overhead.
Request size: the more items you ask to re-rank, the more data needs to be loaded.
A number of feature extractors: the more per-item features are defined in the config, the more data is loaded during the request processing.
So while planning your installation, expect Metarank to be within 20-30 ms latency budget.
Memory usage
Using the same reference RankLens dataset, we built a fuzzy synthetic dataset generator and generated the following dataset variations:
N users, 100k items.
Each user made 2 rankings within a single session.
Each ranking event has 2 clicks made by the user.
Metarank only tracks aggregated data required for the re-ranking and does not store raw events. Therefore, memory usage depends on the following characteristics of your dataset:
A number of unique users: per-user click-through events are used as input for the ML model training.
A number of users: if you define per-user features like
interacted_with
, we persist a per-user list of interacted items.A number of features: stored click-through events also contain snapshots of all per-item feature values used to build the ranking back in time.
The resulting memory usage for a binary
state encoding format and Redis
as a persistence store is shown in the diagram below:
So while planning your installation, expect Metarank to use around 0.8 GiB per 1M users.
Last updated