Warmup
When running Metarank in production, you may hit the cold-start problem:
after a restart (due to redeployment/autoscaling) Metarank pod starts cold with empty caches,
JVM also has not yet compiled most of the code,
due to this, reranking latency of first N requests is too high.
To handle this Metarank (starting from v0.7.6) supports explicit API warmup:
while training, sample N random but real reranking requests and persist them in store
on API startup, replay this random traffic sample
when replay done, bring up the API - so k8s readiness probe will be successful.
Configuring warmup
API warmup is only supported for LambdaMART models and is configured per-model:
models:
my-super-model:
type: lambdamart
warmup:
sampledRequests: 100 # how many requests sample during training
duration: 5s # how long to replay the traffic during warmup
# ...Warmup is disabled by default, and you need to retrain your model if you trained it in Metarank prior to 0.7.6.
After the warmup is enabled, you will see the following log output when starting the API:
Last updated
Was this helpful?