# What is Metarank? [Metarank](https://metarank.ai) is an open-source ranking service. It can help you to build a personalized semantic/neural search and recommendations. If you just want to get started, try: * the [quickstart](https://docs.metarank.ai/introduction/quickstart) tutorial of implementing Learning-to-Rank on top of your search engine. * a [guide on using cross-encoder LLM for search reranking](broken://pages/tshwWIN8Y3BS5WQs2cyU) of building an LLM-based neural search. * a [Collaborative Filtering recommendations guide](https://github.com/metarank/metarank/blob/stabledoc/doc/TODO/README.md) to create a "you may also like" widget as seen on many e-commerce stores. ## Why Metarank? With Metarank, you can make your existing search and recommendations **smarter**: * Integrate customer signals like clicks and purchases into the ranking - and optimize for maximal CTR! * Track [visitor profile](https://docs.metarank.ai/reference/overview/feature-extractors/user-session) and make search results adapt to user actions with real-time personalization. * Use [LLMs in bi- and cross-encoder mode](https://docs.metarank.ai/reference/overview/feature-extractors/text) to make your search understand the true meaning of search queries. Metarank is **fast**: * optimized for reranking latency, it can handle even large result sets within 10-20ms. See [benchmarks](https://docs.metarank.ai/introduction/performance). * as a stateless cloud-native service (with state managed by Redis), it can scale horizontally and process thousands of RPS. See [Kubernetes deployment guide](https://docs.metarank.ai/reference/deployment-overview/kubernetes) for details. Save your **development time**: * Metarank can compute dozens of typical ranking signals out of the box: CTR, referer, User-Agent, time, etc - you don't need to write custom ad-hoc code for most common ranking factors. See [the full list of supported ranking signals](https://docs.metarank.ai/reference/overview/feature-extractors) in our docs. * There are integrations with many possible streaming processing systems to ingest visitor signals: See [data sources](https://docs.metarank.ai/reference/overview/data-sources) for details. ## What can you build with Metarank? Metarank helps you build advanced ranking systems for search and recommendations: * Semantic search: use state-of-the-art LLMs to make your Elasticsearch/OpenSearch understand the meaning of your queries * Recommendations: traditional collaborative-filtering and new-age semantic content recommendations. * Learning-to-Rank: optimize your existing search ## Content Blog posts: * [Learn-to-Rank with OpenSearch and Metarank](https://opensearch.org/blog/ltr-with-opensearch-and-metarank/) * [Hybrid Search and Learning-to-Rank with Metarank](https://www.pinecone.io/learn/metarank/) * [Solving a search cold-start problem with aggregated CTR](https://blog.metarank.ai/solving-a-search-cold-start-problem-with-aggregated-ctr-b88c14f4d03c) * [Personalized search with Metarank and Elasticsearch](https://blog.metarank.ai/personalized-search-with-metarank-and-elasticsearch-a5a098548da7) Meetups and conference talks: * [Building an open-source online Learn-to-rank engine](https://www.youtube.com/watch?v=lbbp4CFWZGk), Haystack EU 23, [slides](https://metarank.github.io/haystack-eu22/#/) * [Overcoming position and presentation biases in search and recommender systems](https://www.youtube.com/watch?v=PqbYdDiwKBY), Data Natives Meetup Berlin, [slides](https://metarank.github.io/bias-talk/#/) * [Learning-to-rank: Deep, fast, precise - choose any two](https://www.youtube.com/watch?v=oXfFqAKf4Ac), DataTalks meetup, [slides](https://metarank.github.io/datatalks-ltr-talk/#/) ## Main features * Semantic neural search: \[TODO] * Recommendations: [trending](/reference/overview/recommendations/trending) and [similar-items](/reference/overview/recommendations/similar) (MF ALS). * Personalization: [secondary reranking](/introduction/quickstart) (LambdaMART) * AutoML: [automatic feature generation](/how-to/autofeature) and [model re-training](/how-to/model-retraining) * A/B testing: [multiple model serving](/reference/overview#models) ## Demo You can play with Metarank demo on [demo.metarank.ai](https://demo.metarank.ai): ![Demo](/files/sYoy4ImekwoQjkokC51k) The demo itself and [the data used](https://github.com/metarank/msrd) are open-source and you can grab a copy of training events and config file [in the github repo](https://github.com/metarank/metarank/tree/master/src/test/resources/ranklens). ## Metarank in One Minute Let us show how you can start personalizing content with LambdaMART-based reranking in just under a minute: 1. Prepare the data: we will get the dataset and config file from the [demo.metarank.ai](https://demo.metarank.ai) 2. Start Metarank in a standalone mode: it will import the data, train the ML model and start the API. 3. Send a couple of requests to the API. ### Step 1: Prepare data We will use the [ranklens dataset](https://github.com/metarank/ranklens), which is used in our [Demo](https://demo.metarank.ai), so just download the data file ```bash curl -O -L https://github.com/metarank/metarank/raw/master/src/test/resources/ranklens/events/events.jsonl.gz ``` ### Step 2: Prepare configuration file We will again use the configuration file from our [Demo](https://demo.metarank.ai). It utilizes in-memory store, so no other dependencies are needed. ```bash curl -O -L https://raw.githubusercontent.com/metarank/metarank/master/src/test/resources/ranklens/config.yml ``` ### Step 3: Start Metarank! With the final step we will use Metarank’s `standalone` mode that combines training and running the API into one command: ```bash docker run -i -t -p 8080:8080 -v $(pwd):/opt/metarank metarank/metarank:latest standalone --config /opt/metarank/config.yml --data /opt/metarank/events.jsonl.gz ``` You will see some useful output while Metarank is starting and grinding through the data. Once this is done, you can send requests to `localhost:8080` to get personalized results. Here we will interact with several movies by clicking on one of them and observing the results. > First, let's see the initial output provided by Metarank without before we interact with it ```bash # get initial ranking for some items curl http://localhost:8080/rank/xgboost \ -d '{ "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"67197"}, {"id":"77561"}, {"id":"68358"}, {"id":"79132"}, {"id":"103228"}, {"id":"72378"}, {"id":"85131"}, {"id":"94864"}, {"id":"68791"}, {"id":"93363"}, {"id":"112623"} ], "user": "alice", "session": "alice1", "timestamp": 1661431886711 }' # {"item":"72998","score":0.9602446652021992},{"item":"79132","score":0.7819134441404151},{"item":"68358","score":0.33377910321385645},{"item":"112623","score":0.32591281190727805},{"item":"103228","score":0.31640256043322723},{"item":"77561","score":0.3040782705414116},{"item":"94864","score":0.17659007036183608},{"item":"72378","score":0.06164568676567339},{"item":"93363","score":0.058120639770243385},{"item":"68791","score":0.026919880032451306},{"item":"85131","score":-0.35794106000271037},{"item":"67197","score":-0.48735167237049154} ``` ```bash # tell Metarank which items were presented to the user and in which order from the previous request # optionally, we can include the score calculated by Metarank or your internal retrieval system curl http://localhost:8080/feedback \ -d '{ "event": "ranking", "fields": [], "id": "test-ranking", "items": [ {"id":"72998","score":0.9602446652021992},{"id":"79132","score":0.7819134441404151},{"id":"68358","score":0.33377910321385645}, {"id":"112623","score":0.32591281190727805},{"id":"103228","score":0.31640256043322723},{"id":"77561","score":0.3040782705414116}, {"id":"94864","score":0.17659007036183608},{"id":"72378","score":0.06164568676567339},{"id":"93363","score":0.058120639770243385}, {"id":"68791","score":0.026919880032451306},{"id":"85131","score":-0.35794106000271037},{"id":"67197","score":-0.48735167237049154} ], "user": "test2", "session": "test2", "timestamp": 1661431888711 }' ``` > Now, let's intereact with the items `93363` ```bash # click on the item with id 93363 curl http://localhost:8080/feedback \ -d '{ "event": "interaction", "type": "click", "fields": [], "id": "test-interaction", "ranking": "test-ranking", "item": "93363", "user": "test", "session": "test", "timestamp": 1661431890711 }' ``` > Now, Metarank will personalize the items, the order of the items in the response will be different ```bash # personalize the same list of items # they will be returned in a different order by Metarank curl http://localhost:8080/rank/xgboost \ -d '{ "event": "ranking", "fields": [], "id": "test-personalized", "items": [ {"id":"72998"}, {"id":"67197"}, {"id":"77561"}, {"id":"68358"}, {"id":"79132"}, {"id":"103228"}, {"id":"72378"}, {"id":"85131"}, {"id":"94864"}, {"id":"68791"}, {"id":"93363"}, {"id":"112623"} ], "user": "test", "session": "test", "timestamp": 1661431892711 }' # {"items":[{"item":"93363","score":2.2013986484185124},{"item":"72998","score":1.1542776301073876},{"item":"68358","score":0.9828904282341605},{"item":"112623","score":0.9521647429731446},{"item":"79132","score":0.9258841742518286},{"item":"77561","score":0.8990921381835769},{"item":"103228","score":0.8990921381835769},{"item":"94864","score":0.7131600718467729},{"item":"68791","score":0.624462038351694},{"item":"72378","score":0.5269765094008626},{"item":"85131","score":0.29198666089255343},{"item":"67197","score":0.16412780810560743}]} ``` ## What's next? Check out a more in-depth [Quickstart](/introduction/quickstart) and full [Reference](/reference/installation). If you have any questions, don't hesitate to join our [Slack](https://metarank.ai/slack)! # Quickstart This guide shows how to install and run Metarank on a single machine using Docker. We will run the service, feed it with sample data and issue queries. ## Prerequisites * Docker: [Docker Desktop for Mac/Windows](https://docs.docker.com/engine/install/), or Docker for Linux * Operating system: Linux, macOS, or Windows 10+ * Architecture: x86\_64. For M1, see [Apple M1 support](/reference/installation#installing-on-macos) * Memory: 2Gb dedicated to Docker This guide is tested with Docker for linux v20.10.16, and [metarank/metarank:0.5.1](https://hub.docker.com/r/metarank/metarank/tags) docker image. ## Getting the dataset For the quickstart, we will use an open [RankLens](https://github.com/metarank/ranklens) dataset and personalize a set of pre-computed movie recommendations based on visitor activity. The dataset is used to build a [Metarank Demo](https://demo.metarank.ai/) website and includes the following event types:
Movie metadata: genres, actors, tags, votes. ```json { "id": "b4951b85-a87f-4fdc-b2af-9ff06783def4", "item": "3114", "timestamp": "1636907100000", "fields": [ {"name": "title", "value": "Toy Story 2"}, {"name": "popularity", "value": 112.767}, {"name": "vote_avg", "value": 7.6}, {"name": "vote_cnt", "value": 11025.0}, {"name": "budget", "value": 9.0E+7}, {"name": "runtime", "value": 92.0}, {"name": "release_date", "value": 9.412416E+8}, {"name": "genres", "value": ["animation", "comedy", "family"]}, {"name": "tags", "value": ["pixar", "disney", "animation", "sequel"]}, {"name": "actors", "value": ["tom hanks", "joan cusack", "frank welker"]}, {"name": "director", "value": "john lasseter"}, {"name": "writer", "value": "andrew stanton"} ], "event": "item" } ```
Visitor impressions: what was displayed to a visitor ```json { "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"67197"}, {"id":"77561"}, {"id":"68358"}, {"id":"72378"}, {"id":"85131"}, {"id":"94864"}, {"id":"68791"}, {"id":"109487"}, {"id":"59315"}, {"id":"120466"}, {"id":"90405"}, {"id":"117529"}, {"id":"130490"}, {"id":"92420"}, {"id":"122882"}, {"id":"113345"}, {"id":"2571"}, {"id":"122900"}, {"id":"88744"}, {"id":"95875"}, {"id":"60069"}, {"id":"2021"}, {"id":"135567"}, {"id":"122902"}, {"id":"104243"}, {"id":"112852"}, {"id":"102880"}, {"id":"96610"}, {"id":"741"}, {"id":"166528"}, {"id":"164179"}, {"id":"71057"}, {"id":"3527"}, {"id":"6365"}, {"id":"6934"}, {"id":"114935"}, {"id":"8810"}, {"id":"173291"}, {"id":"1580"}, {"id":"1917"}, {"id":"135569"}, {"id":"106920"}, {"id":"1240"}, {"id":"85056"}, {"id":"780"}, {"id":"1527"}, {"id":"5459"}, {"id":"8644"}, {"id":"60684"}, {"id":"7254"}, {"id":"44191"}, {"id":"97752"}, {"id":"2628"}, {"id":"541"}, {"id":"106002"}, {"id":"2012"}, {"id":"79357"}, {"id":"6283"}, {"id":"113741"}, {"id":"27660"}, {"id":"34048"}, {"id":"1882"}, {"id":"1748"}, {"id":"34319"}, {"id":"1097"}, {"id":"115713"}, {"id":"2916"} ], "user": "alice", "session": "alice1", "timestamp": 1661345221008 } ```
Visitor interactions: which movies the visitor liked after observing the ranking. ```json { "id": "580a09e9-a002-4d59-a527-a556a38aa04f", "item": "4002", "timestamp": "1636993839000", "ranking": "84074af6-25fb-4791-81da-2f622871b194", "user": "90df34e521cc3d53af5f42f5c16ecb60", "session": "90df34e521cc3d53af5f42f5c16ecb60", "type": "click", "fields": [], "event": "interaction" } ```
For this quickstart, you need two files from the dataset: 1. [config.yml](https://raw.githubusercontent.com/metarank/metarank/master/src/test/resources/ranklens/config.yml) - metarank configuration file used in the [demo.metarank.ai](https://demo.metarank.ai), describing how to map visitor events to ML features. For your own dataset, you don't always need to write this file from scratch, Metarank can automatically try to deduce most typical feature mappings based on your dataset. See [Autofeature](/how-to/autofeature) for details. 2. [events.jsonl.gz](https://github.com/metarank/metarank/blob/master/src/test/resources/ranklens/events/events.jsonl.gz) - a dump of historical visitor interactions used for ML training. ```bash curl -o config.yml https://raw.githubusercontent.com/metarank/metarank/master/src/test/resources/ranklens/config.yml ``` ```bash curl -o events.jsonl.gz https://media.githubusercontent.com/media/metarank/metarank/master/src/test/resources/ranklens/events/events.jsonl.gz ``` ```bash ls -l total 172 drwxr-xr-x 2 user user 4096 Aug 23 14:24 . drwxr-xr-x 81 user user 16384 Aug 23 14:24 .. -rw-r--r-- 1 user user 2542 Aug 23 14:24 config.yml -rw-r--r-- 1 user user 150264 Aug 23 14:24 events.jsonl.gz ``` ## Running Metarank in Docker ```bash docker run -i -t -p 8080:8080 -v $(pwd):/opt/metarank metarank/metarank:latest standalone --config /opt/metarank/config.yml --data /opt/metarank/events.jsonl.gz ``` This command will: * run the dataset import process from the current directory, * train the [ML model for ranking](/reference/overview/supported-ranking-models), * start the [API](/reference/api) on port 8080. ![import and training process](/files/HIp3M3rAInwGaN0IgKly) ## First query We're going to send a set of initial candidates for reranking into Metarank's REST API `/rank` endpoint for an `xgboost` model defined in config.yml. Let's take top-100 popular movies tagged as Sci-Fi, and ask Metarank to reorder them to maximize CTR: ```bash curl http://localhost:8080/rank/xgboost -d '{ "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"67197"}, {"id":"77561"}, {"id":"68358"}, {"id":"79132"}, {"id":"103228"}, {"id":"72378"}, {"id":"85131"}, {"id":"94864"}, {"id":"68791"}, {"id":"93363"}, {"id":"112623"}, {"id":"109487"}, {"id":"59315"}, {"id":"120466"}, {"id":"90405"}, {"id":"122918"}, {"id":"70286"}, {"id":"117529"}, {"id":"130490"}, {"id":"92420"}, {"id":"122882"}, {"id":"87306"}, {"id":"82461"}, {"id":"113345"}, {"id":"2571"}, {"id":"122900"}, {"id":"88744"}, {"id":"111360"}, {"id":"134130"}, {"id":"95875"}, {"id":"60069"}, {"id":"2021"}, {"id":"135567"}, {"id":"103253"}, {"id":"111759"}, {"id":"122902"}, {"id":"104243"}, {"id":"112852"}, {"id":"102880"}, {"id":"56174"}, {"id":"107406"}, {"id":"96610"}, {"id":"741"}, {"id":"166528"}, {"id":"164179"}, {"id":"187595"}, {"id":"589"}, {"id":"71057"}, {"id":"3527"}, {"id":"6365"}, {"id":"6934"}, {"id":"1270"}, {"id":"6502"}, {"id":"114935"}, {"id":"8810"}, {"id":"173291"}, {"id":"1580"}, {"id":"182715"}, {"id":"166635"}, {"id":"1917"}, {"id":"135569"}, {"id":"106920"}, {"id":"1240"}, {"id":"5502"}, {"id":"316"}, {"id":"85056"}, {"id":"780"}, {"id":"1527"}, {"id":"5459"}, {"id":"94018"}, {"id":"33493"}, {"id":"8644"}, {"id":"60684"}, {"id":"7254"}, {"id":"44191"}, {"id":"101864"}, {"id":"132046"}, {"id":"97752"}, {"id":"2628"}, {"id":"541"}, {"id":"106002"}, {"id":"1200"}, {"id":"5378"}, {"id":"2012"}, {"id":"79357"}, {"id":"6283"}, {"id":"113741"}, {"id":"90345"}, {"id":"2011"}, {"id":"27660"}, {"id":"34048"}, {"id":"1882"}, {"id":"1748"}, {"id":"2985"}, {"id":"104841"}, {"id":"34319"}, {"id":"1097"}, {"id":"115713"}, {"id":"2916"} ], "user": "alice", "session": "alice1", "timestamp": 1661431892711 }' ``` The API will respond with a list of 100 re-ranked movie ids: ```json5 { "items": [ {"item": "72998", "score": 2.0272045135498047}, {"item": "589", "score": 1.838820457458496}, {"item": "134130", "score": 1.7281458377838135}, {"item": "5459", "score": 1.7237709760665894}, {"item": "1917", "score": 1.7038706541061401}, {"item": "2571", "score": 1.6998087167739868}, {"item": "1527", "score": 1.6812316179275513}, {"item": "97752", "score": 1.6692591905593872}, {"item": "1270", "score": 1.648807406425476}, {"item": "1580", "score": 1.5384368896484375}, {"item": "109487", "score": 1.5244081020355225}, {"item": "79132", "score": 1.4934355020523071}, // other 88 items are skipped ] } ``` Which looks like a diverse set of sci-fi movies with some generic non-personalized ranking, as we haven't sent any interaction events yet. ![sci-fi movies](/files/GRQCO1rt3E3G8D6y39Br) ## Sending visitor feedback Metarank expects to receive impression events (what was displayed to the visitor) and interaction events (what the visitor did after seeing the listing). Impression event contains only the items that were displayed to the user, so if your response is paginated, impression event will indicate only items from the current page. In our case, the impression event is a set of top 12 movies from the previous `/rank` request, starting with `Terminator 2` and ending with `MIIB`: ```bash curl http://localhost:8080/feedback -d '{ "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"589"}, {"id":"134130"}, {"id":"5459"}, {"id":"1917"}, {"id":"2571"}, {"id":"1527"}, {"id":"97752"}, {"id":"1270"}, {"id":"1580"}, {"id":"109487"}, {"id":"79132"} ], "user": "alice", "session": "alice1", "timestamp": 1661431894711 }' ``` Now let's send a click on `Men in Black` with id=1580: ```bash curl http://localhost:8080/feedback -d '{ "event": "interaction", "type": "click", "id": "id2", "ranking": "id1", "item": "1580", "user": "alice", "session": "alice1", "timestamp": 1661431896711 }' ``` ## Getting personalized ranking Now, we are ready to get some personalized! Let's send the same first ranking request with top-100 sci-fi movies we did before, and see how response will change after providing some visitor feedback: ```bash curl http://localhost:8080/rank/xgboost -d '{ "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"67197"}, {"id":"77561"}, {"id":"68358"}, {"id":"79132"}, {"id":"103228"}, {"id":"72378"}, {"id":"85131"}, {"id":"94864"}, {"id":"68791"}, {"id":"93363"}, {"id":"112623"}, {"id":"109487"}, {"id":"59315"}, {"id":"120466"}, {"id":"90405"}, {"id":"122918"}, {"id":"70286"}, {"id":"117529"}, {"id":"130490"}, {"id":"92420"}, {"id":"122882"}, {"id":"87306"}, {"id":"82461"}, {"id":"113345"}, {"id":"2571"}, {"id":"122900"}, {"id":"88744"}, {"id":"111360"}, {"id":"134130"}, {"id":"95875"}, {"id":"60069"}, {"id":"2021"}, {"id":"135567"}, {"id":"103253"}, {"id":"111759"}, {"id":"122902"}, {"id":"104243"}, {"id":"112852"}, {"id":"102880"}, {"id":"56174"}, {"id":"107406"}, {"id":"96610"}, {"id":"741"}, {"id":"166528"}, {"id":"164179"}, {"id":"187595"}, {"id":"589"}, {"id":"71057"}, {"id":"3527"}, {"id":"6365"}, {"id":"6934"}, {"id":"1270"}, {"id":"6502"}, {"id":"114935"}, {"id":"8810"}, {"id":"173291"}, {"id":"1580"}, {"id":"182715"}, {"id":"166635"}, {"id":"1917"}, {"id":"135569"}, {"id":"106920"}, {"id":"1240"}, {"id":"5502"}, {"id":"316"}, {"id":"85056"}, {"id":"780"}, {"id":"1527"}, {"id":"5459"}, {"id":"94018"}, {"id":"33493"}, {"id":"8644"}, {"id":"60684"}, {"id":"7254"}, {"id":"44191"}, {"id":"101864"}, {"id":"132046"}, {"id":"97752"}, {"id":"2628"}, {"id":"541"}, {"id":"106002"}, {"id":"1200"}, {"id":"5378"}, {"id":"2012"}, {"id":"79357"}, {"id":"6283"}, {"id":"113741"}, {"id":"90345"}, {"id":"2011"}, {"id":"27660"}, {"id":"34048"}, {"id":"1882"}, {"id":"1748"}, {"id":"2985"}, {"id":"104841"}, {"id":"34319"}, {"id":"1097"}, {"id":"115713"}, {"id":"2916"} ], "user": "alice", "session": "alice1", "timestamp": 1661431898711 }' ``` As you can see, the response is quite different from the first attempt: ```json5 { "items": [ {"item": "1580", "score": 3.345952033996582}, {"item": "5459", "score": 2.873959541320801}, {"item": "8644", "score": 2.500633478164673}, {"item": "56174", "score": 2.2979140281677246}, {"item": "2571", "score": 2.0133864879608154}, {"item": "1270", "score": 1.807900071144104}, {"item": "109487", "score": 1.7143194675445557}, {"item": "589", "score": 1.706472396850586}, {"item": "780", "score": 1.7030035257339478}, {"item": "1527", "score": 1.6445566415786743}, {"item": "60069", "score": 1.6372750997543335}, {"item": "1917", "score": 1.6299139261245728} // other 88 items skipped ] } ``` ![reranked](/files/X46uLd9IhBYieBYkBQIa) Ranking was adjusted to a taste of the visitor, we can see that: * `Men in Black 2` went to the top, as it has similar tags and actors. * other space movies also went up (like `Armageddon`), as they're also space-related. * both parts of `Back to the Future` went significantly down, as not related to past visitor's clicks. ## What's next? * play with the contents of [config.yml](/reference/overview), enabling and disabling different features and see how ranking changes depending on used features. * generate your own set of [input events](/reference/event-schema), describing your use case. # Performance Metarank is a secondary re-ranker: it's an extra **non-free** step in your retrieval process. In a typical scenario, you should expect the following extras: * Re-ranking latency: 10-30 ms * Redis memory usage: 1-10 GiB * Data import throughput: 1000-3000 events/second. ## Response latency On a [RankLens](https://github.com/metarank/ranklens) dataset in a [synthetic latency test](https://github.com/metarank/metarank/blob/master/src/test/scala/ai/metarank/util/performance/LatencyBenchmark.scala), we observed the following distribution: ![Response latency depends on request size](/files/pOoQ0uHdjhwNWBp6oKF6) Each Metarank installation is unique, but there are common things affecting the overall latency: * [**State encoding format**](/reference/overview/persistence#state-encoding-formats): `binary` is faster than `json` due to its compact representation. * **Metarank-Redis network latency**: Metarank pulls all features for all re-ranked items in a single large batch. There are no multiple network calls and only a constant overhead. * **Request size**: the more items you ask to re-rank, the more data needs to be loaded. * [**A number of feature extractors**](/reference/overview/feature-extractors): the more per-item features are defined in the config, the more data is loaded during the request processing. So while planning your installation, expect Metarank to be within **20-30 ms** latency budget. ## Memory usage Using the same reference [RankLens](https://github.com/metarank/ranklens) dataset, we built a fuzzy [synthetic dataset generator](https://github.com/metarank/metarank/blob/master/src/test/scala/ai/metarank/util/SyntheticRanklensDataset.scala) and generated the following dataset variations: * N users, 100k items. * Each user made 2 rankings within a single session. * Each ranking event has 2 clicks made by the user. | Users | Items | Rankings | Clicks | Total events | Uncompressed size | | ----- | ----- | -------- | ------ | ------------ | ----------------- | | 128K | 100K | 256K | 512K | 896K | 287MiB | | 256K | 100K | 512K | 1M | 1.8M | 512MiB | | 512K | 100K | 1M | 2M | 3.5M | 963MiB | | 1M | 100K | 2M | 4M | 7.1M | 1.8GiB | | 2M | 100K | 4M | 8M | 14.3M | 3.5GiB | | 4M | 100K | 8M | 16M | 28.6M | 7.1GiB | Metarank only tracks aggregated data required for the re-ranking and does not store raw events. Therefore, memory usage depends on the following characteristics of your dataset: * **A number of unique users**: per-user click-through events are used as input for the ML model training. * **A number of items**: if you define per-item feature extractors (like [`string`](/reference/overview/feature-extractors/scalar#string-extractors) or [`number`](/reference/overview/feature-extractors/scalar#numerical-extractor)), current point-in-time values of used fields are persisted. * **A number of users**: if you define per-user features like [`interacted_with`](/reference/overview/feature-extractors/user-session#interacted-with), we persist a per-user list of interacted items. * **A number of features**: stored click-through events also contain snapshots of all per-item feature values used to build the ranking back in time. The resulting memory usage for a `binary` state encoding format and `Redis` as a persistence store is shown in the diagram below: ![Redis/heap memory usage](/files/sAUuCuLnvitlczYISv6K) | Users, thousands | Redis, MB | Import time, min | | ---------------- | --------- | ---------------- | | 128 | 386 | 8 | | 256 | 448 | 16 | | 512 | 561 | 30 | | 1024 | 812 | 61 | | 2048 | 1210 | 110 | | 4096 | 1860 | 205 | So while planning your installation, expect Metarank to use around **0.8 GiB per 1M users**. # Search In this series of guides we will go through typical cases of using Metarank to improve the relevance of your search engine. There are two main approaches to search reranking: * **Zero-shot**: using generic approaches not fine-tuned on your dataset and visitor behavior. This does not require any telemetry collection and is a good starting point. * **Learn-to-Rank**: adapt ranking to the dataset and visitor behavior. Can yield better quality, with the drawback of requiring proper visitor analytics. ## Zero-shot re-ranking If you're not familiar with concepts of re-ranking and Metarank, start with these intro guides to get better understanding about how things work: * [Search re-ranking with cross-encoder LLMs](/guides/index/cross-encoders): How to use a general-purpose cross-encoder, pre-trained on MS-MARCO dataset to improve your Elasticsearch search relevance. * TODO: Semantic search with sentence-transformers and Qdrant: setting up Metarank as an inference server for bi-encoders for semantic retrieval with vector search. ## Learn-to-Rank * TODO: Setting up data collection * TODO: explicit and implicit relevance labels * TODO: Configuring ranking factors * TODO: Automatic config generation based on your existing data * TODO: Personalization and tracking visitor profile # Reranking with cross-encoders In this guide we will set up Metarank as a simple inference server for cross-encoder LLMs (Large Language Models). In other words, we will use an open-source cross-encoder model to reorder your search results in a zero-shot manner, without collecting any visitor feedback data. We will use a pre-trained `MS-MARCO MiniLM-L6-v2` cross-encoder from the [sentence-transformers](https://sbert.net) package. ## What are cross-encoders? Cross-encoder LLM is a way to leverage the semantic power of neural network to reorder top-N matching documents for your query. You can think about it as asking ChatGPT the following question: ``` For a search query "crocs", reorder the following documents in the decreasing relevance order: 1. Crocs Jibbitz 5-Pack Alien Shoe Charms 2. Crocs Specialist Vent Work 3. Crocs Kids' Baya Clog ``` TODO: add ChatGPT answer For typical reranking scenarios, cross-encoders (even in zero-shot modes) are much more precise compared to bi-encoders. ## Initial setup Let's imagine you already have a traditional search engine running (like Elasticsearch, OpenSearch or SOLR), which already has a good [recall](https://en.wikipedia.org/wiki/Precision_and_recall) level - it retrieves all the relevant products, but it sometimes struggles with precision: there can be some false-positives, and the ranking is not perfect. In this guide we will take top-N matching documents from your search engine, and re-rank them according to their semantic similarity with the query. ![reranking flow](/files/Dgd0hlRQHDt9oEo07uiv) ### Importing data We will use an [Amazon ESCI](https://github.com/amazon-science/esci-data) e-commerce dataset as a toy but realistic example: it has 1.7M real products that we can easily query with Elasticsearch. You can download the JSON-encoded version here: . Assuming that you have Elasticsearch service running on `http://localhost:9200`, you can import the complete dataset with the following Python script: ```python import json from elasticsearch import Elasticsearch es = Elasticsearch(hosts="http://localhost:9200") with open('esci.json', 'r') as f: for line in f.readlines(): doc = json.loads(line.rstrip()) if 'title' in doc: # index only title and asin fields es.index(index="esci", document={'title': doc['title'], 'asin': doc['asin']}) ``` And then you can perform simple keyword searches over the data. For example, you can search for "crocs": ```bash curl -XPOST -d @search.json -H "Content-Type: application/json" http://localhost:9200/esci/_search ``` Where the `search.json` looks like this: ```json { "query": { "multi_match": { "query" : "crocs", "fields": ["title"] } }, "fields": ["asin","title"] } ``` For this search query, Elasticsearch returned 30 matching products, but we will take only top-10 of them for further reranking: ```json { "took": 7, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 30, "relation": "eq" }, "max_score": 10.184343, "hits": [ { "_index": "esci", "_id": "17qoxocBFbzZBgn-7-iy", "_score": 10.184343, "_source": { "title": "Crocs Jibbitz 5-Pack Alien Shoe Charms | Jibbitz for Crocs", "asin": "B089YD2KK5" }, "fields": { "asin": [ "B089YD2KK5" ], "title": [ "Crocs Jibbitz 5-Pack Alien Shoe Charms | Jibbitz for Crocs" ] } } ] } } ``` ## Metarank as an inference server As we got our top-10 search results for reranking, we're now going to configure Metarank in inference mode for cross-encoders. This can be done with the following configuration file: ```yaml inference: msmarco: type: cross-encoder model: metarank/ce-msmarco-MiniLM-L6-v2 ``` After start-up, Metarank will expose it's HTTP API and you can query the `/inference` API endpoint to perform the reranking. See the [API Reference](/reference/api#inference-with-llms) for details about payload format: ```bash curl -XPOST -d @rerank.json -H "Content-Type: application/json" http://metarank:8080/inference/cross/msmarco ``` Where `rerank.json` request looks like: ```json {"input": [ {"query": "crocs", "text": "Crocs Jibbitz 5-Pack Alien Shoe Charms"}, {"query": "crocs", "text": "Crocs Specialist Vent Work"}, {"query": "crocs", "text": "Crocs Kids' Baya Clog"} ]} ``` Metarank will respond with a set of scores, corresponding to the similarity of each query-document pair: ```json {"scores": [0.756001, 0.52617, 0.193747]} ``` ## Cross-encoder latency Note that due to the LLM inference happening for all document-query pairs, cross-encoders can be quite slow for large reranking windows: ``` Benchmark (batch) (model) Mode Cnt Score Error Units encode 1 ce-msmarco-MiniLM-L6-v2 avgt 30 12.298 ± 0.581 ms/op encode 10 ce-msmarco-MiniLM-L6-v2 avgt 30 58.664 ± 2.064 ms/op encode 100 ce-msmarco-MiniLM-L6-v2 avgt 30 740.422 ± 13.369 ms/op ``` As it can be seen from the benchmark above, windows of top-100 products may incur a noticeable latency, so try to keep this reranking window reasonably small. # Installation Metarank is available as a Docker and JAR packages for MacOS, Windows and Linux. ## Docker image Metarank docker images are published on DockerHub as [metarank/metarank](https://hub.docker.com/r/metarank/metarank): * official Metarank images are multi-arch and support both amd64 and arm64/v8 (so will natively work on Mac M1/M2 without emulation). * `latest` tag may point to pre-release versions, use an exact pinned version for stability. * on Mac M1 you can use x86\_64 docker images, or try running the JAR file directly. To start using metarank with docker, just run: ```bash docker run metarank/metarank:0.7.9 --help ``` ## JAR File Metarank is a JVM application and also available as a JAR application on [Releases](https://github.com/metarank/metarank/releases) page. As it bundles a couple of native libraries (interfaces to [LightGBM](https://github.com/metarank/lightgbm4j) and [XGBoost](https://github.com/metarank/xgboost-java)), it supports the following platforms and operating systems: * Linux: x86\_64/AArch64, JVM 11+ * Windows: x86\_64, Windows 10+, JVM 11+ * MacOS: x86\_64/AArch64, MacOS 11+, JVM 11+ To start metarank JAR file, just run: ```bash java -jar metarank.jar --help ``` ### Java To run JVM applications, you need the JVM itself. If you have no JRE/JDK installed, check out the [Eclipse Temurin JDK](https://adoptium.net/installation/) tutorials for different operating systems. Metarank is tested on JDK 11 and 17, but will probably work on 18+. JDK 8, 9, 10 are not supported. ### Installing on MacOS Metarank JAR app requires a [libomp](https://formulae.brew.sh/formula/libomp) to be installed: ```bash brew install libomp ``` Without libomp you may encounter a strange UnsatisfiedLinkError while training the model: ``` 15:32:03.936 INFO ai.metarank.main.command.Train$ - training model for train=7067 test=1706 Loading native lib osx/x86_64/lib_lightgbm.dylib Extracting native lib /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm.dylib Copied 3775632 bytes Extracted file: exists=true path=/var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm.dylib Cannot load library: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm.dylib cause: Can't load library: /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm.dylib Loading native lib osx/x86_64/lib_lightgbm_swig.dylib Extracting native lib /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm_swig.dylib Copied 89308 bytes Extracted file: exists=true path=/var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm_swig.dylib Cannot load library: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm_swig.dylib cause: Can't load library: /var/folders/nl/2p5w70jj5_50ztn25q2xll380000gn/T/lib_lightgbm_swig.dylib Exception in thread "io-compute-1" java.lang.UnsatisfiedLinkError: 'long com.microsoft.ml.lightgbm.lightgbmlibJNI.new_voidpp()' ``` # Event Format Metarank expects to receive a predefined set of events, describing visitor activity and item metadata: 1. Item metadata events. They describe what should be known about items. 2. User metadata events. They describe what we may know about visitors. 3. Ranking events. What was presented to the visitor. 4. Interaction events. What visitor did with the ranking (clicks, likes, purchases). ## Event format Metarank expects to receive these events in JSON format for simplicity. There is also a plan to add support for protobuf in some future version. Each event has a couple of shared required fields: * id - which is a unique identifier for the event, should be generated by your app * timestamp - number of milliseconds from 1970-01-01T00:00:00 (see [timestamp format description](/reference/event-schema/timestamp-formats) on which formats are supported) * event - one of `item`/`user`/`ranking`/`interaction` values. There is also a shared optional "tenant" field for multi-tenancy cases. ### `fields` parameter `fields` parameter is used in all events, however it is **optional** for ranking and interaction events. You can provide some additional context, like a search query, selected filters or shipping data that can be used as features of your personalization model. ```json "fields": [ {"name": "title", "value": "You favourite cat"} ] ``` `fields.name`: name of the field. `fields.value`: can be any of the following types: * boolean * string * number * list of strings * list of numbers ## Item metadata event Metadata event is used to provide Metarank with updates of your content items (new items added, updates of values for existing items). You don't need to pass all values that your items have; only the ones that you might use as your personalization model features. ### Event format ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "item": "item1", "fields": [ {"name": "title", "value": "You favourite cat"}, {"name": "color", "value": ["white", "black"]}, {"name": "is_cute", "value": true} ] } ``` * `id`: event id. * `item`: id of the content item. * `fields`: an array of content item properties, see [event fields chapter](#fields-parameter) for details. ## User metadata event User metadata is useful when you have some extra knowledge about your visitor. For example, if visitor filled a signup form, it could be gender or age. ### Event format ```json { "event": "user", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "fields": [ {"name": "age", "value": 33}, {"name": "gender", "value": "m"} ] } ``` * `id`: event id. This field is not yet used, but the value must be provided at the moment. * `user`: id of the visitor. * `fields`: an array of content item properties, see [event fields chapter](#fields-parameter) for details. ## Ranking event Ranking event is used to indicate what items and in what order are shown to a visitor. This information is used by personalization algorithms to understand which items are relevant for the visitor. ### Event format ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "query", "value": "cat"}, {"name": "source", "value": "search"} ], "items": [ {"id": "item3", "fields": [{"name": "relevancy", "value": 2.0}]}, {"id": "item1", "fields": [{"name": "relevancy", "value": 1.0}]}, {"id": "item2", "fields": [{"name": "relevancy", "value": 0.1}]} ] } ``` * `id`: a request identifier later used to join ranking and interaction events. Should match the value that is sent to the [Ranking API](/reference/api). * `user`: an optional unique visitor identifier. * `session`: an optional session identifier, a single visitor may have multiple sessions. * `fields`: an optional array of extra fields that you can use in your model, as described above. * `items`: which particular items were displayed to the visitor. * `items.id`: id of the content item. Should match the `item` property from metadata event. * `items.fields`: a set of optional per-item fields. * `items.label`: an optional field for explicit relevance judgments. ## Interaction event Interaction event identifies which actions the visitor performed on the items displayed. Some of the example of such events are: click, like, purchase. The `type` field must match the `name` provided in the [Configuration](/reference/overview). ### Event format ```json { "event": "interaction", "id": "0f4c0036-04fb-4409-b2c6-7163a59f6b7d", "ranking": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "type": "purchase", "item": "item1", "fields": [ {"name": "count", "value": 1}, {"name": "shipping", "value": "DHL"} ] } ``` * `id`: a request identifier. * `ranking`: an optional identifier of the parent ranking event. Some interactions may happen outside of the ranking event (for example, likes happened on an item page), so it can be legally empty. * `user`: an optional unique visitor identifier. * `session`: an optional session identifier, a single visitor may have multiple sessions. * `type`: internal name of the event. * `item`: id of the content item. Should match the `item` property from metadata event. * `fields`: an optional array of extra fields that you can use in your model, as described above. # Timestamp formats All input Metarank events have a timestamp field, an example: ```json { "type": "item", "id": "product1", "timestamp": "1599391467000", "fields": [ {"name": "title", "value": "Nice jeans"} ] } ``` The underlying JSON format has [a numerical precision issue with timestamps encoded as double numbers](https://www.techempower.com/blog/2016/07/05/mangling-json-numbers/), so Metarank supports multiple ways of parsing timestamps: * as string literal: number of millis from 1970-01-01 00:00:00 in UTC as a string, like `"1599391467000"` * as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) formatted zoned datetime string, in UTC: `"2022-06-22T11:21:39Z"` * (not recommended) as JSON number, like `1599391467000` # API Metarank's API provides an easy way to integrate Metarank with your applications. * [Feedback API](#feedback) receives the stream of events * [Train API](#train) trains the model on your data * [Ranking API](#ranking) provides personalized results generated by the trained model * [Recommend API](#recommendations) - retrieval of recommendations. * [Inference API](#inference-with-llms) - using LLMs for encoding and cross-encoding texts. * [Prometheus endpoint](#prometheus-metrics) - to have a nice metrics dashboard about Metarank internals. ## Feedback **API Endpoint**: `/feedback` **Method**: `POST` Feedback endpoint receives several types of events: item, user, interaction, ranking. Integrating these events is crucial for personalization to operate properly and provide relevant results. ### Payload format You can find events and their description on the [Supported events](/reference/event-schema). ### Response A JSON message with the following fields: * `accepted`: how many events from the submitted batch were processed * `status`: "ok" when no errors found * `tookMillis`: how many milliseconds batch processing took * `updated`: how many underlying ranking features were recomputed. Example: ```json {"accepted":1,"status":"ok","tookMillis":3,"updated":0} ``` ### Example ```shell $> curl http://localhost:8080/feedback -d '{ "event": "ranking", "id": "id1", "items": [ {"id":"72998"}, {"id":"589"}, {"id":"134130"}, {"id":"5459"}, {"id":"1917"}, {"id":"2571"}, {"id":"1527"}, {"id":"97752"}, {"id":"1270"}, {"id":"1580"}, {"id":"109487"}, {"id":"79132"} ], "user": "alice", "session": "alice1", "timestamp": 1661431894711 }' * Trying 127.0.0.1:8080... * Connected to localhost (127.0.0.1) port 8080 (#0) > POST /feedback HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.86.0 > Accept: */* > Content-Length: 354 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 200 OK < Date: Mon, 28 Nov 2022 13:09:44 GMT < Content-Length: 55 < {"accepted":1,"status":"ok","tookMillis":3,"updated":0} ``` ## Train **API Endpoint**: `/train/` **Method**: `POST` Train endpoint runs the training on persisted click-through data. You can run this method at any time to re-train the model. See the [Model retraining how-to](/how-to/model-retraining) on how to set up the retraining. **Payload**: none ### Response A JSON response with the following fields: * `weights`: per-field model weights * `sizeBytes`: model size in bytes * `features`: test/train error loss while training. Example: ```json { "features": [ { "name": "vote_avg", "weight": 629.0 }, { "name": "profile", "weight": [ 1202.0, 373.0, 627.0, 145.0 ] } ], "iterations": [ { "id": 0, "millis": 274, "testMetric": 0.5787768851757988, "trainMetric": 0.593075630098252 }, { "id": 1, "millis": 104, "testMetric": 0.5903952545996365, "trainMetric": 0.6083208266384491 } ], "sizeBytes": 843792 } ``` ## Ranking **API Endpoint**: `/rank/` **Method**: `POST` **Querystring Parameters**: * `explain: boolean`: used to provide extra information in the response containing calculated feature values. Ranking endpoint does the real work of personalizing items that are passed to it. You need to explicitly define which model to invoke. ### Payload format ```json { "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "query", "value": "jeans"}, {"name": "source", "value": "search"} ], "items": [ {"id": "item3", "fields": [{"name": "relevancy", "value": 2.0}]}, {"id": "item1", "fields": [{"name": "relevancy", "value": 1.0}]}, {"id": "item2", "fields": [{"name": "relevancy", "value": 0.1}]} ] } ``` * `id`: a request identifier later used to join ranking and interaction events. This will be the same value that you will pass to feedback endpoint for impression and ranking events. * `user`: an optional unique visitor identifier. * `session`: an optional session identifier, a single visitor may have multiple sessions. * `timestamp`: when this event happened. (see [timestamp format description](/reference/event-schema/timestamp-formats) on which formats are supported) * `fields`: an optional array of extra fields that you can use in your model, for more information refer to [Supported events](/reference/event-schema). * `items`: which particular items were displayed to the visitor. * `items.id`: id of the content item. Should match the `item` property from item metadata event. * `items.fields`: an optional set of per-item fields, for example BM25 scores coming from ES. See [how to use BM25 scores](/reference/overview/feature-extractors/relevancy#ranking) in ranking. * `items.label`: an optional field for explicit relevance judgments. ### Response format ```json { "took": 5, "items": [ {"item": "item2", "score": 2.0, "features": [{"name": "popularity", "value": 10 }]}, {"item": "item3", "score": 1.0, "features": [{"name": "popularity", "value": 5 }]}, {"item": "item1", "score": 0.5, "features": [{"name": "popularity", "value": 2 }]} ] } ``` * `took`: number of millis spend processing request * `items.id`: id of the content item. Will match `item` property from the item metadata event. * `items.score`: a score calculated by personalization model * `items.features`: an array of feature values calculated by pesonaliization model. This field will be returned if `explain` field is set to `true` in the request. The structure of this object will vary depending on the feature type. ## Recommendations **API Endpoint**: `/recommend/` **Method**: `POST` Recommend endpoint returns recommended items that are produced by [recommendations model types](/reference/overview/recommendations). ### Payload format: ```json { "count": 10, "user": "alice1", "items": ["item4"] } ``` Where: * `count` - number of items to recommend. * `user` - optional, current user id * `items` - context of recommendation. For example, it can be single item for "similar-items" recommendation, and multiple items at once for "cart-style" recommendations. ### Response format ```json { "took": 5, "items": [ {"item": "item2", "score": 2.0}, {"item": "item3", "score": 1.0}, {"item": "item1", "score": 0.5} ] } ``` * `took`: number of millis spend processing request * `items.id`: id of the content item. * `items.score`: a score calculated by recommender model. ## Inference with LLMs Metarank has API for quick and dirty LLM inference and encoding of texts. It can be useful for implementing hybrid search applications, when you need an actual embedding vector for a query. ### LMM bi-encoders **API Endpoint**: `/inference/encoder/` **Method**: `POST` Encode a batch of strings into vectors using configured model ``. #### Payload format ```json {"texts": [ "Berlin is a capital city", "My cat is fast" ] } ``` #### Response format ```json { "took": 5, "embeddings": [ [1, 2, 3, 4], [0, 7, 2, 1] ] } ``` ### LLM with cross-encoders **API Endpoint**: `/inference/cross/` **Method**: `POST` Encode a batch of query-document pairs into similarity scores using configured model ``. #### Payload format ```json {"input": [ {"query": "cat", "text": "my cat is fast"}, {"query": "cat", "text": "it has V8 engine"} ]} ``` #### Response format ```json { "took": 5, "scores": [0.25, 0.01] } ``` ## Prometheus metrics **API Endpoint**: `/metrics` **Method**: `GET` Dumps app and JVM metrics in a prometheus format. ### Example ```shell > GET /metrics HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.86.0 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 28 Nov 2022 13:30:41 GMT < Transfer-Encoding: chunked < # HELP metarank_feedback_events_total Number of feedback events received # TYPE metarank_feedback_events_total counter metarank_feedback_events_total 58441.0 # HELP metarank_rank_requests_total Number of /rank requests # TYPE metarank_rank_requests_total counter metarank_rank_requests_total{model="xgboost",} 5.0 # HELP metarank_rank_latency_seconds rank endpoint latency # TYPE metarank_rank_latency_seconds summary metarank_rank_latency_seconds{model="xgboost",quantile="0.5",} 0.011451508 metarank_rank_latency_seconds{model="xgboost",quantile="0.8",} 0.014340056 metarank_rank_latency_seconds{model="xgboost",quantile="0.9",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.95",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.98",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.99",} 0.119447575 ``` # Command-line options Metarank CLI has a set of command-line options to control its behavior. To run the main app, download the [latest jar file](https://github.com/metarank/metarank/releases) and run the following command: ```shell java -jar metarank-x.x.x.jar ``` ```shell __ __ _____ _____/ |______ ____________ ____ | | __ / \_/ __ \ __\__ \\_ __ \__ \ / \| |/ / | Y Y \ ___/| | / __ \| | \// __ \| | \ < |__|_| /\___ >__| (____ /__| (____ /___| /__|_ \ \/ \/ \/ \/ \/ \/ Metarank v:unknown Usage: metarank Options: -h, --help Show help message -v, --version Show version of this program Subcommand: import - import historical clickthrough data -c, --config path to config file -d, --data path to an input file -f, --format input file format: json, snowplow, snowplow:tsv, snowplow:json (optional, default=json) -o, --offset offset: earliest, latest, ts=1663171036, last=1h (optional, default=earliest) -s, --sort-files-by how should multiple input files be sorted (optional, default: name, values: [name,last-modified] -v, --validation should input validation be enabled (optional, default=false) -h, --help Show help message Subcommand: train - train the ML model -c, --config path to config file -m, --model model name to train -s, --split train/test splitting strategy (optional, default: time=80%, options: random=N%,time=N%,hold_last=N%) -h, --help Show help message Subcommand: serve - run the inference API -c, --config path to config file -h, --help Show help message Subcommand: standalone - import, train and serve at once -c, --config path to config file -d, --data path to an input file -f, --format input file format: json, snowplow, snowplow:tsv, snowplow:json (optional, default=json) -o, --offset offset: earliest, latest, ts=1663171036, last=1h (optional, default=earliest) -s, --sort-files-by how should multiple input files be sorted (optional, default: name, values: [name,last-modified] -v, --validation should input validation be enabled (optional, default=false) -h, --help Show help message Subcommand: validate - run the input data validation suite -c, --config path to config file -d, --data path to an input file -f, --format input file format: json, snowplow, snowplow:tsv, snowplow:json (optional, default=json) -o, --offset offset: earliest, latest, ts=1663171036, last=1h (optional, default=earliest) -s, --sort-files-by how should multiple input files be sorted (optional, default: name, values: [name,last-modified] -v, --validation should input validation be enabled (optional, default=false) -h, --help Show help message Subcommand: sort - sort the dataset by timestamp -d, --data path to a file/directory with input files -o, --out path to an output file -h, --help Show help message Subcommand: autofeature - generate reference config based on existing data -c, --cat-threshold min threshold of category frequency, when its considered a catergory (optional, default=0.003) -d, --data path to an input file -f, --format input file format: json, snowplow, snowplow:tsv, snowplow:json (optional, default=json) -o, --offset offset: earliest, latest, ts=1663171036, last=1h (optional, default=earliest) --out path to an output config file -r, --ruleset set of rules to generate config: stable, all (optional, default=stable, values: [stable, all]) -s, --sort-files-by how should multiple input files be sorted (optional, default: name, values: [name,last-modified] -v, --validation should input validation be enabled (optional, default=false) -h, --help Show help message Subcommand: export - export training dataset for hyperparameter optimization -c, --config path to config file -m, --model model name to export data for -o, --out a directory to export model training files --sample sampling ratio of exported training click-through events -s, --split train/test splitting strategy (optional, default: time=80%, options: random=N%,time=N%,hold_last=N%) -h, --help Show help message Subcommand: termfreq - compute term frequencies for the BM25 field_match extractor -d, --data path to an input file -f, --fields Comma-separated list of text fields -l, --language Language to use for tokenization, stemming and stopwords -o, --out an file to write term-freq dict to -h, --help Show help message For all other tricks, consult the docs on https://docs.metarank.ai ``` The command-line argument structure is: ```shell java -jar metarank.jar ``` ## Running modes Metarank CLI has a set of different running modes: * `import`: import and process historical data, writing state to the chosen [persistence backend](https://github.com/metarank/metarank/blob/stabledoc/configuration/persistence.md) like Redis. * [`train`](#training-the-model): train the ML model with XGBoost/LightGBM. * `serve`: run the inference API to do realtime reranking * `standalone`: run `import`, `train` and `serve` tasks at once. * [`validate`](#validation): validates data nd configuration files. * [`sort`](#historical-data-sorting): pre-sorts the dataset by timestamp. * [`autofeature`](#auto-feature-generation): automatically generates feature configuration based on your data. * [`export`](#dataset-export): export the training dataset for further hyperparam optimization. * [`termfreq`](/reference/cli): compute term frequency dictionary for [BM25 field\_match extractor](/reference/overview/feature-extractors/text#text-based-extractors) ### Validation Metarank CLI provides `validate` command to validate both your data and configuration file. You will need to provide both data and configuration files ```shell java -jar metarank-x.x.x.jar validate --config config.yml --data events.jsonl.gz ``` The above command will output validation checks performed on the files provided and will output information similar to the following: ```shell 17:46:45.790 INFO ai.metarank.config.Config$ - api conf block is not defined: using default ApiConfig(Hostname(localhost),Port(8080)) 17:46:45.793 INFO ai.metarank.config.Config$ - state conf block is not defined: using default MemoryStateConfig() 17:46:45.798 INFO ai.metarank.config.Config$ - Loaded config file, state=memory, features=[popularity,vote_avg,vote_cnt,budget,release_date,runtime,title_length,genre,ctr,liked_genre,liked_actors,liked_tags,liked_director,visitor_click_count,global_item_click_count,day_item_click_count], models=[xgboost] 17:46:45.874 INFO ai.metarank.FeatureMapping - optimized schema: removed 6 unused features 17:46:46.023 INFO ai.metarank.main.command.Validate$ - Dataset validation is enabled 17:46:46.024 INFO ai.metarank.main.command.Validate$ - Validation loads all events to RAM, so use --validation=false to skip in case of OOM 17:46:46.085 INFO ai.metarank.source.FileEventSource - path=events.jsonl.gz is a file 17:46:46.144 INFO ai.metarank.source.FileEventSource - file events.jsonl.gz selected=true (timeMatch=true formatMatch=true) 17:46:46.146 INFO ai.metarank.source.FileEventSource - reading file events.jsonl.gz (with gzip decompressor) 17:46:55.240 INFO ai.metarank.main.command.Validate$ - Validation done 17:46:55.280 INFO a.m.v.checks.EventOrderValidation$ - Event ordering check = PASS (58437 events sorted by timestamp) 17:46:55.351 INFO a.m.v.checks.EventTypesValidation$ - event types check = PASS (2512 item events, 9800 rankings, 46125 interactions) 17:46:55.441 INFO a.m.v.c.FeatureOverMissingFieldValidation$ - field reference check = PASS (16 features referencing existing 12 event fields) 17:46:55.509 INFO a.m.v.c.InteractionKeyValidation$ - interaction-ranking join key check = PASS (9800 rankings, all interactions reference existing ones) 17:46:55.603 ERROR a.m.v.c.InteractionMetadataValidation$ - Interaction metadata check: FAIL (96 interaction happened on never seen items) 17:46:55.604 ERROR a.m.v.c.InteractionMetadataValidation$ - examples: List(ecc0c55e-8e30-4f7d-8c7e-26f05951300b, 22dda925-8dec-46e0-98d2-bc6a32dcdaec, 40342ea0-1575-417f-8c3d-d3e8dcdfa7f5, 1c0504a5-8e2c-4a0d-be92-b9930df8d041, bb9a79b2-689c-492b-b758-9f1ce0fade09, c2b5d050-3dae-4b24-a7ba-1652d6f4b2e2, 0689b95a-4532-4d05-b87d-e69daed6c910, 5592ff1e-cccd-4157-8c7e-052242c56d0b, d9d1fd91-0629-4623-af2d-1a0682ee62a7, 0d9ef491-ef19-4003-9f20-09d0daea481a) 17:46:55.827 INFO a.m.v.c.InteractionPositionValidation$ - interaction positions check = PASS (int distribution: [2435,2309,2199,2068,2005,2008,1970,2003,1890,1925,1933,1847,1807,1778,1790,1781,1787,1778,1713,1783,1745,1789,1791,1991] 17:46:55.858 INFO a.m.v.c.InteractionTypeValidation$ - interaction type check = PASS (46125 interactions have known types: Set(click)) 17:46:55.859 INFO ai.metarank.main.Main$ - My job is done, exiting. ``` ### Historical data sorting Metarank expects your historical data to be ordered by the timestamp in the ascending order. If for any reason, you cannot generate a sorted file, the `sort` sub-command can do the job for you. You can sort both single files and folders with multiple files. In case of folders, `sort` command will merge all data into one sorted file. Sorting one file is a simple as ```bash java -jar metarank.jar sort --data unsorted_file.jsonl.gz --out sorted_file.jsonl.gz ``` You can do sorting with a folder as well ```bash java -jar metarank.jar sort --data /my_folder --out sorted_file.jsonl.gz ``` ### Auto feature generation If you don't know what [features](https://github.com/metarank/metarank/blob/stabledoc/configuration/feature-extractors.md) to include in the configuration file, the `autofeature` sub-command can generate the configuration for you based on the historical data you have. Simply run ```shell java -jar metarank.jar autofeature --data /path/to/events.json --out /path/to/config.yaml ``` Check out more about `autofeature` sub-command in our [Automatic feature engineering guide](/how-to/autofeature). ### Training the model You can train the underlying ML ranking model: ```shell java -jar metarank.jar train --config /path/to/config.yaml ``` * if the `--model ` option is not given, then Metarank will train all the defined models sequentially. While training the model, Metarank will split your data into train/validation datasets with the following supported splitting strategies: * `random`: shuffle all the training samples and take N% as a training part. May result in an implicit model leakage, when information about the future was leaked in the training set. An example: on Christmas items with Santa are selling much better (and not selling at all afterwards), and leaking this knowledge into your training set will result in better offline scores (as model knows that Christmas is coming). In production, it will behave significantly worse, as there is no way to predict the future out of the training data anymore. * `time`: sort all training samples by timestamp and pick first N% as training set (the default option). * `hold_last`: group all samples by user, and sort per-user samples by timestamp. N% first samples within each user are picked into the training dataset. Has the same issue with future leaking in the model, but optimizes the train dataset to focus on last user click. The format of split strategy CLI flag is `--strategy name=ratio%`. For example: * random with 90% ratio: `--split random=90%` * random with a default 80% ratio: `--split random` ### Dataset export Metarank can emit CSV/LibSVM formatted datasets and corresponding config files for LightGBM and XGBoost, so you can later perform a hyper-parameter optimization using your favourite tool: ```shell java -jar metarank.jar export --config /path/to/config.yaml --model --out /export/dir ``` Metarank export format is dependent on model backend type: * For XGBoost, we export a LibSVM-encoded train/test files with embedded `qid`. This format is not compatible with the LightGBM LibSVM reader implementation (and we were unable to make it work with both). An example: ``` 1 qid:1 0:1.0 1:0.2 0 qid:1 0:0.5 1:0.3 1 qid:2 0:0.1 1:1.0 ``` * for LightGBM, we export a CSV-encoded train/test files with header. XGBoost (as for version 1.70) cannot load query information from CSV files, so it cannot be used for LambdaMART. Example: ``` label,group,f1,f2 1.0,1,1.0,0.2 0.0,1,0.5,0.3 1.0,2,0.1,1.0 ``` For both booster implementations Metarank also emits a corresponding config file with all default values filled in. You can just run the lightgbm/xgboost cli tool externally to replicate what's Metarank is doing. For XGBoost: ```shell $> ls -l total 35760 -rw-r--r-- 1 shutty shutty 5852217 Nov 2 12:40 test.svm -rw-r--r-- 1 shutty shutty 23830680 Nov 2 12:40 train.svm -rw-r--r-- 1 shutty shutty 171 Nov 2 12:59 xgboost.conf $> cat xgboost.conf eta=0.1 max_depth=8 subsample=0.8 num_round=5 objective=rank:pairwise eval_metric=ndcg@10 seed=0 data=train.svm test:data=test.svm eval[train=train.svm eval[test]=test.svm $> xgboost xgboost.conf [13:00:09] [0] train-ndcg@10:0.58637041553234925 test-ndcg@10:0.54517512609937979 [13:00:09] [1] train-ndcg@10:0.60275740110564990 test-ndcg@10:0.55988540039586532 [13:00:09] [2] train-ndcg@10:0.61207514143863639 test-ndcg@10:0.56497256071656654 [13:00:10] [3] train-ndcg@10:0.61507850860846780 test-ndcg@10:0.56668620039370876 [13:00:10] [4] train-ndcg@10:0.61723381498663543 test-ndcg@10:0.56796994649060073 ``` For LightGBM: ```shell $> ls -l total 39740 -rw-r--r-- 1 shutty shutty 219 Nov 2 13:01 lightgbm.conf -rw-r--r-- 1 shutty shutty 6845983 Nov 2 13:01 test.csv -rw-r--r-- 1 shutty shutty 28453327 Nov 2 13:01 train.csv $> cat lightgbm.conf objective=lambdarank data=train.csv valid=test.csv num_iterations=5 learning_rate=0.1 seed=0 max_depth=8 header=true label_column=name:label group_column=name:group lambdarank_truncation_level=10 metric=ndcg eval_at=10 $> lightgbm config=lightgbm.conf [LightGBM] [Info] Warning: last line of lightgbm.conf has no end of line, still using this line [LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31). [LightGBM] [Info] Finished loading parameters [LightGBM] [Info] Using column label as label [LightGBM] [Info] Using column group as group/query id [LightGBM] [Info] Construct bin mappers from text data time 0.14 seconds [LightGBM] [Info] Finished loading data in 0.279901 seconds [LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.020266 seconds. You can set `force_row_wise=true` to remove the overhead. And if memory is not enough, you can set `force_col_wise=true`. [LightGBM] [Info] Total Bins 1837 [LightGBM] [Info] Number of data points in the train set: 164232, number of used features: 29 [LightGBM] [Info] Finished initializing training [LightGBM] [Info] Started training... [LightGBM] [Info] Iteration:1, valid_1 ndcg@10 : 0.561739 [LightGBM] [Info] 0.009204 seconds elapsed, finished iteration 1 [LightGBM] [Info] Iteration:2, valid_1 ndcg@10 : 0.572802 [LightGBM] [Info] 0.017980 seconds elapsed, finished iteration 2 [LightGBM] [Info] Iteration:3, valid_1 ndcg@10 : 0.576812 [LightGBM] [Info] 0.032011 seconds elapsed, finished iteration 3 [LightGBM] [Info] Iteration:4, valid_1 ndcg@10 : 0.582428 [LightGBM] [Info] 0.045455 seconds elapsed, finished iteration 4 [LightGBM] [Info] Iteration:5, valid_1 ndcg@10 : 0.582633 [LightGBM] [Info] 0.052650 seconds elapsed, finished iteration 5 [LightGBM] [Info] Finished training ``` Metarank supports the same train/test split strategies for `export` subcommand as for the [train](#training-the-model) one. ### BM25 term frequencies dictionary To use the BM25 score in the [field\_match](/reference/overview/feature-extractors/text#fieldmatch), you need to compute a bit of statistics over your textual information. To do so, run the `termfreq` subcommand: ```shell $> java -jar meratank.jar termfreq --data \ --out /term-freq.json --fields title,description --language en INFO ai.metarank.main.Main$ - Metarank vunknown is starting. INFO ai.metarank.source.FileEventSource - path=src/test/resources/ranklens/events/events.jsonl.gz is a file INFO ai.metarank.source.FileEventSource - file src/test/resources/ranklens/events/events.jsonl.gz selected=true (timeMatch=true formatMatch=true) INFO ai.metarank.source.FileEventSource - reading file src/test/resources/ranklens/events/events.jsonl.gz (with gzip decompressor) INFO ai.metarank.flow.PrintProgress$ - processed 0 events, perf=0rps GC=9.54% heap=0.82%/7.82G INFO ai.metarank.flow.PrintProgress$ - processed 2048 events, perf=1557rps GC=2.51% heap=2.52%/7.82G INFO ai.metarank.flow.PrintProgress$ - processed 14336 events, perf=12130rps GC=0.0% heap=6.44%/7.82G INFO ai.metarank.flow.PrintProgress$ - processed 37888 events, perf=23296rps GC=0.3% heap=1.71%/7.82G INFO ai.metarank.main.command.TermFreq$ - built term-freq lang=en fields=[title, description] terms=11560 INFO ai.metarank.main.command.TermFreq$ - writing /tmp/tf.json, size=204 KB INFO ai.metarank.main.command.TermFreq$ - done INFO ai.metarank.main.command.TermFreq$ - done INFO ai.metarank.main.Main$ - My job is done, exiting. ``` With the resulting `term-freq.json` file you can configure the BM25 score extractor in the following way: ```yaml - name: title_match type: field_match rankingField: ranking.query itemField: item.title method: type: bm25 language: english termFreq: "/path/to/term-freq.json" ``` The same dictionary can be used for multiple `field_match` extractors, for example when you want to have separate BM25 scores for query-title and query-description matches. ## Environment variables Config file can be passed to the Metarank not only as a command-line argument, but also as an environment variable. This is typically used in docker and k8s-based deployments: * `METARANK_CONFIG`: path to config file, for example `s3://bucket/prefix/config.yml` # Configuration Metarank YAML config file contains the following sections: * [Persistence](#persistence): how feature data is stored * [Models](#models): which models should be trained and used in inference * [Recommendations](/reference/overview/recommendations): a special section on recommendations serving * [Features](#features): how features are computed from events * [API](#api): network options for API * [Data sources](#data-sources): where to read events from * [Core](#core): service options, like anonymous tracking and error reporting. See the [sample-config.yml](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/sample-config.yml) file for a full working example. ## Persistence The "state" section describes how computed features and models are stored. Check [Persistence configuration](/reference/overview/persistence) for more information. An example persistence conf block with comments: ```yaml state: # a place to store the feature values for the ML inference and the trained model # Local memory # A node-local in-memory storage without any persistence. # Feature values and the trained model is stored in-memory. # Suitable only for local testing, as in case of a restart it will loose all the data. type: memory # Remote redis, with persistence. # Saves the computed features and trained model in a Redis instance. # You can use remote or local Redis installation. #type: redis #host: localhost #port: 6369 #format: binary # optional, default=binary, possible values: json, binary # Metarank implements several optimization strategies when using Redis: caching and pipelining # Check https://docs.metarank.ai/reference/overview/persistence#redis-persistence for more details #cache: # optional # maxSize: 4096 # size of in-memory client-side cache for hot keys, optional, default=4096 # ttl: 1h # how long should key-values should be cached, optional, default=1h #pipeline: # optional # maxSize: 128 # batch write buffer size, optional, default=128 # flushPeriod: 1s # buffer flush interval, optional, default=1s # enabled: true # toggle pipelining, optional, default=true # can be also overridden from environment variables, see the # https://docs.metarank.ai/reference/overview/persistence#redis-persistence for details #auth: # optional # user: # optional when Redis ACL is disabled # password: # required if Redis server is run with requirepass argument # tls: # optional, defaults to disabled # enabled: true # optional, defaults to false # ca: # optional path to the CA used to generate the cert, defaults to the default keychain # verify: full # optional, default=full, possible values: full, ca, off # full - verify both certificate and hostname # ca - verify only certificate # off - skip verification #timeout: # optional, defaults to 1s for all sub-timeouts # connect: 1s # optional, defaults to 1s # socket: 1s # optional, defaults to 1s # command: 1s # optional, defaults to 1s ``` ## Training Metarank also computes a click-through data structure, which contains the following bits of information: * ranking: which items were presented to the visitor * interactions: what visitor did after seeing the ranking (like clicks, purchases and so on) * feature values, which were used to produce the ranking in the past. These click-through events are essential for model training, as they're later translated into the implicit judgement lists for the underlying LambdaMART model: ![Implicit judgements](/files/4aDQhQdZAE1sDz0sPQCj) Metarank has multiple ways of storing these click-throughs with different pros and cons: * **Redis**: no special configuration needed, it's possible to perform periodic ML model retraining by reading the latest click-through events from it. But it takes quite a lot of RAM and maybe costly in a case when you have millions of click-through events. ```yaml train: type: redis # all options from state.redis here ttl: # optional, default 365 days. ``` * **Discard**: do not store click-through events at all. ```yaml train: type: discard ``` * **Local dir**: takes much less RAM (as ct's are not stored in redis), but you need to manage the directory containing the click-through files by yourself. ```yaml train: type: file path: /path/to/dir # path to a directory which will be used for persistence during export/import format: json # options are: json, binary ``` * **S3**: like local file, but offloads data to an external block storage, suits well for Kubernetes deployments. ```yaml train: type: s3 bucket: # required, S3 bucket name prefix: # required, Prefix/dir name to store files into region: # required, S3 region compress: none | gzip | zst # optional, default: gzip partSizeBytes: 10485760 # optional, pre-compression, default: 10Mb partSizeEvents: 1024 # optional, default: 1024 events partInterval: 1h # optional, default: 1h endpoint: # optional, custom S3 endpoint format: json | binary # optional, default: binary awsKey: "" # optional, you should prefer setting # AWS_KEY_ID and AWS_SECRET_KEY_ID env vars awsKeySecret: "" # optional ``` S3 click-through store can either use hardcoded AWS credentials from config (which is not good from security perspective), or fall-back to the ones defined in env variables. ## Features This section describes how to map your input events into ML features that Metarank understands. See [Feature extractors](/reference/overview/feature-extractors) for an overview of supported types. ```yaml # These features can be shared between multiple models, so if you have a model A using features 1-2-3 and # a model B using features 1-2, then all three features will be computed only once. # You need to explicitly include a feature in the model configuration for Metarank to use it. features: - name: popularity type: number scope: item source: item.popularity # TTL and refresh fields are part of every feature extractor that Metarank supports. # The purpose of TTL is to configure data retention period, so in a case when there were no # feature updates for a long time, it will eventually be dropped. ttl: 60d # Refresh parameter is used to downsample the amount of feature updates emitted. For example, # there is a window_counter feature extractor, which can be used to count a number of clicks that happened for # an item. Incrementing such a counter for a single day is an extremely lightweight operation, but computing # window sums is not. As it's not always required to receive up-to-date counter values in ML models, # these window sums can be updated only eventually (like once per hour), which improves the throughput a lot # (but results in a slightly stale data during the inference process) refresh: 1h - name: genre type: string scope: item source: item.genres values: - drama - comedy - thriller ``` For inspiration, you can use a [ranklens feature configuration](https://github.com/metarank/metarank/blob/master/src/test/resources/ranklens/config.yml) used for [Metarank demo site](https://demo.metarank.ai). ## Models The "models" section describes ML models used for personalization. Check [Supported ranking models](/reference/overview/supported-ranking-models) for more information about ranking models. See also [recommendations models overview](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/recommendations/overview.md) ```yaml models: default: # name of the model, used in the inference process as a part of path, like /rank/default type: lambdamart # model type backend: type: xgboost # supported values: xgboost, lightgbm for lambdamart model iterations: 100 # optional (default 100), number of iterations while training the model seed: 0 # optional (default = random), a seed to make training deterministic weights: # types and weights of interactions used in the model training click: 1 # you can increase the weight of some events to hint model to optimize more for them features: # features from the previous section used in the model - popularity - genre # You can specify several models at once. # This can be useful for A\B test scenarios or while testing different sets of features. #random: # type: shuffle # shuffle model type produces random results # maxPositionChange: 5 # controls the amount of randomness that shuffle can introduce in the original ranking # The noop model does nothing with the original ranking and returns results "as is" #noop: # type: noop # A similar-items MF ALS model similar: type: als interactions: [click] # which types of interactions to use factors: 100 # how many implicit factors to compute iterations: 30 # number of model training iterations # A simple "popular items" model trending: type: trending weights: - interaction: click decay: 1.0 # 0..1, 0.5 means yesterday is 50% less important than today weight: 1.0 # in a case with multiple interaction types ``` ## Inference The "inference" section describes inference model configuration for search results re-ranking. Check the [inference models](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/inference-models.md) section for more information. ```yaml inference: msmarco: # name of the model type: cross-encoder # model type model: metarank/ce-msmarco-MiniLM-L6-v2 # model source ``` ## API The "api" section describes the Metarank API configuration. This section is optional and by default binds service to port 8080 on all network interfaces. ```yaml api: port: 8080 host: "0.0.0.0" ``` ## Data sources The optional "source" section describes the source of the data, and by default expects you to submit all user feedback using [the API](/reference/api). Check [Supported data sources](/reference/overview/data-sources) for more information. ```yaml source: type: file # source type, available options: file, kafka, pulsar, kinesis #path: /home/user/ranklens/events/ # path to events file, alternatively you can use CLI to provide file location #offset: earliest|latest|ts=|last= #default: earliest #format: # file format, default: json # Check https://docs.metarank.ai/reference/overview/data-sources#apache-kafka for more information #type: kafka #brokers: [broker1, broker2] #topic: events #groupId: metarank #offset: earliest|latest|ts=|last= #format: # Check https://docs.metarank.ai/reference/overview/data-sources#apache-pulsar for more information #type: pulsar #serviceUrl: #adminUrl: #topic: events #subscriptionName: metarank #subscriptionType: exclusive # options are exclusive, shared, failover #offset: earliest|latest|ts=|last= #format: # Check https://docs.metarank.ai/reference/overview/data-sources#aws-kinesis-streams for more information #type: kinesis #region: us-east-1 #topic: events #offset: earliest|latest|ts=|last= #format: ``` ## Core This optional section contains parameters related to the metarank service itself. Default setup: ```yaml core: # How rankings and interactions are joined into click-throughs. For details, see the section below in this doc. clickthrough: maxParallelSessions: 10000 # how many active sessions may happen within a `maxSessionLength` period maxSessionLength: 30m # after which period of inactivity session is considered finalized # default = 30m (to be consistent with Google Analytics) # Anonymous usage reporting. It is very helpful to us, so please leave this enabled. tracking: analytics: true errors: true ``` ### Click-through joining Metarank joins ranking and interaction events together into click-through chains, which are later used for machine learning model training. As interactions are happening some time later than rankings, Metarank needs to keep a set of rankings in the buffer, awaiting all the interactions that may happen later. This buffer policy is controlled by the following parameters: * `core.clickthrough.maxSessionLength`: after which time period the session should be considered finalized, so no more interactions are allowed to happen. Default values is 30m, as in Google Analytics. * `core.clickthrough.maxParallelSessions`: how many parallel sessions may hang in buffer awaiting interactions. Default is 10k. ### Anonymous usage analytics By default, Metarank collects anonymous usage analytics to improve the tool. No IP addresses are being tracked, only simple counters track what parts of the service are being used. It is very helpful to us, so please leave this enabled. Counters are sent to `https://analytics.metarank.ai` on each service startup. * We never share collected data with anyone else. * Data is stored for 1 year, and then removed. * Collector code running on is open-source: [github.com/metarank/metarank-lambda-tracker](https://github.com/metarank/metarank-lambda-tracker) An example payload: ```json5 { "state" : "memory", "modelTypes" : [ "shuffle" ], "usedFeatures" : [ { "name" : "price", "type" : "number" } ], "system" : { "os" : "Linux", "arch" : "amd64", "jvm" : "17.0.3", // a SHA256 of your network interface MAC address, used as an installation ID "macHash" : "3e78137877f66cfb4f1a0875e7eadb3100fb3c6c4755089b3cc6d9f074a3c4b5" }, "mode" : "train", "version" : "snapshot", "ts" : 1662030509517 } ``` ### Error logging We use [Sentry](https://www.sentry.io) for error collection. This behavior is enabled by default and can be disabled with `core.tracking.errors: false`. Sentry is configured with the following options: * Breadcrumbs are disabled: so it won't share parts of your console log with us. * PII tracking is disabled: no hostnames and IP addresses are included in the error message. An example error payload is available in [sample-error.json](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/sample-error.json). The whole usage logging and error reporting can be disabled also by setting an env variable to `METARANK_TRACKING=false`. # Feature extractors Most common learn-to-rank tasks usually have typical shared set of ML features. As long as you follow the [ingestion event schema](/reference/event-schema), Metarank tries to automate creation of these features for you. ## Mapping input events into ML features When Metarank receives a stream of events (both online during inference, and offline while training), it joins them together into a single view of visitor click chain: * For each ranking event, we do a per-item join of item metadata events (and also pull user metadata events) * All the interaction events like clicks and purchases are also joined together So each feature extractor has full access to complete view of the click chain. Then a sequence of differently scoped extractors take this view as an input and emit feature values in the following order: * item: uses only item metadata as a source * session: session-specific values * interaction: ones from interaction events ## Configuration All the feature extractors have a set of common fields: * `name`: *required*, *string*. Feature name, should be unique across the whole config. * `refresh`: *optional*, *time*, default value is specific to the extractor. How frequently this feature is updated. * `ttl`: *optional*, *time*, default: *90d* (3 months). How long should this feature store it's value. * `scope`: *optional*. See the [Scoping chapter](#scoping) for more information. ### Scoping Metarank uses scopes for computing and storing feature values. Feature values are stored not as atomic values but as a time-based changelog, where *scope* is a unique key for this particular instance of the ML feature. Metarank has four predefined types of scopes: * *global*: the feature is scoped globally , for example, air temperature outside the server room. * *item*: take the item id for a feature changelog, for example: an item popularity. * *user*: use user id as a unique id, for example: number of past sessions. * *session*: use session id as a unique id, for example: number of items in a cart right now. Scoping is also used for optimizing feature grouping while doing inference and model training: we compute all the ML features defined in the configuration for all candidate items, but some of the features are constant for the full ranking: if you have a site-global feature of air temperature outside, then it will be the same for all the items in the ranking, so no need to repeat the computation again. While building training data for the offline bootstrapping stage, this optimization technique can save a lot of raw processing power. ### Scoping example All ML feature extractors in Metarank have a `scope` option in their configuration. For example: ```yaml - name: popularity type: number field: item.popularity // must be a number scope: item refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` The `scope: item` means that the extracted popularity field from item metadata should be stored by an item identifier key. ## Feature types ### Generic feature extractors * [number](/reference/overview/feature-extractors/scalar#boolean-and-numerical-extractors): uses a raw numerical field value as a feature. * [vector](/reference/overview/feature-extractors/scalar#vector-extractor): reduces a variable length number vector to a fixed size value array. * [boolean](/reference/overview/feature-extractors/scalar#boolean-and-numerical-extractors): uses a raw boolean field as a 1 or 0 feature value. * [string](/reference/overview/feature-extractors/scalar#string-extractors): uses a raw string or list field as an input and does a one-hot encoding of it. * [word\_count](/reference/overview/feature-extractors/generic#word-count): how many words are in a string field. * [list\_size](/reference/overview/feature-extractors/generic#list-size): size of string or numerical list. * [time\_diff](/reference/overview/feature-extractors/generic#time-difference): difference in seconds between current timestamp and the numerical field value. * [field\_match](/reference/overview/feature-extractors/text#field_match): match ranking field over item fields. ### User session feature extractors * [ua/platform](/reference/overview/feature-extractors/user-session#user-agent-field-extractor): a one-hot encoded platform (mobile, desktop, tablet). * [ua/os](/reference/overview/feature-extractors/user-session#user-agent-field-extractor): a one-hot encoded OS (ios, android, windows, linux, macos, chrome os). * [ua/browser](/reference/overview/feature-extractors/user-session#user-agent-field-extractor)\* a one-hot encoded browser (chrome, firefox, safari, edge). * [interacted\_with](/reference/overview/feature-extractors/user-session#interacted-with): for the current item, did this visitor have an interaction with other item with the same field. * *\[coming soon]* ip\_country: a GeoIP-based country. * *\[coming soon]* ip\_city: a GeoIP-based city. * [referer\_medium](/reference/overview/feature-extractors/user-session#referer): a source of traffic for this customer. * *\[coming soon]* session\_length: length of the current visitor session in seconds. * *\[coming soon]* session\_count: number of total sessions tracked for this customer. ### Ranking feature extractors * [relevancy](/reference/overview/feature-extractors/relevancy#ranking): use a supplied per-product relevancy from the rerank request. * [position](/reference/overview/feature-extractors/relevancy#position): Item position in ranking. * [diversity](/reference/overview/feature-extractors/diversity): Search results diversification. ### Interaction feature extractors * [interaction\_count](/reference/overview/feature-extractors/counters#interaction-counter): number of interaction made within this session. * [window\_event\_count](/reference/overview/feature-extractors/counters#windowed-counter): sliding window count of interaction events. * [rate](/reference/overview/feature-extractors/counters#rate): rate of interaction events of type A over interaction events of type B. Useful for CTR/CVR rates. ### Date and time * [local\_time](/reference/overview/feature-extractors/datetime#local_time-extractor): map local visitor date-time to catch seasonality. * [item\_age](/reference/overview/feature-extractors/datetime#item_age): how much time passed since the item was updated? # Counters Event count is a nice and simple signal to affect ranking. But implementation-wise it is quite tricky: * counters are constantly changing in time, so to do proper model training and backtesting, you need to maintain a historical view on counter values. For example, if you're counting clicks over products, you need to know the resulting number on each point in time when each click happened. * global and always incrementing counters may work well for user/session scoped things, but counting clicks over products requires time windowing, as having 100 clicks over the whole lifetime is completely different from\ having the same 100 clicks, but for yesterday. In Metarank, there are two different types of counters implemented: * interaction\_counter: a simple global always incrementing counter on interactions. Good for counting number of clicks within session, cart size and so on. * window\_counter: when you need to time-frame all the events. Good for item popularities. ## Interaction counter Interaction counter is configured in a following way: ```yaml - name: click_count type: interaction_count scope: item // you can also count actions by a user/session interaction: click refresh: 60s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field if there were no updates ``` Refresh field can be useful for counters when you don't want to update the value used for inference frequently and want to limit the write throughput to feature store. ## Windowed counter Windowed counter has the same semantics as the interaction one, but is configured in a different way: ```yaml - name: clicks type: window_count interaction: click scope: item count: 10 // optional, take only the last 10 clicks performed by the user bucket_size: 24h // make a counter for each 24h rolling window windows: [7, 14, 30, 60] // on each refresh, aggregate to 1-2-4-8 week counts refresh: 60s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field if there were no updates ``` So this feature extractor will emit a group of following features: * clicks\_7: 12 * clicks\_14: 34 * clicks\_30: 70 * clicks\_60: 124 These feature values will be updated at least every 60 seconds. Window counters are implemented as a circular buffer of counters. There is an each separate counter for each time bucket, and there are as many time buckets as max window size. As counters are updated frequently, it can be computationally expensive to refresh them on each interaction, so it's usually worth it to limit the refresh rate to something reasonable like 10 minutes. There is also a way to combine multiple windowed counters into a [rate](#rate) to make streaming computation of CTR/Conversion rates easier. ## Rate `rate` feature extractor is useful to calculate things like conversion rate, click-through rate and other values where you need to divide one interaction counter by another. You can configure it in the following way: ```yaml - name: ctr type: rate # name of the feature extractor used as a dividend top: click # name of the feature extractor that is used as a divider bottom: impression // it's a special synthetic event generated by metarank. bucket: 24h periods: [7,30] scope: item # optional, default item, options: item, item., ranking. refresh: 1h # optional, how frequently we should update the value, 1h by default ttl: 90d # optional, how long should we store this field if there were no updates normalize: # optional, disabled by default weight: 10 ``` In this example, we use a `bottom: impression` type of interaction. It's a special synthetic interaction event generated by metarank over the items which were examined by the visitor. See [click models](https://github.com/metarank/metarank/blob/stabledoc/doc/click-models.md) for details. As for current version 0.5.x, rate feature can be only scoped to an item. ### Rate normalization Rate value can be quite noisy for cases with low number of events. Consider these two CTRs: * impressions=100, clicks=50. CTR=50% * impressions=2, clicks=1. CTR=50% Both of these CTRs have the same value, but have completely different confidence. Inspired by the talk [An approach to modelling implicit user feedback](https://haystackconf.com/us2022/talk-2/) from [HaystackUS22](https://haystackconf.com) conference, Metarank can normalize per-item rate based on a global prior rate: ``` rate = (a + item_clicks) / (b + item_impressions) ``` Where `a` and `b` can be defined as a normalization constants, where `a / b` is proportional to the global prior CTR rate. As an example, imagine that we have the following click statistics: * over all items: 100 impressions, 10 clicks. * over item A: 2 impressions, 1 click. Non-normalized CTR for item A is 50%, which is way above an average 10% CTR for the whole inventory. But we can mix global CTR with per-item CTR: ``` item_ctr = (10 total clicks + 1 click on item A) / (100 total impressions + 2 impressions on A) ``` To maintain a constant scaling factor `w`, we can define normalized per-item CTR in the following way: ``` item_ctr = (w + item_clicks) / (w * (global_impressions / global_clicks) + item_impressions) ``` Then, our example above will look in a more reasonable way: | weight | global impressions | global clicks | item impressions | item clicks | global CTR | item CTR | normalized item CTR | | ------ | ------------------ | ------------- | ---------------- | ----------- | ---------- | -------- | ------------------- | | 10 | 100 | 10 | 2 | 1 | 10.00% | 50.00% | 10.78% | | 1 | 100 | 10 | 2 | 1 | 10.00% | 50.00% | 16.67% | | 10 | 100 | 10 | 10 | 3 | 10.00% | 30.00% | 11.82% | | 10 | 20 | 5 | 10 | 3 | 25.00% | 30.00% | 26.00% | With this normalization approach, per-item CTR will have much fewer outlier values for cases with low number of clicks. Choosing the right value of `w` coefficient is dependent on your dataset: * typical range is between 5 and 10 * the more outlier CTRs you have, the higher `w` should be * default value is 10 ### Field-scoped rates #### Grouping by item field When the `scope: item.` parameter is set, rate feature can aggregate counters per a specific item field value. This is useful as a way to solve the cold-start problem, when there's not enough interactions made per item, but you can aggregate it per category/color/brand. So in a case when your inventory has a field `brand=cocacola`, you can compute CTR per each brand, using the following feature extractor: ```yaml - name: ctr type: rate top: click bottom: impression bucket: 24h periods: [7,30] scope: item.brand ``` * normalization is also supported * the target aggregating field should have low cardinality. * item field should have a string type, arrays are not supported. #### Grouping by ranking field Even more, you can also scope rates per ranking field AND item id! This can be useful when you have a lot of interactions and want to compute CTR within each query. So in a case if all your ranking events have a field `query=cola`, you can compute item CTR within each query: ```yaml - name: ctr type: rate top: click bottom: impression bucket: 24h periods: [7,30] scope: ranking.query ``` * normalization is also supported * the target aggregating field should have low cardinality. * ranking field should have a string type, arrays are not supported. # Date and Time ## local\_time extractor This extractor is useful when you need to parse a local date-time and get a time-of-day (or something similar) from there to catch a seasonality: maybe visitor behavior is different on morning and in evening? Given the event: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "localts", "value": "2021-12-03T10:15:30+01:00"} ], "items": [ {"id": "item3"}, {"id": "item1"}, {"id": "item2"} ] } ``` and the following feature config: ```yaml - name: time type: local_time parse: time_of_day // can be day_of_week/time_of_day/day_of_month/month_of_year/year/second source: ranking.localts // can only work with ranking event types, the field must be string with ISO-formatted zoned datetime ``` This extractor will pull the `10:15:30+01:00`, and map it into a `0..23.99` range, so one second before midnight will be 0.99, and midday will be 0.5. This extractor can use both a separate field or an event-level `ranking.timestamp` one. Supported `parse` field values: * `day_of_week`: day number in 1..7 range, where Monday is 1 * `time_of_day`: local time in 0.0..23.99 range * `day_of_month`: day of current month in 1..31 range * `month_of_year`: current month in 1..12 range * `year`: absolute current year value * `second`: current local timestamp in seconds from epoch start ## item\_age Sometimes it can be useful to know how fresh is the item in the ranking? Consider the following item metadata event: ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "item": "product1", "timestamp": "1599391467000", "fields": [ {"name": "created_at", "value": "2021-12-03T10:15:30+01:00"} ] } ``` It's possible to compute how much time has passed from the `created_at` field value till now, with the following config snippet: ```yaml - name: freshness type: item_age source: item.created_at // can only work with item metadata event types refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` The `source` field should have any of the following types: * `string`, ISO8601 date+time+timezone, example: "2021-12-03T10:15:30+01:00" * `number`, unixtime (number of seconds from epoch start), example `1648483661` * `string`, unixtime as a string (so there will be no json number rounding), example: `"1648483661"` * you can reference the event timestamp (not the free-form field, but the native top-level event timestamp) with a `item.timestamp` # Generic ## Word count Sometimes it can be useful to get the word length of a field, especially for models designed to personalize different types of content. You can use the `word_count` feature extractor to get the length of a string field with the following config: ```yaml - name: title_length type: word_count scope: item field: item.title // must be a string ``` ## Relative number > Update, v0.7.x: relative\_number is deprecated and removed. Both XGBoost and LightGBM natively support this out of the box for all numeric features, so please use the [number](/reference/overview/feature-extractors/scalar#numerical-extractor) feature. ## List size Counts the number of items in a string or numerical list. Example: ```yaml - name: toggled_filters_count type: list_size field: filters source: item ``` # Relevancy ## Ranking While implementing Learn-to-Rank systems, Metarank is designed to be a satellite secondary reranking system. It assumes that there exists another service which generates candidates for the reranking process: * in search: Elasticsearch or SOLR * in recommendations: output of spark mmlib ALS recommender * in ecommerce: inventory database Most of these primary sources of input may also have a per-item score: how much this item is matching the original query: * BM25 or TF/IDF score in search * cosine difference between embeddings in recommendations Metarank [ranking event schema](/reference/event-schema) allows adding a per-item fields, which can be used for relevance score, see the example: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "query", "value": "cat"}, {"name": "source", "value": "search"} ], "items": [ {"id": "item3", "fields": [{"name": "relevancy", "value": 2.0}]}, {"id": "item1", "fields": [{"name": "relevancy", "value": 1.0}]}, {"id": "item2", "fields": [{"name": "relevancy", "value": 0.1}]} ] } ``` This per-item "relevancy" field is the one holding information about score from the upstream ranking system, like BM25 score. Metarank <= 0.5.11 included now deprecated `relevancy` extractor. With Metarank 0.5.12+ you can use regular [`number`](/reference/overview/feature-extractors/scalar#numerical-extractor) extractor for this case: ```yaml - name: relevancy type: number scope: item field: item.relevancy ``` ### Multiple retrievers As there can be multiple per-item fields in the ranking event, it means that it's also possible to have multiple first-level relevancy signals. For example, when you have a hybrid search application with two retrievals: * Elasticsearch/OS/Solr for regular term search, giving you a per-document BM25 score. * PineCone/Vespa/QDrant/etc. vector search engine, doing a k-NN lookup over neural-based query embedding, giving you cosine similarity score. The app retrieves top-N documents from both sources, and then merges them together in a single list. Some documents may come from both retrievers, and some - only from one. Then your ranking event may look like: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "query", "value": "cat"} ], "items": [ {"id": "item3", "fields": [{"name": "bm25", "value": 2.0}]}, {"id": "item1", "fields": [ {"name": "bm25", "value": 1.0}, {"name": "cos", "value": 0.75} ]}, {"id": "item2", "fields": [{"name": "cos", "value": 0.02}]} ] } ``` In this case, `item1` retrieved by both search engines (so there are two relevancy factors: `bm25` and `cos`), `item2` is coming only from vector search, and `item3` only from term search engine. Not supplying one of relevancy scores is possible, as underlying ML model implementations both support handling missing values (see docs for [XGBoost](https://xgboost.readthedocs.io/en/stable/faq.html#how-to-deal-with-missing-values) and [LightGBM](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#missing-value-handle) on how it's done under the hood). ## Position A bias elimination technique based on a paper [PAL: a position-bias aware learning framework for CTR prediction in live recommender systems](https://www.researchgate.net/publication/335771749_PAL_a_position-bias_aware_learning_framework_for_CTR_prediction_in_live_recommender_systems). * on offline training, the feature value equals to the item position in the ranking * on online inference, it is equals to a constant position value for all items. The main idea of such approach is that the underlying ML model will learn the impact of position on ranking, but then, setting all items position factors to the same constant value, you make it look like from the model point-of-view that all items are located on the same position. So position has constant impact on the ranking for all the items. To configure this feature extractor, use the following YAML snippet: ```yaml - type: position name: position position: 5 ``` To choose the best `position` value: * Start with a value around middle of your average ranking length. So if you present 20 items, set it to 10. Usually it's already a good number for most of the cases. * Do a grid search of the best value around it with `metarank standalone`. Select the best `position` based on offline NDCG value. # Scalars The most typical use case of mapping data from incoming events to ML features is to use them as is, without any transformations. Metarank has a set of basic extractors to simplify the process even more: * `boolean`: take a true/false field and map it to 1 and 0 * `number`: take a number and use it as is * `string`: do a one-hot encoding of low-cardinality string contents of the field ## Boolean and numerical extractors Consider this type of incoming event, emitted by your backend system when a product goes in stock, or it's price changes: ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "item": "product1", "timestamp": "1599391467000", "fields": [ {"name": "availability", "value": true}, {"name": "price", "value": 69.0} ] } ``` We can add the following extractor, so it will use the availability data for the ranking: ```yaml - name: availability type: boolean scope: item field: item.availability // must be a boolean refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` In practice, you can use not only fields from item metadata events, but also from ranking. An example for ranking events: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "banner_examined", "value": true} ], "items": [ {"id": "product3", "fields": [{"name": "relevancy", "value": 2.0}]}, {"id": "product1", "fields": [{"name": "relevancy", "value": 1.0}]}, {"id": "product2", "fields": [{"name": "relevancy", "value": 0.1}]} ] } ``` So you can extract this `banner_examined` value using the following config: ```yaml - name: banner_examined type: boolean scope: item field: ranking.banner_examined ``` It is also possible to extract per-item fields from the ranking event. For example, the `relevancy` field can be extracted this way: ```yaml - name: relevancy type: number scope: item field: ranking.relevancy ``` Extracting fields from interaction events is not possible, as at the moment of ranking request happening, there are no interactions happened yet, they will happen in the future. ## Numerical extractor With the same approach as for boolean extractor, you can pull a `price` field out of a item metadata message into the explicit feature: ```yaml - name: price type: number field: item.price // must be a number scope: item refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` So the last price will be present in the set of ML features uses in the ranking. It's also possible to use `user` fields in a case when you have some pre-existing information about the visitor (for example, when visitor filled a form before). Then with this `user` event: ```json { "event": "user", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "user": "user1", "timestamp": "1599391467000", "fields": [ {"name": "age", "value": 30} ] } ``` You can map the `age` field into a feature this way: ```yaml - name: user_age type: number field: user.age // must be a number scope: user refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` ## Vector extractor Numerical vectors require special handling: their dimension is not statically known (or they can be empty), so we need to perform a set of transformations to reduce these to a static size used inside the ML model. For example, given an item with field `sizes: [10, 12, 13]`, we can use a `vector` extractor with the following configuration: ```yaml - name: sizes type: vector field: item.sizes // must be a singular number or a list of numbers scope: item # which reducers to use. optional. Default: [min, max, size, avg] reduce: [first, last, min, max, avg, random, sum, size, euclidean_distance, vectorN] refresh: 0s # optional, how frequently we should update the value, 0s by default ttl: 90d # optional, how long should we store this field ``` Supported reducers are: * `first`/`last`/`min`/`max`/`random` - take first/last/min/max/random element of the list, or zero if empty. * `avg`/`sum` - compute mean value or sum of values. * `euclidean_distance` - compute a Euclidean distance of numerical vector, which is a squared root of sum of squares. * `vectorN` - take first N items from the sequence, and pad remaining with zeroes. So `vector10` means a vector of 10 dimensions. The `vectorN` reducer can also be useful if you compute embeddings (fields with constant predefined size) for your user/items as you can wrap them as ranking features directly. For example, when your item has a field `als_embedding: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]`, you can define a `vector` feature with `reduce: vector10` and the raw embedding will be short-cirquited as a set of 10 separate numerical features for the ranking model. ## String extractors With string values there is no easy way to map them into a finite set of ML features. But in a case when the string has low cardinality (so there is a finite and low number of possible values), we have a couple of options on how to treat these: * [one-hot encoding](https://en.wikipedia.org/wiki/One-hot) to convert it to a number vector. * Index encoding, which may work better when cardinality is high (e.g. > 10). ### One-hot encoding Imagine you have field `color: "red"` and there is only a small finite set of possible values for this field: it can be either red, green or blue. So we can do the actual mapping in the following way: ```yaml - name: color type: string scope: item encode: onehot // optional, default = index, options = onehot | index values: [red, green, blue] field: item.color // must be either a string, or array of strings ``` This snippet will emit the following ML feature group for a `color: "red"` input: * color\_red: 1 * color\_green: 0 * color\_blue: 0 The underlying string field can also be an array of strings like `color: ["red", "blue"]`, which will toggle two bits instead of one in the resulting vector. ### Index encoding One-hot encoding does not suit the cases, when your list has high cardinality (more than 10 distinct values, e.g. country list) as the dimensionality of the training dataset can fly into the sky (you can have tens or even hundreds of model fields that represent just one feature). For such use-cases it is much more effective to use index encoding. * LightGBM backend supports proper split selection for categorical features. You can check out the [LightGBM documentation](https://lightgbm.readthedocs.io/en/latest/Features.html#optimal-split-for-categorical-features) for more details. * XGBoost itself supports it, but it's not yet exposed in the Java wrapper, so it will treat index-encoded category as a regular numeric feature. ```yaml - name: color type: string scope: item encode: index // optional, default = onehot, options = onehot | index values: [red, green, blue] field: item.color // must be either a string, or array of strings ``` This snippet will emit the following ML feature group for a `color: "green"` input: * color: 2 Please note the **limitations** of the index encoder: * Index encoder can only work with singular field values, so if it spots multiple colors, only the first value from the array will be used. * empty values are encoded as *zero*, existing - starting from *one*. ### Index vs one-hot, what to choose? In common scenarios: * index encoding is always faster than one-hot one due to lower dataset dimensionality on tree-based backends (e.g. LightGBM and XGBoost) * index encoding results in same or better NDCG metric on LightGBM backend, compared to one-hot * on XGBoost usually results in the similar NDCG, but better result is not guaranteed. If you're not sure what to choose - prefer index encoding, the default option. # Text ## field\_match An extractor which can match a field from ranking event over an item field. In practice, it can be useful in search related tasks, when you need to match a search query over multiple separate fields in document, like title-tags-category. Field match extractor supports the following matching methods: * BM25: a Lucene-specific BM25 score between ranking and item fields (for example, between query and item title) * ngram: split item/query fields to N-grams and compute intersection over union score * term: use Lucene to perform language specific tokenization * bert: build LLM embeddings for item/query fields and compute a distance between them ### Dataset example Given this metadata event: ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "item": "item1", "timestamp": "1599391467000", "fields": [ {"name": "title", "value": "red socks"}, {"name": "category", "value": "socks"}, {"name": "brand", "value": "buffalo"}, {"name": "description", "value": "lorem ipsum dolores sit amet"} ] } ``` And a following ranking event: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "query", "value": "sock"} ], "items": [ {"id": "item3"}, {"id": "item1"}, {"id": "item2"} ] } ``` ### BM25 score As BM25 formula requires term frequencies and some other index statistics, using BM25 requires you to build the term-freq dictionary beforehead, [see the CLI `termfreq` docs](/reference/cli#bm25-term-frequencies-dictionary) on how to do it. Having the `term-freq.json` file in hand, you can then configure Metarank to compute BM25 score between ranking field (for example, `query`) and item field (like `title`): ```yaml - name: title_match type: field_match rankingField: ranking.query itemField: item.title method: type: bm25 language: english termFreq: "/path/to/term-freq.json" ``` ### Ngram matching With the following config file snippet you can do a per-field matching of `ranking.query` field over `item.title` field of the items in the ranking with 3-grams: ```yaml - name: title_match type: field_match itemField: item.title // must be a string rankingField: ranking.query // must be a string method: type: ngram // for now only ngram and term are supported language: en // ISO-639-1 language code n: 3 refresh: 0s // optional, how frequently we should update the value, 0s by default ttl: 90d // optional, how long should we store this field ``` ### Term matching In a similar way you can do the same with term matching: ```yaml - name: title_match type: field_match itemField: item.title // must be a string rankingField: ranking.query // must be a string method: type: term // for now only ngram and term are supported language: en // ISO-639-1 language code ``` Both term and ngram matching methods leverage Lucene for text analysis and support the following set of languages: * *generic*: no language specific transformations * *en*: English * *cz*: Czech * *da*: Danish * *nl*: Dutch * *et*: Estonian * *fi*: Finnish * *fr*: French * *de*: German * *gr*: Greek * *it*: Italian * *no*: Norwegian * *pl*: Polish * *pt*: Portuguese * *es*: Spanish * *sv*: Swedish * *tr*: Turkish * *ar*: Arabic * *zh*: Chinese * *ja*: Japanese Both term and ngram method share the same approach to the text analysis: * text line is split into terms (using language-specific method) * stopwords are removed * for non-generic languages each term is stemmed * then terms/ngrams from item and ranking are scored using intersection/union method. ### LLM Bi-Encoders This text matching method: * computes an embedding for both query and document * then computes a cosine between both embeddings. Semantically-similar query-document pairs will have higher score than irrelevant ones. Then with the following config snippet we can compute a cosine distance between title and query embeddings: ```yaml - type: field_match name: title_query_match rankingField: ranking.query itemField: item.title distance: cos # optional, default cos, options: cos/dot method: type: bi-encoder model: metarank/all-MiniLM-L6-v2 # optional, can be only cache-based dim: 384 # required, dimensionality of the embedding itemFieldCache: /path/to/item.embedding # optional, pre-computed embedding cache for items rankingFieldCache: /path/to/query.embedding # optional, pre-computed embedding cache for rankings ``` Metarank supports two embedding methods: * `bi-encoder`: ONNX-encoded versions of the [sentence-transformers](https://sbert.net/docs/pretrained_models.html) models. See the [metarank HuggingFace namespace](https://huggingface.co/metarank) for a list of currently supported models. * `cross-encoder`: ONNX-encoded versions of [sentence-transformers](https://sbert.net/docs/pretrained_models.html) For both models, Metarank supports fetching model directly from the HuggingFace Hub, or loading it from a local dir, depending on the model handle format: * `namespace/model`: fetch model from the HFHub * `file://///`: load ONNX-encoded embedding model from a local file. #### Using CSV cache of precomputed embeddings In some performance-sensitive cases you don't want to compute embeddings in realtime, but only use offline precomputed ones. This is possible with the `csv` `field_match` method: ```yaml - type: field_match name: title_query_match rankingField: ranking.query itemField: item.title distance: cos # optional, default cos, options: cos/dot method: type: bi-encoder # note that there is no model reference, only caches dim: 384 itemFieldCache: /path/to/item.embedding rankingFieldCache: /path/to/query.embedding ``` In this case Metarank will load item and query embeddings from a CSV file in the following format: itemFieldCache: ``` item1,0,1,2,3,4,5 item2,5,4,3,2,1,9 item3,1,1,1,1,1,1 ``` rankingFieldCache: ``` bananas,0,1,2,3,4,5 red socks,5,4,3,2,1,9 stone,1,1,1,1,1,1 ``` * when both query and item embeddings are present, then `field_match` will produce a cosine distance between them. * when at least one of the embeddings is missing, then `field_match` with `csv` method will produce a `nil` missing value. * when at least one of the embeddings is missing, then `field_match` with `transformer` method will compute the embedding real-time. ### LLM Cross-encoders Cross-encoders are quite similar to bi-encoders, but instead of separately computing embedding for query and document, they feed both texts to the neural network, which produces the matching score. Compared to the bi-encoder approach: * cross-encoders are much more precise, even generic pre-trained models. * require much more resources, as there's no way to pre-compute embeddings for docs and queries - you need to perform a full neural inference query-time. Enabling cross-encoders in Metarank can be done with the following snippet: ```yaml - type: field_match name: title_query_match rankingField: ranking.query itemField: item.title method: type: cross-encoder model: metarank/ce-msmarco-MiniLM-L6-v2 # optional, can be only cache-based cache: /path/to/ce.cache # optional, pre-computed query-doc scores ``` Note that as cross-encoders are very CPU heavy to run, you can pre-compute a set of query-doc scores offline and supply Metarank with a cache in the following CSV format: ``` query1,doc1,0.7 query1,doc2,0.1 query2,doc3,0.2 ``` # User Profile ## User-Agent field extractor A typical HTTP User-Agent field has quite a lot of embedded meta information, which can be useful for ranking: * is it mobile or desktop? Mobile visitors behave differently compared to desktop ones as they scroll less and get distracted quicker. * iOS or Android? Assuming that on average Apple devices are more expensive than Android ones, it can also provide more insights on visitor goals. * Stock browser or something custom? * How old is the OS? On Android, an ancient version of OS can mean an old and unsupported device, so it can be also a signal on your ranking. But User-Agent string is quite cryptic: ``` Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.47 Mobile/15E148 Safari/604.1 Mozilla/5.0 (Linux; Android 10; LM-Q720) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.48 Mobile Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15 Mozilla/5.0 (iPhone; CPU iPhone OS 15_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Mobile/15E148 Safari/604.1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/98.0.1108.62 ``` There is a large collaborative effort to build a database of typical UA patterns, (UA-Parser)\[], which is used to extract all the possible item metadata from these strings. To map this to actual ML features, there is a predefined set of mappers: * platform: mobile, desktop, tablet * os: ios, android, windows, linux, macos, chrome os * browser: safari, chrome, firefox, opera, ie, other * bot: is it a known crawler or not To configure the extractor, use this YAML snippet: ```yaml - name: platform_feature // just a name of this feature type: ua # take the UA field from ranking event source: "ranking.ua" # options: platform, os, browser, bot field: "platform" # optional, how frequently we should update the value refresh: 0s # optional, how long should we remember this field ttl: 90d ``` The UA field is taken from each ranking request, so it should be always present. ## Interacted with For the current item, did this visitor have an interaction with other item with the same field? Example: ```yaml - name: clicked type: interacted_with # type of the interaction event (interaction.type field) interaction: click field: [ item.color ] # the field must be a string or string[], # and only works with item fields. # session/user scope: user ``` For this example, Metarank will track all color field values for all items visitor clicked and intersect this set with per-item field values in the ranking. `interacted_with` extractor can also track multiple fields at once within a single visitor profile: ```yaml - name: clicked type: interacted_with interaction: click field: [ item.color, item.tags, item.brand ] # multiple fields at once scope: user ``` ## Referer For user/ranking/interaction events it's possible to parse a HTTP Referer field and extract the source medium. We use a [snowplow referer parser](https://s3-eu-west-1.amazonaws.com/snowplow-hosted-assets/third-party/referer-parser/referers-latest.json) parsing library, so it defines 6 types of referer mediums: unknown, search, internal, social, email, paid. For a ranking event: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "referer", "value": "http://www.google.com"} ], "items": [ {"id": "item1"}, {"id": "item2"} ] } ``` and a configuration: ```yaml - name: referer_medium type: referer source: ranking.referer scope: user // can be user/session ``` It will detect that it's a "search" medium and one-hot-encode it to `[0, 1, 0, 0, 0, 0]`. A source field can be of a user/ranking/interaction type, and feature extractor memorises all the referer fields ingested: * it matches the HTTP Referer semantics, as referer field is sent on each request * there can be multiple referers. For example, visitor lands on a site from google (and gets a "search" referer), then does a couple of interactions with the site (and also gets an "internal" referer medium) In a case when a visitor has multiple referers memorized, then the one-hot-encoded vector will have multiple flags enabled, like `[0, 1, 1, 0, 0, 0]` for a case with search+internal referer mediums. # Diversification ## diversity Computes how different your current ranking item is compared to other items within the same ranking. Numeric and string fields are supported. ### Diversification over numeric fields Consider that all items in your inventory have a numeric `price` field: ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "item": "item1", "timestamp": "1599391467000", "fields": [{"name": "price", "value": 69.0}] } ``` Then for a ranking below: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "items": [ {"id": "item1"}, {"id": "item2"}, {"id": "item3"} ] } ``` we can compute how different each item price compared to the median price across the whole ranking with the following configuration snippet: ```yaml - name: price_diff type: diversity source: item.price # only item.* fields are accepted ttl: 90d # optional, when to expire tracked fields top: 10 # optional, take only top-N items to compute the median ``` For example, given the following item prices: * p1: price=100 * p2: price=200 * p3: price=250 * p4: price=300 * p5: price=220 So for a ranking `[p1, p2, p3, p4, p5]` we compute a median value of 220, and then compute the difference: * p1: price\_diff=-120 * p2: price\_diff=-20 * p3: price\_diff=30 * p4: price\_diff=80 * p5: price\_diff=0 When you have a very long ranking, it's worth to consider limiting the amount of items taken into account, when computing median. When setting `top=3`, for the same set of items in the ranking event above, you'll get the median of 200: * p1: price\_diff=-100 * p2: price\_diff=0 * p3: price\_diff=50 * p4: price\_diff=100 * p5: price\_diff=20 ### Diversification over string fields This type of diversification can be useful to see how different your items over low-cardinality fields like tags, colors, sizes and categories. Both `string` and `string[]` field types are supported. When all your inventory items have a field `color` like in an example below: ```json { "event": "item", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "item": "item1", "timestamp": "1599391467000", "fields": [{"name": "color", "value": "red"}] } ``` Then for a ranking below: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "items": [ {"id": "item1"}, {"id": "item2"}, {"id": "item3"} ] } ``` we can compute how frequently each color is presented in the result set with the following configuration snippet: ```yaml - name: color_diff type: diversity source: item.color # only item.* fields are accepted ttl: 90d # optional, when to expire tracked fields top: 10 # optional, take only top-N items to compute the histogram ``` The difference algorithm builds tag frequencies over the ranking (so `color -> count` in our example above), and then computes relative intersection between tags of item and tag frequencies. An example: * given a frequency of {red: 50%, green: 30%, blue: 20%} * for an item having only red color, the score will be 50%. * for a red-blue item, the score will be 50%+20%=70% # Recommendations Starting from version `0.6.x`, Metarank supports three types of recommendations: * [Trending](/reference/overview/recommendations/trending): popularity-sorted list of items with customized ordering. * [Similar items](/reference/overview/recommendations/similar): matrix-factorization collaborative filtering recommender of items you may also like. * [Semantic](/reference/overview/recommendations/semantic): a content-based semantic similarity recommender, based on neural embeddings. # Trending items `trending` recommendation model is used to highlight the trending (or in other workds, most popular) items in your application. But it's not just about sorting items by popularity! Metarank can: * combine multiple types of interactions: you can mix clicks, likes and purchases with different weights. * time decay: clicks made yesterday are much more important than the clicks from the last months. * multiple configurations: trending over the last week, and bestsellers over the last year. ## Configuration A separate block in the `models` section: ```yaml models: yolo-trending: type: trending weights: - interaction: click decay: 0.8 # optional, default 1.0 - no decay weight: 1.0 # optional, default 1.0 window: 30d # optional, default 30 days - interaction: like decay: 0.9 weight: 1.5 window: 60d - interaction: purchase decay: 0.95 weight: 3.0 ``` The config above defines a trending model, accessible over the `/recommend/yolo-trending` [API endpoint](/reference/api): * the final item score combines click, like and purchase events * purchase has 3x more weight than click, like has 1.5x more weight than click * purchase has less agressive time decay * only the last 30 days of data are used for clicks and purchases, but 60 days are used for likes ## Time decay and weight The final score used to sort the items is defined by the following formula: ``` score = count * weight * decay ^ days_diff(now, timestamp) ``` When multiple interaction types are defined, per-type scores are added together to get the final score. Time decay configuration allows a granular control over the decaying. Here's a click importance is weighted for different `decay` values: ![decay with different options](/files/hy6ketPC0OW3Qf71bBNi) We recommend setting decay: * within a range of 0.8-0.95 for 1-month periods. * within a range of 0.95-0.99 for larger periods. See request & response formats in the [API section](/reference/api#recommendations). # Similar items `similar` recommendation model can give you items other visitors also liked, while viewing the item you're currently observing. Common use-cases for this model are: * you-may-also-like recommendations on item page: the context of the recommendation is a single item you're viewing now. * also-purchased widget on the cart page: the context of the recommendation is the contents of your card. ## Configuration ```yaml similar: type: als interactions: [click, like, purchase] # which interactions to use factors: 100 # optional, number of implicit factors in the model, default 100 iterations: 100 # optional, number of training iterations, default 100 ``` There are two important parameters in the configuration: * `factors`: how many hidden parameters the model tries to compute. The more - the better, but slower. Usually defined within the rage of 50-500. * `iterations`: how many factor refinement attempts are made. The more - the better, but slower. Normal range - 50-300. Rule of thumb - set these parameters low, and then increase slightly until training time becomes completely unreasonable. See request & response formats in the [API section](/reference/api#recommendations). ## Underlying model Metarank uses a variation of [Matrix Factorization](https://developers.google.com/machine-learning/recommendation/collaborative/matrix) collaborative filtering algorithm for recommendations based on the [Fast Matrix Factorization for Online Recommendation with Implicit Feedback](https://arxiv.org/abs/1708.05024) by X.He, H.Zhang, MY.Kan and TS.Chua. ![matrix factorization](/files/3FECuAyWg8WWB6OWwsbq) The ALS family of algorithms for recommendations decomposes a sparse matrix of user-item interactions into a set of smaller dense vectors of implicit user and item features (or user and item embeddings). The cool thing about these embeddings is that similar items will have similar embeddings! So Metarank does the following: * computes item embeddings. * pre-builds a [HNSW](https://www.pinecone.io/learn/hnsw/) index for fast lookups for similar embeddings. * during inference (when you call the [/recommend/modelname](/reference/api#recommendations) endpoint), it makes a k-NN index lookup of similar items. Main pros and cons of such apporach: * *pros*: fast even for giant inventories, simple to implement * *cons*: lower precision compared to neural networks based methods like [BERT4rec](https://arxiv.org/abs/1904.06690), recommendations are not personalized. *There is an ongoing work in Metarank project to implement NN-based methods and make current ALS implementation personalized.* # Semantic similarity `semantic` is a content recommendation model, which computes item similarity only based on a difference between neural embeddings of items. This model is useful for solving a cold-start problem of recommendations, as it requires no user feedback. ## Configuration ```yaml - type: semantic encoder: type: bert model: metarank/all-MiniLM-L6-v2 dim: 384 # embedding size itemFields: [title, description] ``` * itemFields: fields which should be used for embedding * encoder: a method of computing embeddings Metarank has quite limited support for embeddings: * `bert` type of embeddings only supports ONNX-encoded models from sentence-transformers from HuggingFace * `csv` type of embeddings allows loading a custom pre-made dictionary. ```yaml - type: semantic encoder: type: csv dim: 384 # embedding size path: /opt/dic.csv itemFields: [title, description] ``` A dictionary should be a comma-separated CSV-formatted file, where: * 1st column is product id * 2 till N+1 columns - float values for N-dimentional embedding Example: ``` p1,1.0,2.0,3.0 p2,2.0,1.5,1.0 ``` # Models This document lists all the methods Metarank may use for ranking. It's the one defined in the `models..type` part of [config file](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/sample-config.yml): ```yaml models: default: type: lambdamart ``` ## LambdaMART LambdaMART is a Learn-to-Rank model, optimizing the [NDCG metric](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). There is a [Lambdamart in Depth](https://softwaredoug.com/blog/2022/01/17/lambdamart-in-depth.html) article by [Doug Turnbull](https://softwaredoug.com) describing all the details about how it works. In a simplified way, LambdaMART in the scope of Metarank does the following: 1. Takes a ranking and some relevancy judgements over items as an input (judgements can be implicit, like clicks, or implicit like stars in movie recommendations) 2. All items in the ranking have a set of characteristics (ML features, like genre or CTR as an example) 3. A pair of items from the ranking is sampled. 4. ML model must be able to guess which item in this pair may have higher relevancy judgement. 5. Repeat over all pairs in the ranking. 6. Repeat over all the rankings in the dataset. At the end, items with higher judgements should be ranked higher, making the resulting ranking more relevant. In Metarank there are two supported library backends implementing this algorithm: * XGBoost: [rank:pairwise](https://xgboost.readthedocs.io/en/stable/parameter.html) objective * LightGBM: [lambdarank](https://lightgbm.readthedocs.io/en/latest/Parameters.html) objective To configure the model, use the following snippet: ```yaml : type: lambdamart backend: type: xgboost # supported values: xgboost, lightgbm iterations: 100 # optional (default 100), number of interations while training the model seed: 0 # optional (default = random), a seed to make training deterministic weights: # types and weights of interactions used in the model training click: 1 # you can increase the weight of some events to hint model to optimize more for them features: # features from the previous section used in the model - foo - bar # selector: # optional set of selectors to filter events for this specific model # rankingField: source # value: search # split: optional definition of train/test splitting strategy. See below for examples. # eval: optional list of evaluation metrics. # warmup: optional API warmup settings # sampledRequests: 100 # how many requests to sample during training # duration: 5s # how long to perform the warmup. ``` * `backend`: *required*, *xgboost* or *lightgbm*, specifies the backend and it's configuration. * `weights`: *required*, *list of string:number pairs*, specifies what interaction events are used for training. You can specify multiple events with different weights. * `features`: *required*, *list of string*, features used for model training, see [Feature extractors](/reference/overview/feature-extractors) documentation. * `selector`: *optional*, *list of selectors*, a set of rules to filter which events should be accepted by this model. * `split`: *optional*, a train/test splitting strategy. Default: `time=80%`. Options: `random`/`hold_last`/`time` with an optional ratio (default: 80%, which means 80% allocated to train, 20% to test). Example: `random=80%` means split dataset randomly, 80% should be allocated to the train set. * `eval`: *optional*, a list of eval metrics to measure after training. Default value is `["NDCG@10"]`, supported metrics are `NDCG`, `NDCG@k`, `MAP`, `MAP@k`, `MRR` (where `k` - cutoff value). * `warmup`: *optional*, API warmup settings. See the [API warmup section](/reference/deployment-overview/warmup) for details. ### Interaction weight Interactions define the way your users interact with the items you want to personalize, e.g. `click`, `add-to-wishlist`, `purchase`, `like`. Interactions can be used in the feature extractors, for example to calculate the click-through rate and by defining `weight` you can control the optimization goal of your model: do you want to increase the amount of likes or purchases or balance between them. You can define interaction by `name` and set `weight` for how much this interaction affects the model: ```yaml click: 1.0 ``` ### Event selectors When serving multiple models, there are cases when you need to separate ranking and interaction events per model. This is useful when your models have different contexts, e.g. you do personalized ranking for search results and recommendation results using the same Metarank installation, but utilizing different models. Metarank supports `selector` configuration that can be used to route your events to correct model or drop events in certain scenarios. Metarank supports the following event selectors: * **Accept selector**. Enabled by default to accept all events, if no selectors are defined. ```yaml selector: accept: true # true = accept all, false = reject all ``` * **Field selector**. Accepts event when it has a specific string (or string-list) field defined for a ranking event. For example: ```yaml selector: rankingField: source value: search ``` The filter above will accept only events that have the `source=search` field defined in the `fields` section of the event: ```json { "event": "ranking", "id": "81f46c34-a4bb-469c-8708-f8127cd67d27", "timestamp": "1599391467000", "user": "user1", "session": "session1", "fields": [ {"name": "source", "value": "search"} ], "items": [ {"id": "item1", "fields": [{"name": "relevancy", "value": 1.0}]}, ] } ``` * **Sampling selector**: randomly accept or drop an event, depending on the acceptance ratio: ```yaml selector: ratio: 0.5 ``` The sampling selector above will accept only half of events randomly. * **Max interaction position selector**: only accept click-through events when interaction position is not too high/low. Can be useful to exclude visitor sessions with discovery-style browsing behavior, or too short rankings. ```yaml selector: maxInteractionPosition: 10 minInteractionPosition: 3 # both fields are optional ``` * **Ranking length selector**: only accept click-through events with number of items within a defined range. ```yaml selector: minItems: 10 # so there should be at least 10 items in the ranking maxItems: 20 # both min and max are optional ``` * **AND/OR/NOT selector**: combine multiple selectors within a single boolean combination: ```yaml selector: and: - rankingField: source value: search - or: - rankingField: segment value: test - ratio: 0.5 - not: accept: false ``` AND and OR selectors take a list of nested selectors as arguments, NOT selector only takes a single selector argument. ### Train/test splitting strategies Metarank supports three train/test splitting strategies: * `random`: split dataset randomly. * `hold_last`: for each session having multiple rankings, take last N% of rankings as a test set. Can be useful to measure an in-session personalization impact . * `time`: split dataset by a timestamp. Each strategy definition in a config file can be optionally configured with a split ratio - 80% by default. An example: * `random=80%`: split dataset randomly. Be careful with random splitting, as it may introduce label leaking. * `hold_last`: split within session with a default 80% splitting ratio. ### XGBoost and LightGBM backend options * *iterations*: *optional*, *number*, default: *100*, number of trees in the model. * *learningRate*: *optional*, *number*, default: *0.1*, higher the rate - faster training - less precise model. * *ndcgCutoff*: *optional*, *number*, default: *10*, only N first items may affect the NDCG. * *maxDepth*: *optional*, *number*, default: *8*, the depth of the tree. * *seed*: *optional*, *string* or *number*, default: *random* to make model training deterministic. * *sampling*: *optional*, default: 0.8, fraction of features used to build a tree, useful to prevent over-fitting. * *debias*: *optional*, default: false. Enable booster-native position bias removal support. See these two articles about the unbiased LTR for [XGBoost](https://xgboost.readthedocs.io/en/latest/tutorials/learning_to_rank.html#position-bias) and [LightGBM](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#support-for-position-bias-treatment) for details. LightGBM also supports these specific options: * *numLeaves*: *optional*, *number*, default: *16*, how many leaves the tree may have. Please consult [LightGBM](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) and [XGBoost](https://xgboost.readthedocs.io/en/stable/parameter.html) docs about tuning these parameters. ## Shuffle A `shuffle` is a baseline model, which may be used in the a/b tests as a "worst-case" ranking scenario, when the order of items is random. The shuffle model is configured in the following way: ```yaml : type: shuffle maxPositionChange: 5 ``` * Parameter `maxPositionChange` controls the amount of randomness that shuffle can introduce in the original ranking. In other words, `maxPositionChange` sets how far away an item can drift from its original position. ## Noop A `noop` is also a baseline model, which does nothing. The main purpose of this model to be a baseline of the original ranking sent to metarank during a/b tests. It's configured with the following snippet: ```yaml : type: noop ``` It has no options and does not do any modifications to the ranking, just bouncing it back as-is. # Data Sources Metarank has two data processing stages: * *import*: consume historical visitor interaction events and produce latest point-in-time snapshot of the system and all the ML features. * *inference*: continues processing live events that come realtime. An overview diagram of event flow during inference/bootstrap is shown below: ![bootstrap and inference event flow](/files/9y5ebXaeKIvHJsoHp0oY) Metarank supports the following list of connectors: | Name | Import | Inference | | ------------------------------------------- | ----------------------- | --------- | | [Apache Kafka](#apache-kafka) | yes | yes | | [Apache Pulsar](#apache-pulsar) | yes | yes | | [AWS Kinesis Streams](#aws-kinesis-streams) | yes, but actually no \* | yes | | [RESTful API](#rest-api) | yes | yes | | [Files](#files) | yes | no | `*` AWS Kinesis has a strict max 7 days retention period, so if you want to store more than 7 days of historical clickthrough events, choose something else (for example, add AWS Kinesis Firehose to write events from Kinesis topic to S3 files to pick them with `Files` connector). ## Common options for bootstrapping connectors All supported connectors have some shared options: * offset: a time window in which events are read * `earliest` - start from the first stored message in the topic * `latest` - consume only events that came recently (after Metarank connection) * `ts=` - start from a specific absolute timestamp in the past * `last=` - consume only events that happened within a defined relative duration (duration supports the following patterns: `1s`, `1m`, `1h`, `1d`) * format: event record encoding format, possible options: * `json`: Both Json-line (newline separated records) and Json-array (`[{event:1}, {event:2}]`) formats are supported. * `snowplow:tsv|snowplow:json` - Snowplow-native format, see [Snowplow integration](broken://pages/tUianAitTMSZREn0VYsh) for details on how to set it up ### File Config file definition example: ```yaml type: file path: /home/user/ranklens/events/ offset: earliest|latest|ts=|last= format: sort: # optional, default name ``` The *path* parameter is a node-local file or directory with the input dataset. The `file` data source supports: * compression, auto-detected based on file extension: ZStandard and GZip are supported * directories with multiple input files. Metarank sorts files by the specified `sort` method to read them in a proper sequence. ### Apache Kafka [Apache Kafka](https://kafka.apache.org/) is an open source distributed event streaming platform. If you already use Kafka in your project, Metarank can connect to an existing Kafka topic to read incoming and stored events both for import (with an offset set to some time in the past) and inference (when offset is set to `latest`) stages. Kafka connector is configured in the following way: ```yaml type: kafka brokers: [broker1, broker2] topic: events groupId: metarank offset: earliest|latest|ts=|last= format: options: # optional connector raw parameters, map ``` Extra connector options can be taken from [ConsumerConfig](https://kafka.apache.org/32/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html) in the following way: ```yaml type: kafka options: client.id: 'helloworld' enable.auto.commit: 'false' ``` ### Apache Pulsar [Apache Pulsar](https://pulsar.apache.org/) is an open source distributed messaging and streaming platform. If you already use Pulsar in your project, Metarank can connect to an existing Pulsar topic to read incoming and stored events both for import (with an offset set to some time in the past) and inference (when offset is set to `latest`) stages. Metarank supports Pulsar *2.8+*, but using *2.9+* is recommended. Pulsar connector is configured in the following way: ```yaml type: pulsar serviceUrl: adminUrl: topic: events subscriptionName: metarank subscriptionType: exclusive # options are exclusive, shared, failover offset: earliest|latest|ts=|last= format: options: # optional connector parameters, map ``` Extra connector options can be taken from [Configuring Pulsar consumer](https://pulsar.apache.org/docs/client-libraries-java/#configure-consumer) in the following way: ```yaml type: pulsar options: receiverQueueSize: 10 acknowledgementsGroupTimeMicros: 50000 ``` ### AWS Kinesis Streams [AWS Kinesis Streams](https://aws.amazon.com/kinesis/) is a fully-managed event streaming platform. Metarank uses a connector for [Apache Flink](https://flink.apache.org) which is well-maintained and feature complete. To configure the connector, use this reference YAML block: ```yaml type: kinesis region: us-east-1 topic: events offset: earliest|latest|ts=|last= format: ``` Important things to note when using AWS Kinesis connector for bootstrap: * AWS Kinesis has a hard limit on [max retention time of 7 days](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html). If you want to store more data there, use AWS Firehose to offload these events to S3 and pick them with the `File` connector. * AWS limits max throughput per shard to 2Mb/s, so it may take some time to pull large dataset from kinesis. You may need to consider using an EFO consumer for a dedicated throughput. #### AWS Authentication Kinesis source uses a default auth chain from the AWS SDK, so all the possible ways of AWS SDK authentication methods are supported, but in short: * Use IAM roles when possible * add AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY env vars to manually supply the keys. ### REST API It's possible to ingest real-time feedback events directly using the REST API of Metarank: * `POST /feedback` - push feedback events to Metarank The `/feedback` endpoint is always enabled and there is no need to configure it explicitly. You can read more about Metarank REST API in the [API Documentation](/reference/api). You can bundle multiple events in a single batch using [batch payloads](/reference/api#feedback), so REST API can be used for batch dataset import instead of a separate `import` step: ```bash $ java -jar metarank serve --config conf.yaml $ curl -d @events.json http://localhost:8080/feedback ``` # Persistence Metarank supports two possible persistence modes for storing features: * [Memory](#memory-persistence): ephemeral; all state is in RAM. * [Redis](#redis-persistence): state persisted in remote Redis. Persistence mode is configured by the optional `state` section in the [configuration file](/reference/overview). By default, if the section is not defined, Metarank uses [memory persistence](#memory-persistence). > See also [training click-through persistence configuration](/reference/overview#training). ## Memory persistence Memory persistence is no persistence at all: the complete Metarank state is stored only in RAM, is ephemeral, and will be entirely lost on each service restart. Nevertheless, memory persistence can be useful: * While testing Metarank locally in a [standalone mode](/reference/deployment-overview/standalone), as it has no external service dependencies. * As a staging env to validate configuration changes before going to production. To configure memory persistence, use the `type: memory` option: ```yaml state: type: memory ``` ## Redis persistence Metarank can use [Redis 6+](https://redis.io) as a persistence method. To enable it, use the following [config file](/reference/overview) snippet: ```yaml state: type: redis host: localhost port: 6379 format: binary # optional, default=binary, possible values: json, binary cache: # optional maxSize: 1024 # size of in-memory client-side cache for hot keys, optional, default=1024 ttl: 1h # how long should key-values should be cached, optional, default=1h clientTracking: true # should we subscribe for CLIENT TRACKING invalidation events pipeline: # optional maxSize: 128 # batch write buffer size, optional, default=128 flushPeriod: 1s # buffer flush interval, optional, default=1s enabled: true # toggle pipelining, optional, default=true auth: # optional user: # optional when Redis ACL is disabled password: # required if Redis server is run with requirepass argument tls: # optional, defaults to disabled enabled: true # optional, defaults to false ca: # optional path to the CA used to generate the cert, defaults to the default keychain verify: full # optional, default=full, possible values: full, ca, off # full - verify both certificate and hostname # ca - verify only certificate # off - skip verification timeout: # optional, defaults to 1s for all sub-timeouts connect: 1s # optional, defaults to 1s socket: 1s # optional, defaults to 1s command: 1s # optional, defaults to 1s db: # optional, defaults to [0,1,2,3]: which redis dbs to use for persistence state: 0 # can be used to co-locate multiple metarank instances values: 1 # on a single redis server rankings: 2 models: 3 ``` Redis persistence is sensitive to network latencies (as it needs to perform a couple of round-trips on each event), hence Metarank leverages a couple of Redis performance optimization strategies: * [Pipelining](https://redis.io/docs/manual/pipelining/): all write operations are batched together and sent all at once * [Client-side caching](https://redis.io/docs/manual/client-side-caching/): read cache for hot keys with server-assisted invalidation. A note on optional cache & pipelining related settings: * Metarank has a separate cache per underlying feature type (like scalar/counter/map/etc, 10 total), so `cache.maxSize` is set per cache type, so keep in mind an implicit multiplication: default value `1024` in reality means `10240`. * `cache.ttl` defines expiration interval after last read, so hot features may be cached almost indefinitely. The problem of stale cache values is solved with [server-assisted invalidation](https://redis.io/docs/manual/client-side-caching/): Redis server sends a notification to Metarank when key value was changed by someone else. * `pipeline.maxSize` going above `128` is usually giving no benefit on low latencies (e.g. when Redis server is located in the same datacenter/AZ) * `pipeline.flushPeriod` controls the level of "eventualness" in the overall eventual consistency. With values larger than `10` seconds, a second Metarank instance may not see write buffered in a first instance. ## Disk persistence Metarank has also an experimental option of using disk persistence instead of Redis. The main drawback of such an approach is that the deployment becomes stateful and you need to maintain a disk persistence. Metarank supports two disk backends for file-based persistence: * MapDB: uses a mmap-based storage for data, works well for smaller datasets. * RocksDB: uses an LSM-tree storage, suits for large datasets. The file persistence configured in the following way: ```yaml state: type: file path: /path/to/dir # required format: binary # optional, default=binary, possible values: json, binary backend: # optional, default mapdb type: rocksdb # required, values: rocksdb, mapdb ``` ### RocksDB options RocksDB can be configured by defining the following values in the config file: ```yaml state: type: file path: /path/to/dir # required backend: # optional, default mapdb type: rocksdb lruCacheSizeMb: 1024000000 # LRU cache size in bytes, optional, default 1Gb blockSize: 8192 # Block size in bytes, optional, default 8kb ``` A rule of thumb defining these parameters: * higher LRU cache size leads to better read throughput at the cost of extra memory usage. If not sure, set it to 50% of your RAM. * blockSize defines a size of page RocksDB reads from disk. In a perfect world it should match your actual disk block size: For cloud-attached disks like AWS EBS it should be 16kb, for local drives 1-2kb. ### MapDB options MapDB can be configured in the following way: ```yaml state: type: file path: /path/to/dir # required backend: # optional, default mapdb type: mapdb mmap: true # should MapDB use mmap or raw disk reads for data access? Optional, default true. maxNodeSize: 16 # what is the node size for internal db index. Optional, default 16. ``` ### TLS Support Metarank supports connecting to Redis using TLS for transport encryption, but there is no way to autodetect the type of connection. To connect to a TLS-enabled Redis server with self-signed certificate, you need to specify the CA used to sign the certificate (for self-signed certs it will be the server certificate itself): ```yaml enabled: true ca: /tls/key.crt ``` To connect to a TLS-enabled Redis server with a certificate generated with default CA (for example, AWS ElastiCache Redis), then you don't need to specify any custom CA: ```yaml enabled: true ``` In a case when you have cert trust issues connecting to a TLS-enabled redis, you can downgrade the verification level. Supported levels are: * `full` - verify both certificate and hostname * `ca` - verify only the certificate * `off` - skip verification, trust all An example: ```yaml enabled: true verify: off ``` ### Authentication `auth.user` and `auth.password` can control the credentials used to connect to Redis. As hardcoding the credentials into the config file is not usually considered secure, you can supply the credentials from environment variables: * `METARANK_REDIS_USER` - only needed when Redis ACL is enabled. * `METARANK_REDIS_PASSWORD` - the pre-shared password used to connect to the Redis instance. Metarank's [Helm chart](/reference/deployment-overview/kubernetes) has a placeholder for the env variables passed inside the container inside Kubernetes. Usage example: ```yaml env: - name: METARANK_REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD ``` ### State encoding formats Metarank Redis persistence supports `json` and `binary` encoding formats for data stored in Redis: * `json`: focused on readability and debugging simplicity. * `binary`: low-overhead binary encoding format, with better performance and smaller memory footprint. `binary` format on typical datasets (like [RankLens](https://github.com/metarank/ranklens)) is \~2x faster and takes \~4x less RAM. We recommend it for larger datasets, when memory usage and associated costs are an important factor. ### Redis support limitations * Metarank requires Redis 6+ due to a lack of client-side caching support in 5.x * you can disable client caching altogether (for example, for managed Redis-compatible engines, like GCP Memorystore Redis) with `cache.maxSize: 0`. * For GCP Memorystore Redis, you can also set `state.cache.clientTracking: false` to disable the `CLIENT TRACKING` cache eviction support: GCP Memstore has client-side caching disabled even on 7.x Redis cluster. * Redis Cluster is not yet supported; see ticket [568](https://github.com/metarank/metarank/issues/568) for the progress. # Deployment ## Deployment * [Standalone deployment](/reference/deployment-overview/standalone): run it as a simple JVM application, great for local testing runs and baremetal installs. * [Docker](/reference/deployment-overview/docker): run as a docker container * [Kubernetes](/reference/deployment-overview/kubernetes): production-ready deployment in Kubernetes ## Monitoring * [Prometheus metrics](/reference/deployment-overview/prometheus): export a prometheus-compatible set of JVM and application metrics. # Standalone As a Java application, Metarank can be run locally either as a JAR-file or Docker container, there is no need for Kubernetes and AWS to start playing with it. Check out the [installation guide](/reference/installation) for detailed setup instructions. ## Running modes Metarank has multiple running modes: * `import` - import historical clickthroughs to the store * `train` - run traing the machine learning model using the imported data * `serve` - start the ranking inference API * `standalone` - which is a shortcut for `import`, `train` and `serve` jobs run together. * `validate` - a set of sanity checks on your configuration file and event dataset. Metarank's standalone mode is made to simplify the initial onboarding on the system: * it's a shortcut to run [`import`, `train` and `serve`](/reference/cli) tasks all at once * with [memory persistence](/reference/overview/persistence#memory-persistence) it can process large clickthrough histories almost instantly. ## Why standalone? Standalone mode is useful for these cases: * testing Metarank without deployment. With [in-memory persistence](/reference/overview/persistence#memory-persistence) it has zero service dependencies and is the easiest way to try it out. * simple staging deployments on VM/on-prem hardware. With [redis persistence](/reference/overview/persistence#redis-persistence) it can handle typical cases with small/medium load. Standalone mode has the following limitations: * feedback ingestion and inference throughput are limited by a single node. Please use the [Kubernetes deployment](/reference/deployment-overview/kubernetes) for a better experience. * model training happens within the inference process, and is a memory hungry process, which may cause latency spikes and OOMs. To overcome this limitation, you can train the machine learning model externally and upload it to the same Redis instance. ## Running Metarank in standalone mode To run the JAR file, make sure to follow the [installation manual for your OS](/reference/installation) and run it: ```bash $ java -jar metarank.jar standalone --data /path/to/events.json --config /path/to/config.yml ``` Another option is to run Metarank standalone mode from a docker container: ```bash $ docker run -v /data/: metarank/metarank:latest standalone --data /data/events.json --config /data/config.yml ``` The follwing options are used for the docker container: * `-v /data:` to map a directory with input files and configuration into the container * `--data /data/events.json` to pass the name of [input events file](/reference/event-schema), from the mapped volume * `--config /data/config.yml` to pass the [configuration file](/reference/overview) During the startup process Metarank will: * import your dataset and compute all historical event statistics useful for machine learning model training * train the machine learning model you defined in the configuration file * start the inference API for real-time personaization. ![import and training process](/files/HIp3M3rAInwGaN0IgKly) For a more detailed walkthrough of running Metarank in playground, check out the [quickstart guide](/introduction/quickstart). # Docker Metarank official image is published in docker hub as [metarank/metarank](https://hub.docker.com/r/metarank/metarank/tags). We publish the `:latest` tag, although it's not always recommended to have any production deployments without pinning a specific version. Official docker images are multi-arch, and cross-built for the following platforms: * linux/amd64: a regular docker image * linux/arm64/v8: docker image which will work natively (so without emulation) on platforms like Mac M1/M2. ## Running the docker image All metarank sub-commands are wrapped into a single command-line API. To see the [CLI](/reference/cli), run the docker container: ```shell $ docker run metarank/metarank:0.7.9 --help + exec /opt/java/openjdk/bin/java -jar /app/metarank.jar --help __ __ _____ _____/ |______ ____________ ____ | | __ / \_/ __ \ __\__ \\_ __ \__ \ / \| |/ / | Y Y \ ___/| | / __ \| | \// __ \| | \ < |__|_| /\___ >__| (____ /__| (____ /___| /__|_ \ \/ \/ \/ \/ \/ \/ ver:0.7.9 Usage: metarank ``` ### Resources Metarank image exposes a `/data` volume to handle all the local IO. For example, you can pass the input training dataset from your local host using the docker's `-v` switch: ```shell docker run -v /home/user/input:/data metarank/metarank:latest train ``` #### Memory Metarank docker container uses 1Gb of JVM heap by default. In practice the actual RSS memory usage is a bit higher than the heap size due to JVM's extra overhead. This can be configured with the `JAVA_OPTS` environment variable: ```shell docker run -e JAVA_OPTS="-Xmx5g" metarank/metarank:latest train ``` ### Ports The image exposes the following ports: * 8080 for API access for the inference and ingestion APIs To map these ports to your host, use the `-p` flag: ```shell docker run -p 8080:8080 metarank/metarank:latest serve ``` # Kubernetes Metarank can be deployed in a distributed fashion inside a Kubernetes cluster. ![installation overview](/files/qyNQNjdCyQTvX6EN1ApI) ## Prerequisites For a distributed K8S deployment, metarank requires the following external services and tools to be already available: 1. Helm: used to install the Metarank chart. 2. Redis: as an almost-persistent data store for inference. Can be also installed either inside k8s with helm, or as a managed service like AWS ElastiCache Redis. 3. Distributed event bus for event ingestion: Kafka, Pulsar, Kinesis and internal RESTful API are supported. ## Data Import Metarank supports multiple ways of ingesting training data into the system: * event file can be HTTT POSTed to the `/feedback` endpoint using the [REST API](/reference/api). Metarank does not do any in-memory buffering, so if your dataset is below 1GiB in size, this may be the simplest way to ingest. * event can be imported from a Kafka/Pulsar/Kinesis topic or read from files **locally**. Note that distributed import is not yet supported. We suggest to start with a HTTP-based event import, and switch to offline local import if you have any issues with it. ## Tuning the Helm chart With Helm installed according to its official [installation guide](https://helm.sh/docs/intro/install/), you need to add a [Metarank Helm repo](https://github.com/metarank/helm-charts): ```shell $> helm repo add metarank https://metarank.github.io/helm-charts "metarank" has been added to your repositories $> helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "metarank" chart repository Update Complete. Happy Helming! $> helm pull metarank/metarank --untar $> cd metarank ``` In the chart directory there are `metarank.conf` and `values.yaml` files you'll need to update before the deployment: ```shell total 24 drwxr-xr-x 2 user user 4096 Oct 4 14:25 charts -rw-r--r-- 1 user user 124 Oct 4 17:23 Chart.yaml -rw-r--r-- 1 user user 376 Oct 4 17:23 metarank.conf -rw-r--r-- 1 user user 989 Oct 4 17:23 README.md drwxr-xr-x 3 user user 4096 Oct 4 17:23 templates -rw-r--r-- 1 user user 1889 Oct 4 17:23 values.yaml ``` The `metarank.conf` file is a regular metarank configuration file, so you can check [the configuration guide](/reference/overview) to set things up manually, or use an automatic [data-based config generator](/how-to/autofeature). The `metarank.conf` file requires you to define a Redis endpoint for state store. A good-looking config file is shown below: ```yaml api: host: 0.0.0.0 state: type: redis host: add-redis-hostname-here port: 6379 models: xgboost: type: lambdamart backend: type: xgboost iterations: 50 weights: click: 1 features: - popularity features: - name: popularity type: number scope: item source: metadata.popularity ``` The `values.yaml` is a generic helm deployment configuration file. You can tune it, but default one **usually** requires no extra changes. ### Resources The default helm chart sets no specific memory requests & limits, but it can be configured with `values.yaml`. The Metarank docker container accepts a `JAVA_OPTS` environment variable to control the JVM memory usage. It defaults to `JAVA_OPTS="-Xmx1g -verbose:gc"` which means: * Use 1Gb for JVM heap. The actual RSS memory usage should be a bit higher due to JVM extra overhead. * Enable verbose GC logging. You may notice the following lines in the log, they are normal: ``` [282.621s][info][gc] GC(26) Pause Young (Allocation Failure) 55M->36M(67M) 2.718ms ``` ## Installing the chart The chart itself is agnostic to the Metarank version, and has separate versioning. For the latest Metarank `0.7.9` release, use the following command to install the chart: ```shell helm install metarank . --set-file config=metarank.conf --set image.tag=0.7.9 NAME: metarank LAST DEPLOYED: Tue Oct 4 15:32:47 2022 NAMESPACE: default STATUS: deployed REVISION: 1 ``` After that, a single metarank pod will be running: ```shell $> kubectl get pods NAME READY STATUS RESTARTS AGE metarank-6c577f46f6-9c9mz 1/1 Running 0 136m redis-master-0 1/1 Running 0 171m ``` ## Next steps After successful deployment you may want to do the following: * Enable ingress in `values.yaml` so Metarank can be accessible from outside. * HTTP POST the training data to the [`/feedback` REST API](/reference/api#feedback). * Send your first reranking request according to the [quickstart guide](/introduction/quickstart) * Configure Kafka/Pulsar/Kinesis as [a real-time data source](/reference/overview/data-sources). # Prometheus metrics export Metarank exports a set of internal metrics you can use to monitor its health. See the [`/metrics`](/reference/api#prometheus-metrics) endpoint description for details on how to access them. ## Application metrics All application metrics have a common `metarank_` prefix: 1. `metarank_rank_requests`: counter, number of requests received by the `/rank` endpoint. This metric also counts requests by model name. ``` metarank_rank_requests_total{model="model_name",} 5.0 ``` 1. `metarank_feedback_events`: counter, number of feedback events received both from API and any other connector (like kafka/pulsar/kinesis). ``` metarank_feedback_events_total 58441.0 ``` 1. `metarank_rank_latency_seconds`, histogram, latency distribution for `/rank` requests, scoped by a model. Percentiles tracked: 50%, 80%, 90%, 95%, 98%, 99%. ``` metarank_rank_latency_seconds{model="xgboost",quantile="0.5",} 0.011451508 metarank_rank_latency_seconds{model="xgboost",quantile="0.8",} 0.014340056 metarank_rank_latency_seconds{model="xgboost",quantile="0.9",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.95",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.98",} 0.119447575 metarank_rank_latency_seconds{model="xgboost",quantile="0.99",} 0.119447575 metarank_rank_latency_seconds_count{model="xgboost",} 5.0 metarank_rank_latency_seconds_sum{model="xgboost",} 0.16446094099999997 ``` ## JVM metrics Metarank also exports a set of [default JVM metrics](https://github.com/prometheus/client_java/blob/main/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/DefaultExports.java), related to buffers, classloaders, GC, allocation and threadpools. All the JVM metrics have common `jvm_` prefix. ## Grafana dashboard coming soon. # Custom logging In a cases when you need to override default Metarank logging configuration, you may need to build a custom docker image, based on an original upstream one from Metarank. Metarank uses the following entrypoint for the docker container: ```shell #!/bin/bash set -euxo pipefail OPTS=${JAVA_OPTS:-"-Xmx1g -verbose:gc"} exec /usr/bin/java $OPTS -cp "/app/*" ai.metarank.main.Main "$@" ``` So you should note the following configuration toggles here: * env variable JAVA\_OPTS can be used to pass custom JVM flags, like path to a custom logger configuration. * by default Metarank loads all the JAR files found in the `/app/` directory. ## Example: logstash-logback-encoder To enable structured logging via [logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder), you can build a custom Docker image with the following Dockerfile: ```shell FROM metarank/metarank:0.7.9-amd64 # add logback configuration file to the image ADD logback.xml /app/ # add the logstash-logback-encoder with all its runtime dependencies to the classpath ADD https://repo1.maven.org/maven2/net/logstash/logback/logstash-logback-encoder/7.4/logstash-logback-encoder-7.4.jar /app/ ADD https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar /app/ ADD https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.16.0/jackson-databind-2.16.0.jar /app/ # override default logback configuration ENV JAVA_OPTS="-Xmx1g -Dlogback.configurationFile=/app/logback.xml -Dlogback.debug=true" ``` Such a custom image will successfully load the custom conviguration with non-default appender: ``` + OPTS='-Xmx1g -Dlogback.configurationFile=/app/logback.xml -Dlogback.debug=true' + exec /usr/bin/java -Xmx1g -Dlogback.configurationFile=/app/logback.xml -Dlogback.debug=true -cp '/app/*' ai.metarank.main.Main --help 19:14:30,083 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 0.7.3 19:14:30,113 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [/app/logback.xml] at [file:/app/logback.xml] 19:14:30,175 |-WARN in ch.qos.logback.core.joran.action.IncludeAction - Could not find resource corresponding to [logback-properties.xml] 19:14:30,225 |-INFO in ch.qos.logback.core.model.processor.StatusListenerModelHandler - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener] 19:14:30,227 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [CONSOLE_JSON] 19:14:30,227 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 19:14:30,242 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventJsonProviders] for [providers] property 19:14:30,250 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventFormattedTimestampJsonProvider] for [timestamp] property 19:14:30,253 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider] for [pattern] property 19:14:30,258 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [net.logstash.logback.composite.loggingevent.StackTraceJsonProvider] for [stackTrace] property 19:14:30,383 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO 19:14:30,383 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [CONSOLE_JSON] to Logger[ROOT] 19:14:30,383 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@24fcf36f - End of configuration. 19:14:30,384 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4d02f94e - Registering current configuration as safe fallback point ``` # Warmup When running Metarank in production, you may hit the cold-start problem: * after a restart (due to redeployment/autoscaling) Metarank pod starts cold with empty caches, * JVM also has not yet compiled most of the code, * due to this, reranking latency of first N requests is too high. To handle this Metarank (starting from v0.7.6) supports explicit API warmup: * while training, sample N random but real reranking requests and persist them in store * on API startup, replay this random traffic sample * when replay done, bring up the API - so k8s readiness probe will be successful. ## Configuring warmup API warmup is only supported for LambdaMART models and is configured per-model: ```yaml models: my-super-model: type: lambdamart warmup: sampledRequests: 100 # how many requests sample during training duration: 5s # how long to replay the traffic during warmup # ... ``` Warmup is disabled by default, and you need to retrain your model if you trained it in Metarank prior to 0.7.6. After the warmup is enabled, you will see the following log output when starting the API: ``` 15:46:41.221 INFO a.metarank.main.command.Standalone$ - model 'default' training finished 15:46:41.484 INFO ai.metarank.main.command.Serve$ - warmup of model default: config=WarmupConfig(100,10 seconds) requests=100 15:46:42.486 INFO ai.metarank.flow.PrintProgress$ - processed 900 warmup requests, perf=898rps GC=0.0% heap=6.63%/8.0G 15:46:43.505 INFO ai.metarank.flow.PrintProgress$ - processed 2100 warmup requests, perf=1178rps GC=0.0% heap=6.63%/8.0G 15:46:44.548 INFO ai.metarank.flow.PrintProgress$ - processed 3400 warmup requests, perf=1246rps GC=0.29% heap=6.63%/8.0G 15:46:45.626 INFO ai.metarank.flow.PrintProgress$ - processed 4700 warmup requests, perf=1206rps GC=0.0% heap=6.63%/8.0G 15:46:46.683 INFO ai.metarank.flow.PrintProgress$ - processed 6000 warmup requests, perf=1230rps GC=0.0% heap=6.63%/8.0G 15:46:47.705 INFO ai.metarank.flow.PrintProgress$ - processed 7300 warmup requests, perf=1273rps GC=0.0% heap=6.63%/8.0G 15:46:48.729 INFO ai.metarank.flow.PrintProgress$ - processed 8600 warmup requests, perf=1268rps GC=0.0% heap=6.63%/8.0G 15:46:49.766 INFO ai.metarank.flow.PrintProgress$ - processed 9900 warmup requests, perf=1255rps GC=0.0% heap=6.63%/8.0G 15:46:50.789 INFO ai.metarank.flow.PrintProgress$ - processed 11100 warmup requests, perf=1172rps GC=0.0% heap=6.63%/8.0G 15:46:51.686 INFO ai.metarank.main.command.Serve$ - __ __ _____ _____/ |______ ____________ ____ | | __ / \_/ __ \ __\__ \\_ __ \__ \ / \| |/ / | Y Y \ ___/| | / __ \| | \// __ \| | \ < |__|_| /\___ >__| (____ /__| (____ /___| /__|_ \ \/ \/ \/ \/ \/ \/ 15:46:51.686 INFO ai.metarank.main.command.Serve$ - Starting API... 15:46:51.733 INFO o.h.ember.server.EmberServerBuilder - Ember-Server service bound to address: [::]:8080 ``` # Integrations Metarank out of the box integrates with the following tools: * [Snowplow](broken://pages/tUianAitTMSZREn0VYsh) # Snowplow ![snowplow logo](/files/iXwn8fa4qO7ggCjmjhmP) Metarank can be integrated into existing [Snowplow Analytics](https://snowplowanalytics.com/) setup. We provide a set of [Iglu Schemes](#schema-registry) that you will use to track metadata and interaction events that can later on be read from Snowplow's enriched event stream by Metarank. * [Snowplow Trackers](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/) are used to track Metarank-specific events. * Metarank will use Snowplow's enriched event stream as a source of events. ## Typical Snowplow architecture Typical Snowplow Analytics setup consists of the following parts: * Using [Snowplow Trackers](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/), your application emits a clickstream telemetry to the [Stream Collector](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/stream-collector/) * Stream Collector writes all incoming events into the raw stream * [Enrichment](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/enrichment-components/) validates these events according to the predefined schemas from the Schema Registry * Validated and enriched events are written to the enriched stream * Enriched events are delivered to the Analytics DB Metarank exposes a set of Snowplow-compatible event schemas, and can read events directly from the enriched stream, as shown on the diagram below: ![snowplow typical setup](/files/IjPmyuX6zqlyZLYq5FEy) ### Schema registry All incoming raw events have a strict JSON schema, that consists of the following parts: * predefined fields according to the [Snowplow Tracker Protocol](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/) * unstructured payload with user-defined schema * multiple context payloads with user-defined schemas These user-defined schemas are pulled from the [Iglu Registry](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/), and these schemas are standard [JSON Schema](https://json-schema.org/specification.html) definitions, describing the payload structure. There are four different Metarank event types with the corresponding schemas: 1. `ai.metarank/item/1-0-0`: [item metadata event](https://github.com/metarank/metarank-snowplow/blob/master/schemas/ai.metarank/item/1-0-0) 2. `ai.metarank/user/1-0-0`: [user metadata event](https://github.com/metarank/metarank-snowplow/blob/master/schemas/ai.metarank/user/1-0-0) 3. `ai.metarank/ranking/1-0-0`: [ranking event](https://github.com/metarank/metarank-snowplow/blob/master/schemas/ai.metarank/item/1-0-0) 4. `ai.metarank/interaction/1-0-0`: [interaction event](https://github.com/metarank/metarank-snowplow/blob/master/schemas/ai.metarank/interaction/1-0-0) These schemas are describing native [Metarank event types](/reference/event-schema) without any modifications. Check out [github.com/metarank/metarank-snowplow](https://github.com/metarank/metarank-snowplow) for more details about Metarank schemas. ### Stream transport types Snowplow supports [multiple streaming platforms](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/stream-collector/) for event delivery: * [AWS Kinesis](https://aws.amazon.com/kinesis/): **supported by Metarank** * [Kafka](https://kafka.apache.org/): **supported by Metarank** * [GCP Pubsub](https://cloud.google.com/pubsub): *support is* [*planned in the future*](https://github.com/metarank/metarank/issues/477) * [NSQ](https://nsq.io/): *not supported* * [Amazon SQS](https://aws.amazon.com/sqs/): *not supported* * \[stdout]: *not supported* ## Setting up event tracking Metarank needs to receive 4 types of [events](/reference/event-schema), describing items, users and how users interact with items: * [item metadata](/reference/event-schema#item-metadata-event): like titles, inventory, tags * [user metadata](/reference/event-schema#user-metadata-event): country, age, location * [ranking](/reference/event-schema#ranking-event): what items and in what order were displayed to a visitor * [interaction](/reference/event-schema#interaction-event): how visitor interacted with the ranking These events can be generated both on the frontend side, and on the backend side, depending on your setup and data availability on the front and back ends. ![snowplow trackers](/files/Vmj7mSDrnBAFGAwDR8SL) #### Frontend tracking Using [Snowplow JS Tracker SDK](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/), you can track [self-describing events](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/), which are JSONs with attached schema references. An example of tracking a ranking event: ```js import { trackSelfDescribingEvent } from '@snowplow/browser-tracker'; trackSelfDescribingEvent({ event: { schema: 'iglu:ai.metarank/ranking/jsonschema/1-0-0', data: { event: 'ranking', id: '81f46c34-a4bb-469c-8708-f8127cd67d27', timestamp: '1599391467000', user: 'user1', session: 'session1', fields: [ { name: 'query', value: 'cat' }, { name: 'source', value: 'search' } ], items: [ { id: "item1" }, { id: "item2" } ] } } }); ``` Check out the [JSON-Schema definitions for events](https://github.com/metarank/metarank-snowplow/) and [event format](/reference/event-schema) articles for details on fields, event types and their meaning. Metarank schemas are language-agnostic and you can instrument your app using any supported [Snowplow Tracker SDK](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/) for your favourite language/framework of choice. #### Backend tracking It Often happens that the frontend doesn't have all the required information to generate events. A good example is item metadata event (usually tags, titles and price are altered in some back-office system and are not directly exposed to the frontend). In this case you can generate such events on the backend side. For a sample Java backend application, you can track an item update event with the following code, using the [Snowplow Java Tracker SDK](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/java-tracker/): ```java import com.snowplowanalytics.snowplow.tracker.*; import com.snowplowanalytics.snowplow.tracker.emitter.*; import com.snowplowanalytics.snowplow.tracker.events.Unstructured; import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson; import java.util.HashMap; import java.util.List; import java.util.Map; public class JavaTrackerExample { public static void main(String[] args) { BatchEmitter emitter = BatchEmitter.builder() .url("http://collectorEndpoint") .build(); Tracker tracker = new Tracker .TrackerBuilder(emitter, "trackerNamespace", "appId") .build(); Map payload = new HashMap<>(); payload.put("event", "item"); payload.put("id", "81f46c34-a4bb-469c-8708-f8127cd67d27"); payload.put("timestamp", String.valueOf(System.currentTimeMillis())); payload.put("item", "item1"); Map fields = new HashMap<>(); fields.put("title", "your cat"); fields.put("color", List.of("white", "black")); payload.put("fields", fields); Unstructured unstructured = Unstructured.builder() .eventData(new SelfDescribingJson("iglu:ai.metarank/item/1-0-0", payload)) .build(); tracker.track(unstructured); } } ``` ## Installing ai.metarank schemas Metarank schemas are available on a public Iglu server on `https://iglu.metarank.ai`. To use it, add the following snippet to the `resolver.json` snowplow-enrich config file: ```json { "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1", "data": { "cacheSize": 500, "repositories": [ { "name": "Metarank", "priority": 0, "vendorPrefixes": [ "ai.metarank" ], "connection": { "http": { "uri": "https://iglu.metarank.ai" } } } ] } } ``` Both `http` and `https` schemas are supported, but `https` is recommended. ## Connecting Metarank with Snowplow Snowplow enrich emits processed records in [TSV format](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/understanding-the-enriched-tsv-format/) into the downstream Kinesis/Pubsub/etc. topic. This topic is usually later monitored by "Loaders", like a Snowflake loader, or [S3 Loader](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/loaders-storage-targets/s3-loader). An example loader integration diagram is shown below: ![snowplow loaders](/files/hUJ5zVpddEK56tLegMQ6) Snowplow is flexible enough to use different data loading destinations (Redshift, Postgres, Snowflake, S3, etc.), but to access both live and historical enriched event data, Metarank needs an access to: * Enriched event stream * Historical enriched stream dumps done with S3 Loader **At the moment Metarank supports loading historical events only from S3 Loader.** ### Realtime events from AWS Kinesis Snowplow enrich is usually configured with three destination streams output/pii/bad, with the same HOCON definition: ```properties "output": { # Enriched events output "good": { "type": "Kinesis" # Name of the Kinesis stream to write to "streamName": "enriched" # Optional. Maximum amount of time an enriched event may spend being buffered before it gets sent "maxBufferedTime": 100 millis } } ``` To make metarank connect to this stream, configure the [kinesis source](/reference/overview/data-sources#aws-kinesis-streams) in the following way: ```yaml inference: port: 8080 host: "0.0.0.0" source: type: kinesis region: us-east-1 topic: enriched offset: latest format: snowplow:tsv ``` All the supported Metarank sources have an optional `format` field, which defines the underlying format of the payload in this stream. Valid options are: * `json`: default value, Metarank native format * `snowplow`, `snowplow:tsv`: Snowplow default TSV stream format * `snowplow:json`: Snowplow optional JSON stream format With the `format: snowplow:tsv`, Metarank will read TSV events and transform them into native format automatically. ### Historical events from AWS S3 [Snowplow S3 Loader](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/loaders-storage-targets/s3-loader) offloads realtime enriched events to gzip/lzo compressed files on S3. Given the following [sample S3 Loader config snippet](https://github.com/snowplow/snowplow-s3-loader/blob/master/config/config.hocon.sample): ```properties { # Optional, but recommended "region": "us-east-1", # Options are: RAW, ENRICHED_EVENTS, JSON "purpose": "ENRICHED_EVENTS", # Input Stream config "input": { # Kinesis Client Lib app name (corresponds to DynamoDB table name) "appName": "acme-s3-loader", # Kinesis stream name "streamName": "enriched", # Options are: LATEST, TRIM_HORIZON, AT_TIMESTAMP "position": "LATEST", # Max batch size to pull from Kinesis "maxRecords": 10 }, "output": { "s3": { # Full path to output data "path": "s3://acme-snowplow-output/enriched/", # Partitioning format; Optional # Valid substitutions are {vendor}, {schema}, {format}, {model} for self-describing jsons # and {yy}, {mm}, {dd}, {hh} for year, month, day, hour partitionFormat: "{vendor}.{schema}/model={model}/date={yy}-{mm}-{dd}" # Output format; Options: GZIP, LZO "compression": "GZIP" } } } ``` You can instrument Metarank to load these GZIP-compressed event dumps for the bootstrapping process with the [file source](/reference/overview/data-sources) in the following way: ```yaml bootstrap: source: type: file path: "s3://acme-snowplow-output/enriched/" offset: earliest format: snowplow:tsv ``` ## Validating the setup With Metarank configured to pick live events from the enriched stream, and historical events from the offloaded files in S3, it should be straightforward to do the usual routine of [setting up](/reference/overview) and [running it](/reference/deployment-overview) Metarank. # Automated ML model retraining A problem: how to periodically re-train an ML model on a new data? ## Click-through collection While receiving and processing incoming events, Metarank collects click-through records: * On each [ranking event](/reference/event-schema#ranking-event), it logs all ML feature values used to produce it. As dynamic features constantly change in time, it allows to easily know, what was the value of any feature back in time. ![join](/files/JgmU53KZTzOJWXXLiplr) * Within a default 30-minute window (see a [`core.clickthrough.maxSessionLength`](/reference/overview#core) option for details) all interactions within this ranking event are collected. So you can know which items a visitor has seen, and later interacted with. * After a click-trough join window is finalized, then ranking, interactions, and feature values are persisted in the store. ![click-through](/files/BM9KzzQkvdcGtPXGNhVL) These click-through records can be unfolded into an [implicit judgement list](https://softwaredoug.com/blog/2021/02/21/what-is-a-judgment-list.html). A judgement list later can be translated into a ML backend specific training dataset for a LambdaMART model training. Metarank collects click-through records automatically, you don't need to tune anything to enable this behavior. ## Manual retraining Given that you have a production Metarank instance running somewhere in the cloud, you can re-train a ML model based on a history of already collected click-through records locally: ```shell $> java -jar metarank.jar train --config /path/to/config.yml 14:03:26.630 INFO ai.metarank.main.Main$ - Metarank is starting. 14:03:27.279 INFO ai.metarank.config.Config$ - api conf block is not defined: using default ApiConfig(Hostname(localhost),Port(8080)) 14:03:27.292 INFO ai.metarank.config.Config$ - Loaded config file, state=redis://Hostname(localhost):Port(6379), features=[position,popularity,vote_avg,vote_cnt,budget,release_date,runtime,title_length,genre,ctr,liked_genre,liked_actors,liked_tags,liked_director,visitor_click_count,global_item_click_count,day_item_click_count], models=[xgboost] 14:03:27.320 INFO ai.metarank.main.Main$ - usage analytics disabled: METARANK_TRACKING=None isRelease=false 14:03:27.359 INFO ai.metarank.FeatureMapping - optimized schema: removed 5 unused features 14:03:32.267 INFO ai.metarank.main.command.Train$ - loaded 8773 clickthroughs, 8773 with clicks 14:03:34.108 INFO ai.metarank.main.command.Train$ - generated training dataset: 8773 groups, 27 dims 14:03:34.115 INFO ai.metarank.main.command.Train$ - training model for train=7054 test=1719 14:03:34.773 INFO i.g.m.l.ranking.pairwise.LambdaMART - [0] NDCG@train = 0.5594632053375398 NDCG@test = 0.5348729264515503 14:03:39.768 INFO i.g.m.l.ranking.pairwise.LambdaMART - [49] NDCG@train = 0.643993571767862 NDCG@test = 0.5979538819484762 14:03:39.905 INFO a.m.fstore.redis.RedisPersistence - flushing redis pipeline 14:03:40.920 INFO a.m.fstore.redis.RedisPersistence - redis pipeline flushed 14:03:40.921 INFO ai.metarank.main.command.Train$ - model uploaded to store, 693384 bytes 14:03:41.327 INFO ai.metarank.main.command.Train$ - budget: weight=884.0 zero=24154 nz=186398 dist=[0.00,4500000.00,10000000.00,16000000.00,25000000.00,34000000.00,48000000.00,70000000.00,110000000.00] 14:03:41.328 INFO ai.metarank.main.command.Train$ - ctr: weight=[1042.0,0.0] zero=1450 nz=419654 dist=[0.03,0.07,0.13,0.19,0.24,0.30,0.43,0.85,1.61] 14:03:41.329 INFO ai.metarank.main.command.Train$ - genre: weight=[105.0,79.0,77.0,62.0,88.0,63.0,96.0,56.0,93.0,44.0,97.0,58.0,47.0,49.0,46.0] zero=2582113 nz=576167 dist=[0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,1.00] 14:03:41.330 INFO ai.metarank.main.command.Train$ - liked_actors: weight=469.0 zero=198057 nz=12495 dist=[0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00] 14:03:41.330 INFO ai.metarank.main.command.Train$ - liked_director: weight=403.0 zero=206886 nz=3666 dist=[0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00] 14:03:41.330 INFO ai.metarank.main.command.Train$ - liked_genre: weight=1047.0 zero=52726 nz=157826 dist=[0.00,0.00,0.10,0.17,0.22,0.29,0.33,0.46,0.57] 14:03:41.331 INFO ai.metarank.main.command.Train$ - liked_tags: weight=749.0 zero=149001 nz=61551 dist=[0.00,0.00,0.00,0.00,0.00,0.00,0.01,0.03,0.08] 14:03:41.331 INFO ai.metarank.main.command.Train$ - position: weight=868.0 zero=8773 nz=201779 dist=[2.00,4.00,7.00,9.00,12.00,14.00,16.00,19.00,21.00] 14:03:41.331 INFO ai.metarank.main.command.Train$ - release_date: weight=1060.0 zero=533 nz=210019 dist=[555897600.00,728697600.00,822960000.00,900028800.00,974851200.00,1055289600.00,1144800000.00,1244160000.00,1375315200.00] 14:03:41.332 INFO ai.metarank.main.command.Train$ - runtime: weight=735.0 zero=533 nz=210019 dist=[90.00,95.00,100.00,104.00,108.00,114.00,119.00,127.00,137.00] 14:03:41.332 INFO ai.metarank.main.command.Train$ - title_length: weight=328.0 zero=533 nz=210019 dist=[1.00,1.00,2.00,2.00,2.00,3.00,3.00,4.00,5.00] 14:03:41.332 INFO ai.metarank.main.command.Train$ - vote_avg: weight=848.0 zero=533 nz=210019 dist=[5.90,6.20,6.40,6.60,6.80,7.00,7.30,7.50,7.80] 14:03:41.333 INFO a.m.fstore.redis.client.RedisClient$ - closing redis connection 14:03:41.351 INFO a.m.fstore.redis.client.RedisClient$ - closing redis connection 14:03:41.356 INFO a.m.fstore.redis.client.RedisClient$ - closing redis connection 14:03:41.360 INFO a.m.fstore.redis.client.RedisClient$ - closing redis connection 14:03:41.364 INFO ai.metarank.main.Main$ - My job is done, exiting. ``` While training, Metarank will do the following steps: * Pull all stored click-through records from the store. Your local Metarank config file should be the same as the one use by the serving instance: store config and feature definitions should match. * Convert them to a XGBoost/LightGBM compatible judgement lists: ![ltr judgement list](/files/4aDQhQdZAE1sDz0sPQCj) * Do the ML model training. * Upload the model into the store and notify all API serving instances to reload the model. ## Automated retraining Metarank can be [deployed inside a Kubernetes cluster](/reference/deployment-overview/kubernetes) using an official [Helm manifest](https://github.com/metarank/metarank/tree/master/deploy/kubernetes). This Helm manifest's configuration file can create a [Kubernetes CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/), which will do the same retraining action as described in the previous section, but automatically with a user-defined schedule: ```yaml train: enabled: true # retrain all models once a day, at 6AM schedule: '"0 6 * * *"' # a separate set of resource definitions for model retraining job resources: {} ``` You can define a custom cron-compatible schedule using a `train.schedule` option. ML model training may require a lot of resources, so it's recommended to properly define resource configuration, so you won't hit OOM error. A retraining CronJob is created by default by the Metarank Helm chart, so if you're using it for production deployment, you're ready to go without any configuration changes. # Automatic feature engineering A typical problem: to write a Metarank [config file](/reference/overview) with event to feature mapping, you need to read the docs on [feature extraction](/reference/overview/feature-extractors) and well-understand your click-through input dataset: * Which fields do items have? Which values does each field have? * Do these values look like categories? * How many unique values are there per field? Nobody likes reading docs and writing YAML, so Metarank has an [AutoML level-4 style](https://medium.com/@tunguz/six-levels-of-auto-ml-a277aa1f0f38) generator of typical feature extractors based on the [historical click-through dataset](/reference/event-schema) you already have. ## Running the autofeature generator Use the `autofeature` sub-command from the main binary: ```bash __ __ _____ _____/ |______ ____________ ____ | | __ / \_/ __ \ __\__ \\_ __ \__ \ / \| |/ / | Y Y \ ___/| | / __ \| | \// __ \| | \ < |__|_| /\___ >__| (____ /__| (____ /___| /__|_ \ \/ \/ \/ \/ \/ \/ ver:None Usage: metarank Subcommand: autofeature - generate reference config based on existing data -d, --data path to a directory with input files -f, --format input file format: json, snowplow, snowplow:tsv, snowplow:json (optional, default=json) -o, --offset offset: earliest, latest, ts=1663161962, last=1h (optional, default=earliest) --out path to an output config file -r, --ruleset set of rules to generate config: stable, all (optional, default=stable, values: [stable, all]) -c, --cat-threshold min threshold of category frequency, when its considered a catergory (optional, default=0.003) -h, --help Show help message For all other tricks, consult the docs on https://docs.metarank.ai ``` An example minimal command to generate the config file for your dataset: ```bash java -jar metarank.jar autofeature --data /path/to/events.json --out /path/to/config.yaml ``` For a [RankLens](https://github.com/metarank/ranklens) dataset, for example, it will emit the following: ``` 15:32:11.284 INFO a.metarank.main.command.AutoFeature$ - Generating config file 15:32:11.524 INFO ai.metarank.source.FileEventSource - path=/home/shutty/code/metarank/src/test/resources/ranklens/events/events.jsonl.gz is a file 15:32:11.537 INFO ai.metarank.source.FileEventSource - file /home/shutty/code/metarank/src/test/resources/ranklens/events/events.jsonl.gz selected=true (timeMatch=true formatMatch=true) 15:32:11.538 INFO ai.metarank.source.FileEventSource - reading file /home/shutty/code/metarank/src/test/resources/ranklens/events/events.jsonl.gz (with gzip decompressor) 15:32:15.686 INFO a.metarank.main.command.AutoFeature$ - Event model statistics collected 15:32:15.691 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'writer' 15:32:15.692 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'tags' 15:32:15.692 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'director' 15:32:15.693 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'title' 15:32:15.693 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'genres' 15:32:15.693 INFO a.m.m.c.a.r.InteractionFeatureRule$ - generated interacted_with feature for interaction 'click' over field 'actors' 15:32:15.700 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field popularity in the range 0.6..967.351 15:32:15.704 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field vote_avg in the range 3.2..8.7 15:32:15.705 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field release_date in the range 3.18816E8..1.56168E9 15:32:15.707 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field budget in the range 0.0..3.8E8 15:32:15.708 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field vote_cnt in the range 15.0..30232.0 15:32:15.709 INFO a.m.m.c.a.r.NumericalFeatureRule$ - generated `number` feature for item field runtime in the range 3.0..242.0 15:32:15.719 INFO a.m.m.c.a.rules.StringFeatureRule - field writer is not looking like a categorial value, skipping 15:32:15.726 INFO a.m.m.c.a.rules.StringFeatureRule - field tags is not looking like a categorial value, skipping 15:32:15.728 INFO a.m.m.c.a.rules.StringFeatureRule - field director is not looking like a categorial value, skipping 15:32:15.730 INFO a.m.m.c.a.rules.StringFeatureRule - field title is not looking like a categorial value, skipping 15:32:15.731 INFO a.m.m.c.a.rules.StringFeatureRule - item field genres has 19 distinct values, generated 'string' feature with index encoding for top 11 items 15:32:15.734 INFO a.m.m.c.a.rules.StringFeatureRule - field actors is not looking like a categorial value, skipping 15:32:15.735 INFO a.m.m.c.a.rules.RelevancyRule$ - skipped generating relevancy feature: non_zero=0 min=Some(0.0) max=Some(0.0) 15:32:15.851 INFO ai.metarank.main.Main$ - My job is done, exiting. Process finished with exit code 0 ``` ## Supported heuristics Metarank has multiple sets of heuristics to generate feature configuration, toggled by the `--ruleset` CLI option: * stable: a default one, ruleset with less agressive heuristics, proven to be safe in production use. * all: generates all features it can, even the problematic ones (like CTR, which may introduce biases). The following `stable` heuristics are supported: * **Numeric**: all numerical item fields are encoded as a [number](/reference/overview/feature-extractors/scalar#numerical-extractor) feature. So for a numeric field `budget` describing a movie budget in dollars, it will generate the following feature extractor defitition: ```yaml - source: item.budget type: number name: budget scope: item ``` * **String**: string item fields with low-cardinality are encoded as a [string](/reference/overview/feature-extractors/scalar#string-extractors) feature. So movie genres field is a good candidate for this type of heuristic due to its low cardinality: ```yaml - name: genres type: string source: item.genres scope: item encode: index values: - drama - comedy - thriller - action - adventure - romance - crime - science fiction - fantasy - family - horror ``` If you have a lot of distinct categories, and Metarank does not pick them up (e.g. decides that a category is too infrequent, and you get much less possible categories than expected), you can lower the category frequency threshold with a `--cat-threshold` flag. The default `--cat-threshold` value of 0.003 means that only categories with frequencies above 0.3% are included. * **InteractedWith**: all interaction over low-cardinality fields are translated to [interacted\_with](/reference/overview/feature-extractors/user-session#interacted-with) feature. So if a user clicked on an item with horror genre, other horror movies may get extra points: ```yaml - name: click_genres type: interacted_with scope: user interaction: click field: item.genres ``` * **Relevancy**: if rankings with non-zero relevancy are present, then a feature [relevancy](/reference/overview/feature-extractors/relevancy) is built: ```yaml - name: relevancy type: relevancy ``` * **Vector**: all numerical vectors are transformed into statically-sized features. Vectors of static size are passed through as-is, and variable-length vectors are reduced into a quadruplets of `[min, max, size, avg]` values: ```yaml - name: embedding type: vector field: item.embedding // must be a singular number or a list of numbers scope: item # which reducers to use. optional. Default: [min, max, size, avg] reduce: [vector16] ``` The `all` ruleset contains all `stable` heuristics with an addition of a couple of extra ones: * **Rate**: For all interaction types a [rate](/reference/overview/feature-extractors/counters#rate) feature is generated over multiple typical time windows: ```yaml - name: click_rate type: rate top: click bucket: 1d bottom: impression scope: item periods: - 3 - 7 - 14 - 30 ``` * **InteractionCount**: all interaction types are wrapped into [interaction\_count](/reference/overview/feature-extractors/counters#interaction-counter) feature. ```yaml - name: count_click type: window_count bucket: 1d scope: item interaction: click periods: - 3 - 7 - 14 - 30 ``` ## Why stable ruleset has no counters? The main difference between two rulesets is the lack of `rate`/`window_count` features, which is made deliberately: * `rate`/`window_count` features usually introduce a popularity bias to the final ranking: as people tend to click more on popular items, ML model may attempt to always put popular items on top just because they're popular. * this behavior may be not that bad from the business KPI standpoint, but may make your ranking more static and less affected by past visitor actions. # Running in production These are general recommendations on running Metarank in a production environment. ![Production environment overview](/files/9tT7PYR41Tj2lH5Ku7jV) ## Persistence Metarank provides several [Persistence](/reference/overview/persistence) options, however for production setup we recommend using only [Redis persistance](/reference/overview/persistence#redis-persistence) as it operates separately from running Metarank instances. Redis does not depend on Metarank instances being re-deployed and should be configured with [disc backup](https://redis.io/docs/manual/persistence/). At the moment, Metarank stores only processed events in Redis, so we recommend storing all events separately. ## API Serving [Metarank CLI](/reference/cli) exposes several modes with which you can run Metarank: `standalone` and `serve`. Although `standalone` mode is great for development purposes, it can't be used for production deployment: * standalone mode cannot be scaled as it's not possible to run several instances that point to the same database * you cannot re-train the model without restarting Metarank For production deployment, you should only use the `serve` mode. You can have as many `serve` instances as you need, depending on the load you have and you can perform graceful restarts of Metarank with 0 downtime. Resource consumption of the `serve` mode is relatively low as it performs minimal computations, so you can use cheaper nodes than when training the model. At the moment, Metarank does not provide clustering capabilities out of the box, so you will need to use an external load balancer when deploying multiple API instances. ## Re-training Metarank exposes a `train` mode that re-trains your model based on the calculated features. Training is a long-running process with high memory consumption, which depends on the amount of data that is stored, so we recommend running this process on-demand. You can re-train your model once a week or once a month, so there's no need to keep a large instance online all the time. # Changelog In a human-readable format. For a technical changelog for robots, see [github releases page](https://github.com/metarank/metarank/releases). Check our [blog](https://blog.metarank.ai) for more detailed updates. ## 0.7.9 * expose redis click-through store TTL to config ## 0.7.8 * a bigfix release: slash/semicolon in key/value, kinesis retries ## 0.7.7 * a bugfix release: race condition in cache invalidation, booster native memleak ## 0.7.6 * [API Warmup support](/reference/deployment-overview/warmup) * Rate feature now can be scoped to [ranking.field + item](/reference/overview/feature-extractors/counters#grouping-by-ranking-field) * You can now specify [which eval metrics](/reference/overview/supported-ranking-models#lambdamart) should be computed on training. * Proper handling of [GCP Memorystore Redis](/reference/overview/persistence#redis-support-limitations) ## 0.7.5 * [Unbiased LTR support](/reference/overview/supported-ranking-models#xgboost-and-lightgbm-backend-options) * [Train/test splitting strategy support](/reference/overview/supported-ranking-models#traintest-splitting-strategies) ## 0.7.4 * support for rocksdb-backed file storage ## 0.7.3 * a bugfix release ## 0.7.2 * Support for kv-granular Redis TTLs * Support HF tokenizers for biencoders: now you can run a multi-lingual E5 model in Metarank! ## 0.7.1 * `/inference`: [Inference API](/reference/api#inference-with-llms) to expose bi- and cross-encoders. ## 0.7.0 * Support for [LLM embedding-based content recommendations](/reference/overview/recommendations/semantic) * `field_match` support for [BM25](/reference/overview/feature-extractors/text#bm25-score) * `field_match` support for [LLM bi-encoders](/reference/overview/feature-extractors/text#llm-bi-encoders) * `field_match` support for [LLM cross-encoders](/reference/overview/feature-extractors/text#llm-cross-encoders) * Relevance judgments can now also [be explicit](/reference/event-schema#ranking-event) ## 0.6.4 * a minor bugfix release ## 0.6.3 * [diversity](/reference/overview/feature-extractors/diversity) feature extractor * [scoped rate](/reference/overview/feature-extractors/counters#field-scoped-rates) feature * fixed an important bug with dataset preparation (symptom: NDCG reported by the booster was higher than NDCG computed after the training) - prediction quality should go up a lot. ## 0.6.2 * print NDCG before and after reranking * print statistics for mem usage after training ## 0.6.1 * fix for crash when using file-based clickthrough store ## 0.6.0 Upgrading: note that redis state format has a non backwards compatible change, so you need to reimport the data when upgrading. * [recommendations](/reference/overview/recommendations) support for similar and trending items models. * Local caching for state, the import should be 2x-3x faster. ## 0.5.16 * [expose](/reference/deployment-overview/docker#memory) `JAVA_OPTS` env variable to control JVM heap size. * fix bug for a case when there is a click on a non-existent item. ## 0.5.15 * `cache.maxSize` for redis now disables client-side caching altogether. Makes Metarank compatible with GCP Memstore Redis. * fixed mem leak in clickthrough joining buffer. * lower mem allocation pressure in interacted\_with feature. ## 0.5.14 * interacted\_with feature now supports string\[] fields * fixed a notorious bug with local-file click-through store. ## 0.5.13 * XGBoost LambdaMART impl now supports categorical encoding. * [Event selector support](/reference/overview/supported-ranking-models#event-selectors) when serving multiple models. * [Clickthrough storage configuration](/reference/overview#training) for storing clickthrough data. ## 0.5.12 * Redis AUTH override with [env vars](/reference/overview/persistence#redis-persistence) * Prometheus `/metrics` [endpoint](/reference/deployment-overview/prometheus) * Per-item [ranking fields](/reference/event-schema#ranking-event) ## 0.5.11 * linux/aarch64 support, so docker images on Mac M1 are OK. * \--split option for CLI with [multiple splitting strategies](/reference/cli#training-the-model) * a ton of bugfixes ## 0.5.10 * Redis [TLS support](/reference/overview/persistence#tls-support) * Redis [timeout configuration](/reference/overview#persistence) ## 0.5.9 * [`interacted_with`](/reference/overview/feature-extractors/user-session#interacted-with) feature now has much less overhead in Redis, and supports multiple fields in a single visitor profile. * [click-through events now can be stored in a file](/reference/overview#training), and not inside Redis, also reducing the overall costs of running Metarank * it is now possible to [export lightgbm/xgboost-compatible](/reference/cli#dataset-export) datasets for further hyper-parameter optimization. ## 0.5.8 * bugfix: add explicit sync on /feedback api call * bugfix: config decoding error for field\_match over terms * bugfix: version detect within docker was broken * bugfix: issue with improper iface being bound in docker ## 0.5.7 * [Request latency benchmark](/introduction/performance), with ballpark estimations useful for resource planning. * [Vector feature extractor](/reference/overview/feature-extractors/scalar#vector-extractor) with reducer and autofeature support. * [Redis AUTH support](/reference/overview/persistence#redis-persistence) which is common on managed Redis setups. ## 0.5.6 * [Binary state serialization format](/reference/overview/persistence#state-encoding-formats), which is 2x faster and 4x more compact than JSON * [Multi-arch docker images](/reference/deployment-overview/docker), so metarank can now be run natively on Mac M1/M2. * [Kubernetes Helm chart](/reference/deployment-overview/kubernetes) and an official guide on how to do production deployment on k8s. * [Training dataset export](/reference/cli#training-the-model) for further hyper-parameter tuning. ## 0.5.5 Notable features: * [Rate normalization](/reference/overview/feature-extractors/counters#rate-normalization) support, so having 1 click over 2 impressions is not resulting in a 50% CTR anymore. * [Position de-biasing](/reference/overview/feature-extractors/relevancy#position) based on a dynamic position feature. ## 0.5.4 Most notable improvements: * [AutoML style feature engineering](/how-to/autofeature) based on an existing dataset ## 0.5.3 Highlights of this release are: * added [historical data sorting](https://github.com/metarank/metarank/blob/stabledoc/doc/cli/README.md#historical-data-sorting) sub-command * added [core.clickthrough.maxSessionLength and core.clickthrough.maxParallelSessions](/reference/overview#click-through-joining) parameters to improve memory consumption ## 0.5.2. Highlights of this release are: * added [analytics](/reference/overview#anonymous-usage-analytics) and [error tracking](/reference/overview#error-logging) ## 0.5.1 Highlights of this release are: * Flink is rermoved. As a result only `memory` and `redis` [persistance](/reference/overview/persistence) modes are supported now. * [Configuration file](https://github.com/metarank/metarank/blob/stabledoc/doc/configuration/sample-config.yml) now has updated structure and is not compatible with previous format. * CLI is updated, most of the options are moved to [configuration](/reference/overview). * We have updated the [`validate`](/reference/cli#validation) mode of the CLI, so you can use it to validate your data and configuration. ## 0.4.0 Highlights of this release are: * Kubernetes support: now it's possible to have a production-ready metarank deployment in K8S * Kinesis source: on par with Kafka and Pulsar * Custom connector options pass-through ### Kunernetes support Metarank is a multi-stage and multi-component system, and now it's possible to get it deployed in minutes inside a Kubernetes cluster: * Inference API is just a regular [Deployment](https://github.com/metarank/metarank/blob/master/deploy/kubernetes/deployment.yaml) * Bootstrap, Upload and Update jobs can be run both locally (to simplify things up for small datasets) and inside the cluster in a distributed mode. * Job mabagement is done with [flink-kubernetes-operator](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) See [this doc section](https://docs.metarank.ai/deployment/kubernetes) for details. ### Kinesis source [Kinesis Streams](https://aws.amazon.com/kinesis/data-streams/) can also be used as an event source. It still has a couple of drawbacks compared with Kafka/Pulsar, for example, due to max 7 day data retention it cannot be effectively used as a permanent storage of historical events. But it's still possible to pair it with a AWS Firehose writing events to S3, so realtime events are coming from Kinesis, and historical events are offloaded to S3. Check out [the corresponding part of docs](https://docs.metarank.ai/introduction/configuration/data-sources#aws-kinesis-streams) for details and examples. ### Custom connector options pass-through As we're using Flink's connector library for pulling data from Kafka/Kinesis/Pulsar, there is a ton of custom options you can tune for each connector. It's impossible to expose all of them directly, so now in connector config there is a free-form `options` section, allowing to set any supported option for the underlying connector. Example to force Kinesis use EFO consumer: ```yaml type: kinesis topic: events region: us-east-1 offset: latest options: flink.stream.recordpublisher: EFO flink.stream.efo.consumername: metarank ``` See [this doc section](https://docs.metarank.ai/introduction/03_configuration/data-sources#common-options-for-bootstrapping-connectors) for details. # Building from source Metarank is written in [Scala](https://www.scala-lang.org) and uses an [SBT](https://www.scala-sbt.org) build system. It can be built on Windows, Linux and MacOS in the following way: 1. Clone the `metarank/metarank` repo with your favourite git client. 2. Install SBT, using its [official installation manual](https://www.scala-sbt.org/download.html) for your OS. 3. From shell, run the `sbt assembly` command, and metarank build will be built into `target/scala-2.13/metarank.jar` ```bash $ sbt assembly [info] welcome to sbt 1.7.1 (Eclipse Adoptium Java 11.0.15) [info] loading global plugins from /home/code/.sbt/1.0/plugins [info] loading settings for project metarank-build from plugins.sbt ... [info] loading project definition from /home/code/metarank/project [info] loading settings for project root from build.sbt ... [info] set current project to metarank (in build file:/home/code/metarank/) [info] compiling 28 Scala sources to /home/code/metarank/target/scala-2.13/classes ... [info] compiling 24 Scala sources to /home/code/metarank/target/scala-2.13/classes ... [success] Total time: 48 s, completed Aug 30, 2022, 3:40:06 PM ``` ## Building docker image Docker image can be built the same way as the JAR bundle, with the following SBT command: ```bash $ sbt docker [info] welcome to sbt 1.7.1 (Eclipse Adoptium Java 11.0.15) [info] loading global plugins from /home/code/.sbt/1.0/plugins [info] loading settings for project metarank-build from plugins.sbt ... [info] loading project definition from /home/code/metarank/project [info] loading settings for project root from build.sbt ... [info] set current project to metarank (in build file:/home/code/metarank/) [info] compiling 20 Scala sources to /home/code/metarank/target/scala-2.13/classes ... [info] Assembly up to date: /home/code/metarank/target/scala-2.13/metarank.jar [info] Sending build context to Docker daemon 154.5MB [info] Step 1/6 : FROM adoptopenjdk:11.0.11_9-jdk-hotspot-focal [info] 11.0.11_9-jdk-hotspot-focal: Pulling from library/adoptopenjdk [info] Digest: sha256:4030cc79415a4afc721e5ab8382b93673e118ac6af77ee0eaa02d0a666b88758 [info] Status: Image is up to date for adoptopenjdk:11.0.11_9-jdk-hotspot-focal [info] ---> fd22b5791853 [info] Step 2/6 : RUN apt-get update && apt-get -y install htop procps curl inetutils-ping libgomp1 [info] ---> Using cache [info] ---> ad63a8b493f1 [info] Step 3/6 : ADD 0/metarank.sh /metarank.sh [info] ---> Using cache [info] ---> 5d7a4c02ed99 [info] Step 4/6 : ADD 1/metarank.jar /app/metarank.jar [info] ---> d6b43b43952c [info] Step 5/6 : ENTRYPOINT ["\/metarank.sh"] [info] ---> Running in 2df8553f846e [info] Removing intermediate container 2df8553f846e [info] ---> 40d97b917326 [info] Step 6/6 : CMD ["--help"] [info] ---> Running in c07e81924588 [info] Removing intermediate container c07e81924588 [info] ---> e1caa262b1f1 [info] Successfully built e1caa262b1f1 [info] Tagging image e1caa262b1f1 with name: metarank/metarank:latest [info] Tagging image e1caa262b1f1 with name: metarank/metarank:0.7.5 [success] Total time: 26 s, completed Aug 30, 2022, 3:41:27 PM $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE metarank/metarank 0.7.5 e1caa262b1f1 45 seconds ago 632MB ```