Changelog
In a human-readable format. For a technical changelog for robots, see github releases page. Check our blog for more detailed updates.

Notable features:

Most notable improvements:

Highlights of this release are:

Highlights of this release are:

Highlights of this release are:
  • Flink is rermoved. As a result only memory and redis persistance modes are supported now.
  • Configuration file now has updated structure and is not compatible with previous format.
  • CLI is updated, most of the options are moved to configuration.
    • We have updated the validate mode of the CLI, so you can use it to validate your data and configuration.

Highlights of this release are:
  • Kubernetes support: now it's possible to have a production-ready metarank deployment in K8S
  • Kinesis source: on par with Kafka and Pulsar
  • Custom connector options pass-through

Metarank is a multi-stage and multi-component system, and now it's possible to get it deployed in minutes inside a Kubernetes cluster:
  • Inference API is just a regular Deployment
  • Bootstrap, Upload and Update jobs can be run both locally (to simplify things up for small datasets) and inside the cluster in a distributed mode.
  • Job mabagement is done with flink-kubernetes-operator
See this doc section for details.

Kinesis Streams can also be used as an event source. It still has a couple of drawbacks compared with Kafka/Pulsar, for example, due to max 7 day data retention it cannot be effectively used as a permanent storage of historical events. But it's still possible to pair it with a AWS Firehose writing events to S3, so realtime events are coming from Kinesis, and historical events are offloaded to S3.
Check out the corresponding part of docs for details and examples.

As we're using Flink's connector library for pulling data from Kafka/Kinesis/Pulsar, there is a ton of custom options you can tune for each connector. It's impossible to expose all of them directly, so now in connector config there is a free-form options section, allowing to set any supported option for the underlying connector.
Example to force Kinesis use EFO consumer:
type: kinesis
topic: events
region: us-east-1
offset: latest
options:
flink.stream.recordpublisher: EFO
flink.stream.efo.consumername: metarank
See this doc section for details.
Copy link
Edit on GitHub
On this page
0.5.5
0.5.4
0.5.3
0.5.2.
0.5.1
0.4.0
Kunernetes support
Kinesis source
Custom connector options pass-through