raccoon icon indicating copy to clipboard operation
raccoon copied to clipboard

Feat: Kinesis sink

Open turtleDev opened this issue 1 year ago • 0 comments

This PR adds support for AWS Kinesis sink(#78).

Configuration

This change introduces the following new configurations

  • a new PUBLISHER_TYPE value called kinesis
  • PUBLISHER_KINESIS_AWS_REGION region with the target kinesis stream resides (fallbacks back to AWS_REGION)
  • PUBLISHER_KINESIS_CREDENTIALS path to AWS Credentials file (defaults to `$HOME/.aws/credentials)
  • PUBLISHER_KINESIS_STREAM_PROBE_INTERVAL_MS time interval for stream readiness check.
  • PUBLISHER_KINESIS_STREAM_AUTOCREATE whether Raccoon should create streams that don't exist.
  • PUBLISHER_KINESIS_STREAM_MODE when creating streams, what mode to set. (ON_DEMAND or PROVISIONED)
  • PUBLISHER_KINESIS_STREAM_SHARDS when creating streams, how many shards to configure.
  • PUBLISHER_KINESIS_PUBLISH_TIMEOUT_MS how long to wait for before aborting a publish operation (per batch)

Metrics

This change also introduces the following new metrics

  • kinesis_messages_delivered_total total number of events processed (includes false negatives)
  • kinesis_messages_undelivered_total total number of events that failed
  • kinesis_unknown_stream_failure_total events that failed to publish to an unknown stream
  • kinesis_producebulk_tt_ms batch production time.

turtleDev avatar Jun 26 '24 17:06 turtleDev