objectiv-analytics icon indicating copy to clipboard operation
objectiv-analytics copied to clipboard

Flatten contexts in SP/BQ

Open borft opened this issue 3 years ago • 0 comments

In this PR:

  • restructure the mapping of Objectiv events on the Snowplow event
  • instead of one big custom context, split up the event
  • alle global contexts get their own specific context and iglu schema
  • the location_stack is stored as 1 entity, with its own schema as well:
    • to guarantee / preserve the order
    • to support additional properties

In order for this to work, the new contexts should be added to the iglu server. After that, the snowplow-bigquery-mutator should automagically add columns for any new contexts encountered in an event.

The setup has been tested with the postgres-pubsub-loader. bigquery-pubsub-loader and the s3-kinesis-loader.

NOTE this is a breaking change for the Snowplow pipeline, as this will change the format in which events are persisted (and thus break any query that relies on the original objectiv taxonomy custom context).

borft avatar Aug 02 '22 15:08 borft