objectiv-analytics icon indicating copy to clipboard operation
objectiv-analytics copied to clipboard

Investigate to make Objectiv Event storage in BQ more snowplow-like

Open borft opened this issue 3 years ago • 1 comments

In the current setup, we send an Objectiv event as an integral custom context, with a very loose schema in Iglu. Although this works fine, it may be possible to split up the event in smaller, less nested objects, using more specific custom contexts. This would have the benefit of:

  • having slightly simpler queries, because of less unnesting
  • having slightly simpler queries, because of more predictable column naming (name of a global context, rather than a numerical index for example)
  • stricter validation, because in stead of one schema we could have multiple ones, that are less generic
  • be more in line with other custom contexts in Snowplow

The idea:

  1. generate iglu schema for all (global) contexts from the Objectiv Taxonomy schema
  2. have the collector:
  • generate sp events with specific custom contexts per global contexts
  • generate an object to contain the location stack hierarchy (remove a layer of nesting, whilst preserving order and structure)
  • map objectiv event properties on existing properties in the sp schema
  1. Setup a testing pipeline

borft avatar Aug 02 '22 15:08 borft

  1. It seems all custom contexts in BQ are mapped to a REPEATED RECORD by the mutator. This means all custom contexts are a list of 0 or more instances.

borft avatar Aug 02 '22 16:08 borft