objectiv-analytics
objectiv-analytics copied to clipboard
Investigate to make Objectiv Event storage in BQ more snowplow-like
In the current setup, we send an Objectiv event as an integral custom context, with a very loose schema in Iglu. Although this works fine, it may be possible to split up the event in smaller, less nested objects, using more specific custom contexts. This would have the benefit of:
- having slightly simpler queries, because of less unnesting
- having slightly simpler queries, because of more predictable column naming (name of a global context, rather than a numerical index for example)
- stricter validation, because in stead of one schema we could have multiple ones, that are less generic
- be more in line with other custom contexts in Snowplow
The idea:
- generate iglu schema for all (global) contexts from the Objectiv Taxonomy schema
- have the collector:
- generate sp events with specific custom contexts per global contexts
- generate an object to contain the location stack hierarchy (remove a layer of nesting, whilst preserving order and structure)
- map objectiv event properties on existing properties in the sp schema
- Setup a testing pipeline
- It seems all custom contexts in BQ are mapped to a
REPEATED RECORDby the mutator. This means all custom contexts are a list of 0 or more instances.