CGWire Integration

[tokejepsen]

Goal

Full synchronization with CGWire; #503

Implementation

Initially this PR is mainly tests so we can experiment with the code. Currently only have tests for Avalon to CGWire.

• [x] CGWire basic integration tests.
• [x] CGWire full sync.
• [x] Non-hardcoded determination of CGWire asset/shot.
• [x] Non-hardcoded CGWire host and login.
• [x] Store Avalon Events.
• [ ] Store synced event id to pick up syncing after shutdown.
• [ ] On demand syncing Avalon to CGWire (register sync module).
• [ ] Avalon basic integration tests.
• [ ] Avalon full sync.
• [ ] Store synced event id to pick up syncing after shutdown.
• [ ] On demand syncing CGWire to Avalon (event listener).
• [ ] Travis reports false positive. Load by project name is failing.
• [ ] Fix CI (Azure pipelines)

NOTE: This is a work-in-progress PR.

[tokejepsen]

Current State

• Tests for syncing project, asset, episode, sequence, shot.
• Testing for:
  • name of Avalon entity matches with CGWire entity.
  • existence of avalon_id in CGWire entity's data field.
  • tasks in project config matches CGWire task types.
  • tasks on assets and shots match.
• Clears the databases between tests.
• Full syncing only atm.
• Syncing determines asset/shot with silos names.
• Syncing determines episode/sequence/shot with visual children:
  • No visual children == skip. CGWire requires a minimum of a sequence, so 1 child minimum.
  • One visual child == sequence hierarchy.
  • Two visual children == episode hierarchy.
• Asset and shot are currently hardcoded to be in silos; film and assets.
• Finds existing entities by avalon_id first, then label, then name.

[tokejepsen]

Currently look at storing the io events in Avalon database. Thinking that it might be better to store them on the project, so dropping an entire project will also remove the event log. Storing all the events might also reduce performance when looking up assets.

The other option is to have a separate document with the events. Something like:

client = pymongo.MongoClient(os.environ["AVALON_MONGO"])
db = client["avalon_events"]

This document would mirror the project names, but only store events.

[tokejepsen]

Current State

• Entity types in CGWire are determined from two environment variables; AVALON_CGWIRE_SHOT_PARENTS and AVALON_CGWIRE_ASSET_PARENTS. These can be multiple names of parents since we can have multiple visual root parent (folders) in Avalon.
• CGWire login details are determined with three environment variables; CGWIRE_HOST, CGWIRE_USERNAME and CGWIRE_PASSWORD.
• io events are stored in the database in the same collection as the project with following schema;

    event = {
        "schema": "avalon-core:event-1.0",
        "type": "event",
        "datetime": datetime.now(pytz.utc).isoformat(),
        "method": "insert_one",
        "args": json_util.dumps([item]),
        "kwargs": {}
    }

Events are validated and tested.