Ambient icon indicating copy to clipboard operation
Ambient copied to clipboard

Set up automatic CI for examples/

Open philpax opened this issue 3 years ago • 2 comments

The runtime should have an operating mode where it will run the client and server as normal, and exit with a 0 or error status code depending on one of these:

  1. Error: The client fails to connect to the server
  2. Error: A timeout after initial startup expires (e.g. 5 seconds after the client expires)
  3. Success: The guest code calls a runtime function to notify it that it is done. Upon doing so, the runtime will take a screenshot of the current state. If the screenshot does not match a stored screenshot, it fails. Otherwise, it succeeds.

The examples would have something like

runtime::ci_test(|success, fail| async {
  sleep(1).await;
  if entity::exists(id) { 
    success();
  } else {
    fail("The entity did not exist.".to_string());
  }
});

at the end of their execution. This is not automatic as the guests should be able to specify their own success conditions (e.g. "was this entity created?") and to lock their state for the screenshot (e.g. for the asset loading example, it should lock its cube in place for the screenshot)

philpax avatar Feb 09 '23 10:02 philpax

So I'm thinking a first step towards this would be to just add these three things;

  • [ ] Headless mode
  • [ ] A screenshot function (and after than maybe a screenshot_compare, which would return true if the screenshot was different from the one on disk). This would need #6 though.
  • [ ] An exit(code) function

If we have that, we can build simple test modules that can run on the ci.

FredrikNoren avatar Feb 24 '23 10:02 FredrikNoren

Seems pretty reasonable. I'd suggest wiring up environment variables with WASI, and then we can tell it the screenshot to match (assuming we can pack image data into an env var) and any other conditions that it should know about. Let's give this a shot once #6 is in.

philpax avatar Feb 24 '23 16:02 philpax