nasa
Add positive check to cFS execution in CI
Is your feature request related to a problem? Please describe. CI in #39 only checks for warn/err/fail in cFS output
Describe the solution you'd like Enforce the existence of the following message: "CFE_ES_Main entering OPERATIONAL state"
Describe alternatives you've considered Any other positive check would work, above was the simplest I could think of.
Additional context None
Requester Info Jacob Hageman - NASA/GSFC
If the 1Hz tick and clocks are working one should also see "Stop FLYWHEEL" event from CFE_TIME approximately 2 seconds after that operational message.
But overall I would advocate a more holistic approach to sanity checking the build. For instance the latter event should be checked by subscribing to the event stream and checking for event CFE_TIME_FLY_OFF_EID. Checking the stdout for specific messages is neither reliable nor sustainable for a longer term solution.
Yeah, definitely debated internally what level to go to. Any positive check is better than none, but I agree a specific message on stdout is pretty fragile.
Certainly encourage ideas and thought about really what's useful/appropriate for this Travis CI check.
We lack documentation covering the bigger picture/goals of more sophisticated test setups like dockers emulated platforms, etc but from that perspective/context (and eventually automating build verification) maybe a non-empty file for this test is good enough (and less dependent on text changing)? I'm trying to catch the case where we get a false pass because nothing happened. I'd think it'd be sufficient to just check that something happened (and no warn/fail/err).
It would indeed be nice to automatically check the behavior of the executable, in addition to running unit tests.
In this context, I have implemented a proof of concept consisting of a fully dockerized end-to-end automated test. In substance, this solution builds and runs the cFS (currently: on CentOS 7, Ubuntu 18.04, or Ubuntu 20.04). Simultaneously, a pseudo-GSW—in fact: a program running the test—tries to enable the telemetry of the former and, if successful, asserts that an expected event (e.g. Stop FLYWHEEL) has been emitted by the FSW. Such an assertion is performed by inspecting the datagrams sent by the FSW through the network connection, not via stdout or stderr.
When the test passes, the testing tool exits with code 0. When one modifies the cFS code to simulate an error (e.g. make the FSW not emit a Stop FLYWHEEL), the error is indeed detected by the tool, and a non-zero exit code is returned. This is shown by the following screen captures.
(Note: the green and red borders are for illustrative purposes only).
In short, this prototype checks the cFS execution by simulating GSW/FSW interactions, where the GSW is, in fact, a test runner. And because the tool is fully dockerized, it would be conceivable to directly integrate it into the CI process in the future.
Obviously, this is not a full-fledged testing solution at the moment, but I think that it could help to illustrate the possibilities offered by this approach as well as concretely show its limitations and the technical challenges that it introduces.
In this context, I am wondering if it would be reasonable for me to open a pull request that would submit this tool as well as explain the underlying methodology in more detail.
@lethuillierg Excellent ideas, and appreciate the input! We have discussed a docker approach internally and possibly also testing for multiple platforms/OS's utilizing QEMU, eventually building up to a continuous verification system across multiple architectures. Tagging @jphickey @astrogeco and @acudmore if they have more inputs. I encourage opening a pull request!