spring-cloud
Validate the app coordinates when registering apps
When registering a new app, DSL allows for any valid URI. For example:
$ app register --name time-source --type source --uri file://Users/olegz/Downloads/time-source/target/classes
And then it displays the success message
Successfully registered application 'source:time-source'
But in reality the above application is doomed to fail since the 'file' based URI does not point to a boot JAR.
What makes it even worst is that in SCDF you would typically deploy several apps and then create a stream and then deploy it just to see nothing happening (good, bad or indifferent) untill one actually looks at the logs to see
$ cat stderr_0.log
Error: Invalid or corrupt jarfile /Users/olegz/Downloads/time-source/target/classes
While at some point in the future when things settle down we should probably do full JAR validation to ensure that it is indeed a valid boot JAR, at this point I would at least validate that the 'file' based URI points to a valid resource and that resource is a file and is a JAR. And if not notify the user in the console right away
Based on comments from https://github.com/spring-cloud/spring-cloud-dataflow/pull/1498 the proposed approach is not feasible. Perhaps need to come up with something clever on the deployer(s) side to be able to communicate to the shell user that things didn't go right
We have discussed supporting an option for eagerly fetching resources (at the time of registration) which would then or course mean failing fast in case of invalid resources.
That is yet another valid point @markfisher. Basically now I believe there are two types of enhancements here. That may require an additional flag since one should be able to choose if they want such eager/fail-fast approach (see explanation below)
- Ability to eagerly fetch and validate resources that they are "deployable" (i.e., exist and in a proper/supported format)
- Notifying the shell/UI user (outside of looking at the logs) that something went wrong with the particular microservice during deployment of the stream.
Basically it's pretty clear now that even if #1 succeeds the service may still fail. Further more as @ericbottard pointed out there may be some clever scenarios where the URI may indeed point to a non-existing 'yet' artifact, but that does not necessarily mean that the composed stream is not valid since such artifact could be introduced later especially in the ever volatile cloud. So I may have something like foo | bar | baz where bar may not be available, yet it doesn't mean that foo can not run since the only dependency for foo is '|' which means it can produce and send data. Same goes for baz (except that it won't have any data to consume). In other words think of some clever re-try scenarios to check if artifact is now available etc.
Anyway, i'll keep the issue open for the sake of gathering opinions/stories and tracking, but as I mentioned before the #2 will most likely be a deployer issue.
As next steps, we would proactively apply the validation routines that are backing the stream validate, task validate, and app validate commands.
Based on the discussion, we would do:
- Always validate on
app register ...command - Also, provide an option to skip validation via
app register .. --skip-validation - Document the new option
Data Flow doesn't know about the existence of the registered App artifacts. App deployment (and existence check) is delegated to Skipper and in turn to the concrete platform deployer (e.g. local, k8s, CF ...).
While Dataflow can not check for the existence of the apps it can check if the metadata associated with these apps exists. This could be a reasonable approximation.
- For Docker apps, the metadata is stored inside the application images and therefore the metadata resource existence is equivalent to app resource existence check.
- Similarly for Jar app resource without specified metadata artifact, the metadata is retrieved from the app artifact itself. Therefore the metadata resource existence is be equivalent to app resource existence check.
- For jar apps with specified additional metadata jar artifact we can only assume that the existence of metadata resource corresponds to correct app jar coordinates.
