[Elastic Agent] Add tests for command line flags

[ruflin]

The Elastic Agent with fleet-server offers a wide variety of command line flags. We should have e2e test in place to make sure all these flags work as expected and keep working as expected. To see all the flags, run ./elastic-agent -h or to get more specific flags, ./elastic-agent enroll -h.

I would consider the enroll command the most important to start the testing on all the flags. Here are the current available flags:

./elastic-agent enroll -h
This will enroll the Agent into Fleet.

Usage:
  elastic-agent enroll [flags]

Flags:
  -p, --ca-sha256 string                    Comma separated list of certificate authorities hash pins used for certificate verifications
  -a, --certificate-authorities string      Comma separated list of root certificate for server verifications
  -t, --enrollment-token string             Enrollment token to use to enroll Agent into Fleet
      --fleet-server-cert string            Certificate to use for exposed Fleet Server HTTPS endpoint
      --fleet-server-cert-key string        Private key to use for exposed Fleet Server HTTPS endpoint
      --fleet-server-es string              Start and run a Fleet Server along side this Elastic Agent connecting to the provided elasticsearch
      --fleet-server-es-ca string           Path to certificate authority to use with communicate with elasticsearch
      --fleet-server-host string            Fleet Server HTTP binding host (overrides the policy)
      --fleet-server-insecure-http          Expose Fleet Server over HTTP (not recommended; insecure)
      --fleet-server-policy string          Start and run a Fleet Server on this specific policy
      --fleet-server-port uint16            Fleet Server HTTP binding port (overrides the policy)
      --fleet-server-service-token string   Service token to use for communication with elasticsearch
  -f, --force                               Force overwrite the current and do not prompt for confirmation
  -h, --help                                help for enroll
  -i, --insecure                            Allow insecure connection to Kibana
  -k, --kibana-url string                   URL of Kibana to enroll Agent into Fleet
      --staging string                      Configures agent to download artifacts from a staging build
      --url string                          URL to enroll Agent into Fleet

Global Flags:
  -c, --c string                     Configuration file, relative to path.config (default "elastic-agent.yml")
  -d, --d string                     Enable certain debug selectors
  -e, --e                            Log to stderr and disable syslog/file output
      --environment environmentVar   set environment being ran in (default default)
      --path.config string           Config path is the directory Agent looks for its config file (default "/Users/ruflin/Dev/gopath/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64")
      --path.home string             Agent root path (default "/Users/ruflin/Dev/gopath/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64")
      --path.logs string             Logs path contains Agent log output (default "/Users/ruflin/Dev/gopath/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64")
  -v, --v                            Log at INFO level

It would be great to have tests for passing but also tests that validate that things don't work if it is not setup correctly. For example use an enrollment token in place of the service token and similar scenarios.

[ruflin]

@EricDavisX The above issue is a very generic description. Would it be helpful to specific more details on the cases that should be tested or better leave this to the person implementing it. I definitively prefer the second that the person taking on this task will think through all the possible scenarios and what could go wrong.

[EricDavisX]

Someone definitely needs to do the work to review what tests we have in Unit tests on Kibana / Agent / Fleet-server side against what we have in e2e-testing already to build out what we want to further create. There are more people who know Go and can automate than know the depths of the Fleet code bases well enough to document the tests needed, so I prefer we do the leg work up front - instead of leaving for later. But, in the best of a worst scenario... having a ticket at all is excellent. Further still though, some of the items here (indeed most?) may already be noted elsewhere in the repo as needing coverage. @ruflin thanks for putting the ticket in.

As discussed with the team and team leadership prior, the real effort needs to focus around making usage of this repo by the Fleet, Fleet-Server, and Agent teams a regular task as part of feature dev, we have thus far not achieved that but are inching closer.

[mdelapenya]

@ruflin I'd like to know if there is a way to test the flags in a non-e2e manner. Why? Because the combination of 25 flags (for the root command), without repetition, could be huge, and the cost of provisioning all the services for all those combinations could be very expensive in time. If we start adding subcommands' flags, then that number escapes from orbit 😄

I wonder if, as Eric wisely points out, there is a way to unit test that in a project (the one receiving the flags?) to accept/discard flags depending on that combination (or at least a prioritised subset of it).

OTOH, in the e2e tests side we can verify a reduced subset representing the critical paths, giving us an overview for the satisfaction of the contract between the agent and the other party (ES/Fleet-Server), which is represented by the flags.

I'm really happy to discuss the test strategy for the flags :)

[ruflin]

The flags I was mostly thinking of when writing this issue was everything around enroll and certificates. It requires a full setup to be tested.

But you are correct, many of these flags should potentially be tested without requiring e2e and the full setup and in Beats directly instead.

[mdelapenya]

The flags I was mostly thinking of when writing this issue was everything around enroll and certificates. It requires a full setup to be tested.

But you are correct, many of these flags should potentially be tested without requiring e2e and the full setup and in Beats directly instead.

Cool! It would be great if we explicitly write those use case down here. Something like:

Scenario Outline: Enrolling agent with flags
When the agent is enrolled using "<flags>" flags
Then # add your expected behaviour here
Examples:
| flags                         |
| --flag1 value1                |
| --flag1 value1 --flag2 value2 |
| --booleanFlag3                |
| ...                           |

[ruflin]

This goes back to my comment:

The above issue is a very generic description. Would it be helpful to specific more details on the cases that should be tested or better leave this to the person implementing it.

[mdelapenya]

I'd suggest involving the functional team in starting to write the scenarios, as they are the ones with more valuable knowledge about how to use the flags. Then we (Eric and us?) can review those scenarios and ask questions :)

[ruflin]

Here we go with the scenarios for fleet-server

1. Use --fleet-server-cert and test for that setup fleet-server works as expected
2. Use --fleet-server-cert with and invalid cert and verify that setup fails
3. Use --fleet-server-cert-key and test for that setup fleet-server works as expected
4. Use --fleet-server-cert-key and an invalid key and verify that setup fails
5. Use --fleet-server-es-ca to communicate with ES
6. Use --fleet-server-insecure-http and ensure that fleet-server is exposed through http.
7. Don't use --fleet-server-insecure-http and ensure it is all https
8. Use --fleet-server-policy to pick up a custom policy which does not contain is_default_fleet_server
9. Use --fleet-server-port and check that the port is changed as expected
10. Use --fleet-server-service-token for the setup

I expect some of the above tests are partially already covered by existing tests. Other with the certs might be tricker to add as it requires creation of certificates.