redhat-developer
Automate testing of QuickStart odo.dev
/kind user-story /area-testing
User Story
As a QE I want to automate the test of QuickStart in odo.dev documentation updates So that we can ensure that when following the documentation any steps/commands described in the documentation will complete successfully. Serves as acceptance tests, all test should pass.
Acceptance Criteria
- [x] Research framework to be used
- [x] Research tools that can extract commands from documentation
- [x] The framework should extract commands from the documentation
- [ ] Execute every command described in the documentation.
- [ ] Report failing commands
- [ ] Serves as acceptance testing, all commands should pass
Links
- Related Epic (mandatory):
/kind user-story
We can parse https://raw.githubusercontent.com/openshift/odo/main/docs/website/docs/getting-started/quickstart.md to extract commands from within shell blocks, write them in sequence to a file, then extract commands, in sequence, from the created file and execute each command verifying all commands pass.
@dharmit @rnapoles-rh Can you guys summarize the potential problems that we want to fix? From what I could gather:
- We need to set a context for every command that we test, for e.g., before running
odo service create postgres-op-4.5/PgCluster, we'll need to ensure postgres operator is installed in the cluster. - We require a way to extract commands that should be tested and ignore the others. For this we could follow a set of rule, for e.g. the command should not be followed by a prefix $.
What else am I missing?
@feloy Can you also document your idea about using documentation as a source of truth for integration tests? Using the cucumber framework like SBO does?
The proposed solution will require to have additional processes when modifying / adding documentation. Alternatively we can create a test suite that executes all commands in all the guides/tutorials, make this test suite required to pass. If it passes that does not warranty that new content was tested, so we need to track the last commit ID, if it does not match then we need to manually identify the differences.
The following perl script extracts commands from a target guide and writes them to a text file:
use strict;
use warnings;
use LWP;
my $docToParse = 'doc_to_parse.txt';
my $extractedCommands = 'extractedCommands.txt';
my @commands = ("odo","kubectl","oc","git","cd","tree","mkdir","minikube");
my $shellStart = 0;
my $shellEnd = 0;
my $nextLnIsCmd = 0;
my $browser = LWP::UserAgent->new;
$browser->ssl_opts(
verify_hostname => 0 # turns off ssl verification
);
my $url = 'https://raw.githubusercontent.com/openshift/odo/main/docs/website/docs/getting-started/quickstart.md';
# Read Raw file of the documentation to be tested
my $response = $browser->get( $url );
die "Can't get $url -- ", $response->status_line
unless $response->is_success;
# Write content of raw file to local file
open(FH, '>', $docToParse) or die $!;
print FH $response->content;
close(FH);
open my $info, $docToParse or die "Could not open $docToParse: $!";
open(FH, '>', $extractedCommands) or die $!;
# Parse the file to find commands
while( my $line = <$info>) {
$line =~ s/^\s+//;
if (index($line, "```shell")!= -1 ) {
$shellStart = 1;
$shellEnd = 0;
}elsif (index($line, "```")!= -1 ) {
$shellEnd = 1;
}
if ($shellStart == 1 && $shellEnd == 0 && index($line, "minikube ip") == -1) {
if (substr($line, 0, index($line, ' ')) ~~ @commands) {
print("$line");
print FH $line;
}
}
last unless defined $line;
}
close (FH);
close ($info);
The following test script parses the text file with the commands and executes them expecting the tests to pass:
package docs
import (
"bufio"
"fmt"
"log"
"os"
// "os/exec"
"path/filepath"
"github.com/openshift/odo/tests/helper"
. "github.com/onsi/ginkgo"
)
var _ = Describe("odo.dev guides and tutorials tests", func() {
var componentName, projectDirPath string
var projectDir = "/projectDir"
var commonVar helper.CommonVar
var _ = BeforeEach(func() {
commonVar = helper.CommonBeforeEach()
componentName = helper.RandString(6)
projectDirPath = commonVar.Context + projectDir
helper.MakeDir(projectDirPath)
err_rn := os.Rename(filepath.Join("..","..","..","extractedCommands.txt"), filepath.Join(commonVar.Context,"extractedCommands.txt"))
if err_rn != nil {
log.Fatal(err_rn)
}
helper.Chdir(commonVar.Context)
helper.Chdir(projectDirPath)
})
var _ = AfterEach(func() {
helper.CommonAfterEach(commonVar)
})
Context("context", func () {
It("should run all commands in gettingStarted", func() {
f, err := os.Open(filepath.Join(commonVar.Context,"extractedCommands.txt"))
if err != nil {
log.Fatal(err)
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fmt.Println("Executing [",scanner.Text(), "]")
helper.Cmd(scanner.Text(), commonVar.Project, componentName).ShouldPass()
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
})
})
})
When executing kubectl create -f https://operatorhub.io/install/postgresql.yaml
get the following error
error: unable to recognize "https://operatorhub.io/install/postgresql.yaml": no matches for kind "Subscription" in version "operators.coreos.com/v1alpha1"
Get the same when running the command manually. It used to work just fine.
When executing
kubectl create -f https://operatorhub.io/install/postgresql.yamlget the following errorerror: unable to recognize "https://operatorhub.io/install/postgresql.yaml": no matches for kind "Subscription" in version "operators.coreos.com/v1alpha1"
It loops like you missed the first bullet point in the "Prerequisites" section (https://odo.dev/docs/getting-started/cluster-setup/kubernetes/#installing-the-operator-lifecycle-manager-olm)
When executing
kubectl create -f https://operatorhub.io/install/postgresql.yamlget the following errorerror: unable to recognize "https://operatorhub.io/install/postgresql.yaml": no matches for kind "Subscription" in version "operators.coreos.com/v1alpha1"It loops like you missed the first bullet point in the "Prerequisites" section (https://odo.dev/docs/getting-started/cluster-setup/kubernetes/#installing-the-operator-lifecycle-manager-olm)
You are right, the script looks for commands to be executed within shell blocks, the prerequisites indicates OLM as a prerequisite, but does not provide a command to install it. I will install manually for now and will open a PR to update the doc. Thx
This might be of help - https://cerbos.dev/blog/keeping-documentation-in-sync-with-source-code. I haven't read the article completely myself, hence "might". :)
