cmd/gitannex: Port unit tests to fstest

[dmcardle]

What is the purpose of this change?

This enables the unit tests to run on any given backend, via the -remote flag, e.g. go test -v ./cmd/gitannex/... -remote dropbox:.

We should also port the gitannex e2e tests at some point.

Was the change discussed in an issue or in the forum before?

Issue: #7984

Checklist

• [x] I have read the contribution guidelines.
• [x] I have added tests for all changes in this PR if appropriate.
• [x] I have added documentation for the changes if appropriate.
• [x] All commit messages are in house style.
• [x] I'm done, this Pull Request is ready for review :-)

[dmcardle]

@ncw Got the fast unit tests ported over to fstest, FYI.

I took a stab at enrolling these changes in the integration tester. Can you take a peek to see if I did it right?

[ncw]

Is this ready for another review?

[ncw]

How are you doing with this @dmcardle - I think we are nearly there aren't we?

[dmcardle]

How are you doing with this @dmcardle - I think we are nearly there aren't we?

Hey @ncw, my inbox was on fire, sorry for missing these!

I'm making some slow progress, but this is still not ready to merge. I got the fstest-style tests working locally and on dropbox, but I seem to have regressed the e2e tests. I'll let you know when it's ready for another look!

[dmcardle]

@ncw PTAL!

[dmcardle]

Hey @ncw, PTAL when you can! It's probably worth getting these tests running against the Dropbox backend on the off chance that it catches something related to #8068.

[dmcardle]

@ncw, I think this is in good shape! I'm unsure about the yaml change in the final commit that registers this with fstest, but I suppose it's easily reverted if it breaks something.

[dmcardle]

Hey @ncw, PTAL!

I definitely need eyes on my changes in fstest/test_all/config.yaml. I'm trying to register the gitannex tests with the integration tester, but I don't really know what I'm doing or how to test that it worked. I think the rest of the PR is in decent shape :)

FYI, if we can get these tests running against the B2 backend, it may help resolve #8349.

[ncw]

@dmcardle the results are in! https://pub.rclone.org/integration-tests/current/

It is 28 failures out of 93 which is not terrible :-)

All of the docker based backends failed with something like this (here is TestFTPProftpd)

These remotes don't exist in the config file which is the problem I think. They are defined with environment variables by the test_all script.

You can test this yourself with

$ cd root of rclone source
$ go install ./...
$ test_all -remotes TestFTPProftpd: -tests cmd/gitannex

Also the tests seemed to take 1 hour to timeout after they went wrong

=== RUN   TestGitAnnexFstestBackendCases/HandlesInit
    run.go:180: Remote "ftp://172.17.0.3:21/rclone-test-lijezew7quzu", Local "Local file system at /tmp/rclone652195156", Modify Window "1s"
    gitannex_test.go:285: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:285
        	            				/home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:419
        	            				/home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1216
        	Error:      	Not equal: 
        	            	expected: "INITREMOTE-SUCCESS\n"
        	            	actual  : "INITREMOTE-FAILURE remote does not exist: TestFTPProftpd:\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-INITREMOTE-SUCCESS
        	            	+INITREMOTE-FAILURE remote does not exist: TestFTPProftpd:
        	            	 
        	Test:       	TestGitAnnexFstestBackendCases/HandlesInit
    gitannex_test.go:1207: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/cmd/gitannex/gitannex_test.go:1207
        	            				/home/rclone/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700
        	Error:      	Received unexpected error:
        	            	remote does not exist: TestFTPProftpd:
        	Test:       	TestGitAnnexFstestBackendCases/HandlesInit
2025/02/27 11:29:56 DEBUG : ftp://172.17.0.3:21/rclone-test-lijezew7quzu: closing 1 unused connections
panic: test timed out after 1h0m0s
	running tests:
		TestGitAnnexFstestBackendCases (1h0m0s)
		TestGitAnnexFstestBackendCases/HandlesInit (1h0m0s)

goroutine 48 [running]:
testing.(*M).startAlarm.func1()
	/home/rclone/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2373 +0x385
created by time.goFunc
	/home/rclone/go/pkg/mod/golang.org/[email protected]/src/time/sleep.go:215 +0x2d

It doesn't seem to like any remotes with commas in them, eg TestAzureBlob,directory_markers: which should work. as TestAzureBlob worked fine.

You can test this with

rclone/rclone/cmd/gitannex$ go test -v -remote :local,description=hello:/tmp/src

It doesn't like :memory: also.

And I think there are a couple of genuine failures, eg cloudinary.

So I think a little tweak to remote handling should fix up nearly all of those. I didn't investigate what tweak though ;-)