matrix-org
Specify what matrix version complement is testing
Currently we test a bunch of features while not specifying the exact matrix version(s) we test for, while it’s fine for now, it should be specified somewhere in the near future.
As part of #654 I was thinking of having dedicated directories for matrix spec versions e.g ./tests/v1.1. This has numerous benefits:
- MSCs have a natural progression from
./tests/msc1234to./tests/v1.1when they land in the spec. - Users of Complement can opt-in to a set of matrix spec tests e.g
go test ./tests/v1.1 ./tests/v1.2 ./tests/v1.3 - It provides an "obvious" grouping when adding new tests. If it's something in v1.x you know immediately where it should go, else it's an MSC.
The downside to this is what to do when things are backwards incompatible (breaking changes) in the spec. If the breaking change is with core machinery / helpers used to set up other tests, this is a problem and will likely need to be interfaced out. A good example:
Replace homeserver authorization approach with an Authorization header instead of access_token when talking to the application service, as per MSC2832. (#1200)
In v1.4 would break all AS tests if they were using the old style of authorization. We don't want to support all forms to the end of time, so it may be enough to just support the latest version, or have some kind of grace period before changing things. In practice, sytest seems ok just doing whatever, though whatever is typically ad-hoc, preferring the oldest thing because no one maintains it... For now I would propose the pragmatic approach of keeping helpers up-to-date and interfacing in a PR if we need to for certain servers which aren't caught up.
Another downside is what happens if a feature is removed in the spec, e.g a feature is added in v1.1 and removed in v1.3, then obviously v1.1 tests for that feature will no longer be passing. This is just a hypothetical problem currently, as the only precedent is groups which were removed before versioned releases were made. I propose doing nothing here and tackling it if/when we come to it, rather than devising a complicated scheme up front for a feature which isn't being exercised. In all likelihood, we would probably just break up v1.1 into two: v1.1 and v1.1-deprecated and just not run the deprecated tests e.g go test ./tests/v1.1.
Sounds good, though one suggestion; when a feature is deprecated, maybe move it to the folder with the final version that supported or deprecated that feature? This way, it may cause a bit of churn, but it would still result in tidy folders.