Migrate to a maintained yaml package

[xqqp]

Testify uses the yaml package provided by https://github.com/go-yaml/yaml. After no development for a long period, a few days ago it finally got archived. Testify should migrate to another yaml package which is actively maintained. There are some forks of go-yaml which could be considered. There is also https://github.com/goccy/go-yaml, which is not a fork but largely API compatible.

[dolmen]

Note that the default YAML library is still gopkg.in/yaml.v3, that libraray is already user-pluggable (replaceable) in your own test suite by using build tags. See assert/yaml/yaml_custom.go and #1579.

So anyone using testify v1.10.0 can already switch to use any other YAML library right now and this is documented here: https://pkg.go.dev/github.com/stretchr/testify/assert/yaml

[dolmen]

At this point I would prefer to go further and deprecate assert.YAMLEq and assert.YAMLEqf than to deal with compatibility issues between two YAML libraries.

I can't imagine that the two libraries work exactly the same and that the replacement will not raise issues. Instead removing one dependency would make maintenance of the core easier: one less problem to deal with.

[founderio]

To make that happen, an option would be to remove the dependency completely and make testify_yaml_fail or testify_yaml_custom the default behaviour.

Tehnically, testify_yaml_custom without a configured unmarshaller is not that functionally different from testify_yaml_fail, it would just need a default implementation which fails. This would do away with the build tags as well.

For anyone who needs the original behaviour, a code snippet could be provided in the documentation.

[brackendawson]

My first priority here as with all testify issues is to not break existing users or require them to do extra work. Deprecating the function without replacement or removing the implementation of the assertion will cause some people to have to do a non-trivial amount of work.

Continuing to use the archived yaml implementation requires some to do no work, but there will be people who can't use archived transient dependencies due to company policies.

Switching to goccy/go-yaml would require upgrading the minimum version of Go we require to 1.21. I don't think that would cause a lot of work because someone insisting on using the latest testify but also insisting on using an unsupported Go toolchain is quite a contradiction. I have had a play with #1725 and couldn't find a way the libraries behave differently. There may of course be ways and the fuzzer would be an excellent way to find any. I created a fuzz test here: https://github.com/brackendawson/testify/commit/f3aa8e6e4734de1efc468082f58ee24d610c38a5 but it fails immediately because of https://github.com/goccy/go-yaml/issues/718. So we can already say goccy/go-yaml isn't ready for us to use.

I think my preference would be to move to goccy/go-yaml and increase our minimum Go version. But only once the bug I found in goccy/go-yaml and any other blocking bugs are fixed. There's no rush to do this but a vulnerability in gopkg.in/yaml.v3 would change this.

[founderio]

An option would be to publish that package as a separate module. This incurs a bit of overhead when publishing versions, as this would be a separate go.mod within the same repository. During update, affected users will need to add another line to their go.mod when updating testify, but everyone who does not need the yaml dependency can just remove it/leave it out.

Not exactly clean, but it would incur minimum effort on user side. And the minimum Go version on the main module could remain the same.

[Victor-Elkins]

I've been reading through some of the documentation for this and I'm trying to figure out how to replace the yaml v3 in such a way that allows me to remove the dependency from my go mod all together. I've read through the documentation and the existing approaches but I'm struggling with understanding some of it. I may just have a knowledge gap with how build tags work, but it seems like even if I were to finish and succeed in this approach I would still have yaml v3 in my go mod. I don't actually require any of the functions that do use yaml v3 either so keeping it isn't really necessary I'm just trying to figure how I can manage to remove it and switch to something else. My main priority here is to just get the yaml.v3 off my go mod if that's at all possible, but I'm thinking it may not be.

[brackendawson]

I've been reading through some of the documentation for this and I'm trying to figure out how to replace the yaml v3 in such a way that allows me to remove the dependency from my go mod all together.

You can't, that's a limitation of the swappable yaml unmarshaller feature. You can prove that you don't use it, even if you do use YAMLEq, but you can't remove it from your go.mod file.

[brackendawson]

https://github.com/goccy/go-yaml/issues/718 was fixed very quickly and I re-ran my fuzz test. It quickly found that the string "�" (go: "\xff") is considered invalid yaml by gopkg.in/yaml.v3 but is considered valid yaml for that string value by github.com/goccy/go-yaml. I think this is valid yaml. This shows that there will be differences in behaviour if we switch the yaml provider, in this case it fixes a bug but there may be regressions too. I'm not going to run the fuzzer any further because it will just find more instances of the same bug in gopkg.in/yaml.v3.

[brackendawson]

Curiosity got the better of me, I wondered if there was a case that both considered to be valid yaml but had different interpretations and there is. "!!0" is considered to be "" by gopkg.in/yaml.v3 but nil by github.com/goccy/go-yaml. I don't know who's right here.

[xqqp]

If the change of the used yaml package is communicated clearly, for example in the notes of a new major release, I don't see an issue. I assume people would prefer to use maintained packages that behave slightly differently rather than outdated packages.

[ccoVeille]

Please note there is also https://github.com/yaml/go-yaml/ in preparation

It's a fork of gopkg.in/yaml made by the official YAML team

[jkroepke]

What about move assert.YAMLEq and assert.YAMLEqf to a separate module which can be optionally included, if requested.

I guess not many people using assert.YAMLEq

[rfay]

DDEV moved to https://github.com/yaml/go-yaml/ and it was easy, seems like it's in good shape now. (When I fixed up the go packages, I only had to keep the old yaml.v3 because stretchr/testify uses it :) )

[nikpivkin]

I guess not many people using assert.YAMLEq

I found the use of assert.YAMLEq in about 300 repositories https://sourcegraph.com/search?q=context:global+lang:go+%22assert.YAMLEq%28%22+-file:vendor+-repo:%5Egithub%5C.com/stretchr/testify%24++-repo:%5Egithub%5C.com/Antonboom/testifylint%24+&patternType=keyword&sm=0

[jkroepke]

I found the use of assert.YAMLEq in about 300 repositories

Compared to 25000 starts on GitHub, 300 is "not many", it's nothing.

Other point of view:

assert.Equal: 1.1M files https://github.com/search?q=path%3A*.go+language%3AGo+%22assert.Equal%22+-repo%3AAntonboom%2Ftestifylint++-repo%3Astretchr%2Ftestify+-path%3A%2F%5Evendor%5C%2F%2F+++NOT+is%3Aarchived&type=code&ref=advsearch

vs.

assert.YAMLeq: 812 files https://github.com/search?q=path%3A*.go+language%3AGo+%22assert.YAMLEq%22+-repo%3AAntonboom%2Ftestifylint++-repo%3Astretchr%2Ftestify+-path%3A%2F%5Evendor%5C%2F%2F+++NOT+is%3Aarchived&type=code&ref=advsearch

[dolmen]

Just a reminder: anyone can do this replacement on his own project that uses Testify:

go mod edit -replace=gopkg.in/yaml.v3=go.yaml.in/yaml/v3@latest
go mod tidy

However I would also like to remind everyone that since PR #1579 Testify also has facilities to replace the YAML package in downstream code by using the testify_yaml_custom build tag: see github.com/stretchr/testify/assert/yaml and https://github.com/stretchr/testify/blob/master/assert/yaml/yaml_custom.go

[IvanRadakovicDD]

I think the reason people are passionate about this topic comes down to 2 main issues with gopkg.in/yaml.v3:

1. It is Apache 2.0 licensed, not MIT (like this library is intending to be)
2. It is not maintained, and as other have shown above it has some bugs

testify_yaml_custom and go mod edit address the problem running different yaml code, but it does not help with having gopkg.in/yaml.v3 in your go.mod.

I think that upgrading to a library that meets those criteria outweighs the cons of bumping the minimum Go version for 2 reasons:

1. As others have said

... because someone insisting on using the latest testify but also insisting on using an unsupported Go toolchain is quite a contradiction.

2. At the time the go.mod was updated to Go 1.17, Go 1.17 was less than two years old. If we were to go with goccy/go-yaml, which requires Go 1.21, that version is now more than two years old. If a two year old Go version was acceptable then, it seems like it should be acceptable now. Moreover, moving to Go 1.21 seems like a great candidate for LTS because it contains a major overhaul to forward compatibility and toolchain management.

I don't have a strong preference on if it's yaml/go-yaml or goccy/go-yaml, but I think it would be great to migrate to a maintained and better licensed project.

[dolmen]

testify_yaml_custom and go mod edit address the problem running different yaml code, but it does not help with having gopkg.in/yaml.v3 in your go.mod.

go mod edit -replace definitely helps to get go.yaml.in/yaml.v3 in your go.mod. I understand that it isn't enough and you want to fully remove gopkg.in/yaml.v3 from your dependency chain. I understand this sentiment, but I would like to remind that the replace statement fixes the issue. If any dependency analysis still complains about gopkg.in/yaml.v3 dependency while replace is there, please direct your complaint to the maintainer of that analyzer.

I understand that people want that this project migrate to a newer YAML package. As a maintainer I would like this too. However I also have to balance with the risk of compatibility issue in deserialization in that alternate package. You must understand that gopkg.in/yaml.v3 doesn't strictly implement neither the YAML 1.1 or YAML 1.2 specifications. Other YAML packages have subtle differences. See for example yaml/go-yaml#144, yaml/go-yaml#145, yaml/go-yaml#146. So, being pedantic, switching the YAML parser must be considered as a breaking change. As a maintainer I have to balance introducing breaking changes vs an aging go.mod (with a provided workaround for users).