astropy
Wishlist item: mailing list for pre-release warnings
Since several data providers now have jupyter notebooks or other code depending on pyvo that we might inadvertently break (Bug #298 being my personal brown-bag example; but, really, given that we're dealing with rather complex contracts here, arbitrary variations of that are conceivable), I wonder if we shouldn't offer a mailing list that people who somehow depend on pyvo could subscribe to, the only purpose of which would be to say "we're going to have a new release in 14 days -- please see if we don't break any of your code", probably including quick instructions for how to get the new release running in a venv.
What do people think?
I'm not sure it would be of any great help. We would essentially ask users to test for us and I don't know who has the time/energy to do it. But we could announce it in the PyVO slack channel to give users a heads up.
IMO, these are gaps in regression testing. If we add tests for each of these bugs, we will continue to make it the package more robust to updates. The other thing we need to do is become more agile with releases. The process is not yet automated, but when we achieve that we could quickly address pressing issues but doing minor releases to stay on top of these things.
Just my opinion.
Hi Adrian,
On Wed, Feb 23, 2022 at 12:38:23PM -0800, Adrian wrote:
I'm not sure it would be of any great help. We would essentially ask users to test for us and I don't know who has the time/energy to do it. But we could announce it in the
PyVOslack channel to give users a heads up.
Well, I'm severely unhappy about slack, as it is the prototype of the "subvert a public infrastructure [IRC in this case] and then lock it out once you have sufficient lock-in of your users", but I suppose this would still work for most data centres.
So, if we'd document that and nobody protests they want open-web announcements I'd say go for it.
IMO, these are gaps in regression testing. If we add tests for each
That's true in some way. But the truth is that adopters use our standards in so many creative (in all the word's many meanings) ways that unit or regression tests would probably only work if we could get our adopters to contribute them as they go -- and that, again, would be suboptimal, as that would give us a lot of relatively fragile tests: They'd depend on external resources that do change regularly.
of these bugs, we will continue to make it the package more robust to updates. The other thing we need to do is become more agile with releases. The process is not yet automated, but when we achieve that we could quickly address pressing issues but doing minor releases to stay on top of these things.
Well, doing announced beta releases that people could use to make sure we didn't break anything would of course be an interesting alternative, but I suspect that'd be quite a bit more work over an "advance warning" scheme.
Hi Markus,
Just to try out the effectiveness of slack, I'm going to post this issue on the channel and invite people to comment on it...
Another way to test the PyVO prior to release is to educate users to include PyVO dev (GH main branch) as part of their nightly (periodic tests) such as GH Actions. It is similar with what we have now for astropy. We detect incompatible changes soon after they are merged into astropy main branch. astroquery would be the first candidate to do that when it runs the astropy dev tests.
Most Python applications end up with a stack of dependencies on 3rd-party packages, each of which is capable of upsetting the top-level application with a new release. This house of cards is difficult to manage for any application, with the two main approaches being locking the versions of dependencies and enabling automated tests against dev versions of some of the key dependencies.
Of course each package should endeavor to release minor versions that are fully backward compatible, but as mentioned above, it's impossible to protect against all the ways the package might be being used downstream.
I like the suggestion of a pre-release notice, as it gives a chance for dependent applications to check once without needing to set up an automated test suite. It still won't cover all issues, so mission-critical applications should probably lock the version of dependencies like pyvo, and control when to do their own testing prior to upgrading the version.
For the medium of communication, I would suggest astropy/#pyvo Slack and the IVOA interop mailing list. I'm less enthusiastic about setting up and maintaining a new mail list, but maybe there is something we could try under the auspices of astropy lists or OpenAstronomy.
Small suggestion: write more tutorials demonstrating common use cases, put them on astropy-learn, and use them as part of the test suite.
Also, pay attention to the astroquery test suite - astroquery uses pyvo under the hood a lot, and while it's not there to provide regression tests, it should expand the coverage w/no added effort.
User notebooks and downstream installable packages are two different beasts.
Easier case is the downstream installable package -- Said package devs can add pyvo dev into their CI and nightly tests, and promptly report upstream (or better yet, patch pyvo) when things break.
Harder case is user notebook in the wild. If the notebook is curated by your institution, you can maybe test them or jail your runtime environment in a Docker or something similar. But notebook that a random scientist write on their own, no one can catch that. We rely on people to report back when things break and then add such use case to our own CI over time.
Small suggestion: write more tutorials demonstrating common use cases, put them on astropy-learn, and use them as part of the test suite.
yeah, because learn is in any way robust, well tested and had good devops cycles to report upstream, like they even remotely value infrastructure and stability. :sarcasm:
I wholeheartedly agree with the above that testing with the dev version should be promoted. I also agree with notion above to extend the test suite or pyvo itself. One obvious place to start with would be the documentation itself to have full examples that are guaranteed to be working correctly and enable doctesting to ensure they stay that way.
Since several data providers now have jupyter notebooks or other code depending on pyvo that we might inadvertently break
Would it be possible to keep a tally of these resources in the ivoa capacity? Maintaining large body of code examples and notebooks I'm absolutely certain it's not just pyvo that breaks these notebooks, but other packages in the stack, too. So maybe some share ivoa resource is the answer? Or at least that's sounds to be the obvious place for a mailing list, posting heads up about new releases and breakages.
(And of course the classical approach is to do release candidates before doing API breaking releases, but that would unlikely be any help in recovering any issues with notebooks and user code in a timely fashion)
On Fri, Feb 25, 2022 at 09:26:54AM -0800, Brigitta Sipőcz wrote:
Would it be possible to keep a tally of these resources in the ivoa capacity? Maintaining large body of code examples and notebooks I'm
Totally. As it happens, there already is a Registry extension designed to do that, http://ivoa.net/documents/Notes/EDU/. And there is a UI that pulls these entries out of the Registry and formats them more or less nicely, https://dc.g-vo.org/VOTT. There's no notebooks in there so far, just PDFs and a few HTML pages, but this should work for notebooks, too.
However, it's a tedious uphill battle to get people to register their material, and so I'm a bit reluctant to hope that that's an appropriate answer. On the other hand, getting people to register their notebooks would have quite a few ancillary benefits.
So: if someone reading this has a notebook they'd like to see checked pre-release but isn't sure how to register it: Please speak up (or contact me privately) so we can see how realistic this plan is.
The navo workshop notebooks are one of those, not sure whether they are in that registry. Also, these are not exactly release critical as they are checked against package versions before doing a new workshop, and are updated when neccessary. I suppose there are a large number of notebooks at STScI, too, Tom or Pey Lian could comment on how feasible it is to test those for dependencies.
Anyway, announcement-wise trying out a low traffic mailing list would not hurt. I agree with the above that it's unlikely that people have the capacity to test, but at least they are being warned and won't need to start their investigating from zero.
OK, then, focusing on the easiest approaches, that are self-contained and that we can do something about, I opened #306 for doing narrative doctesting. And I would expect #307 would be useful, too, to clear out warnings from the test runs.
On Fri, Feb 25, 2022 at 11:52:58AM -0800, Brigitta Sipőcz wrote:
The navo workshop notebooks are one of those, not sure whether they are in that registry. Also, these are not exactly release critical
They're not, but I'd be happy to help their maintainer to get them in.
as they are checked against package versions before doing a new workshop, and are updated when neccessary.
It'd still be great if we could somehow use them as regression tests (which, I think, would be rather feasible when they're registered and perhaps very lightly marked up such that a machine can figure out whether their execution failed): Whatever is in such notebooks will be copy-and-pasted into notebooks of astronomers, and not breaking these (unintentionally) I'd argue is a worthy goal.
Do we need to keep this open longer, what are the action items that we could do?