posit-dev
Use preview builds for testing
These failures stem from this change in Connect: https://github.com/posit-dev/connect/commit/88ce4ff4d46ae1c4ba831e99b0136d86c0718cf3#diff-db146a204844f1dd26bd7bcafabfbd64d399001dd8bad4e7174611c628a8bfbe
It removed the Justfile commands that this CI uses (just test-rsconnect-python-repo).
The original intention was to ensure that changes to rsconnect-python would not break Connect's CI, which does the inverse operation. It pulls reconnect-python and uses it for its own test matrix. I think a better approach would be what we did with posit-sdk-py and use Docker to run against the preview version of Connect.
Ugh. Sorry. Those targets lacked breadcrumbs linking their use to this repository. I thought they were obsolete targets orphaned by some earlier cleanup.
Feel free to temporarily add them back, if that would help.
Agreed that it is better to use preview builds.
I did some archaeology (git blame) to try to understand what's happening in these integration test suites. Sources are #331, #384, and #526. test-rsconnect does appear to use the public preview docker image for testing something with vetiver and then another thing added later. test-connect has its own docker/compose setup and is the one that builds Connect from the checkout and runs the rsconnect-python tests contained in it.
It would be interesting to try dropping the test-connect setup and run the test suite it runs using the test-rsconnect setup. That would be a step in the right direction perhaps, and would solve this issue literally (it would make all integration tests use the preview build).
Looking at this in the context of https://github.com/posit-dev/with-connect/issues/27, my proposal is this:
- Drop the
test-connectjob that pulls in the Connect dev source. In its place, add a workflow_dispatch arg to the Connect job that tests rsconnect-python nightly to be able to specify a branch/SHA of rsconnect-python. If you need to test against both dev Connect and dev rsconnect-python, trigger it there (on your Connect PR, even). - Move vetiver integration tests to
vetiver-python, like howpins-pythonhas its integration tests. Vetiver uses pins to push models. rsconnect-python isn't doing special for vetiver, AFAICT. We could also make sure those integration tests are pulling rsconnect-python frommain, if we wanted a way to ensure that something didn't break. (Thoughts @isabelizimm?) - https://github.com/posit-dev/rsconnect-python/blob/main/tests/test_main_system_caches.py needs to be rewritten if we're going to use
with-connectand not bespokedocker composehere. My hunch is that those tests could just assert that they're making the right HTTP requests, and rely on Connect's testing to confirm that if Connect receives such a request, it does the right thing with caches. As in, they don't need to be integration tests at all. Reaching into the container's file system here seems like we're crossing a boundary we shouldn't. (Thoughts @amol-?) - We could add some very basic e2e tests that confirm we can upload bundles and create content, enough to cover the basic API surface, so that we can freely refactor internals. That could be done across a matrix of Connect versions using
with-connect, which is coverage that is completely missing today.
How good do we feel about going through with dropping the test-connect jobs? They're failing again in rsconnect-python, which is blocking me on a bug fix, but i just ran the rsconnect-python-tests-at-night against my branch and they passed: https://github.com/posit-dev/connect/actions/runs/19311989599
How good do we feel about going through with dropping the
test-connectjobs? They're failing again inrsconnect-python, which is blocking me on a bug fix, but i just ran thersconnect-python-tests-at-nightagainst my branch and they passed: https://github.com/posit-dev/connect/actions/runs/19311989599
Can you make a PR that does that? We can discuss further on that (if needed).