Rez test improvements (dev mode, interactive flag)

[fnaum]

Some rez test improvements

• added the ability to run rez test in "developer mode" using rez test . from the root folder of the package.
• Automatically build the package if the package is not found in the REZ_LOCAL_PACKAGES_PATH
• Update the built package.py if the tests section changed
• added --interactive option, to resolve an interactive environment to run the tests. ie. (rez test --interactive TESTNAME). In case the package has multiple variants, you can also specify the variant using --variant
• By default change to a build/rez_test_result directory to run the test and store the test artifacts (this way we can pick those results from tools like sonar, that expect the nose and coverage reports in a known place). If they need to be run on the root of the packages run_in_root can be set in the tests field of the package.py.
• added --clean to clean the test artifacts created from a previous test run.
• Tests added along with the mock module under vendor

related to https://github.com/nerdvegas/rez/issues/665

[nerdvegas]

Update: Hoping to be working on this soon, we need the same functionality at Method. Stay tuned.

[nerdvegas]

So I'm freed up to work on this next week for Method. I think it would make sense to use this PR and https://github.com/nerdvegas/rez/pull/758 as a basis for that work, rather than updating these PRs though.

Reasons are:

• I need to make immediate use of the time I'm getting from Method (rez dev time doesn't happen often!)
• We need to make these changes with https://github.com/nerdvegas/rez/issues/665 in mind, so there will probably be some significant differences to these PRs.

Fede I wanted to let you know in case you thought I'm throwing this work away! Rest assured a chunk of the work is already done here and I'll be incorporating it into the upcoming PRs. Expect to see more next week.

[nerdvegas]

Note that some of this PR has been superseded by https://github.com/nerdvegas/rez/pull/807. However, I'm still open to adding the dev mode functionality shown here, as long as there's consensus. It's not clear to me how this should work exactly - as mentioned above, the way the package is automatically re-installed if the tests definition changes, feels inconsistent.

Wrt --interactive, it definitely makes sense to do this, and I'm aiming to make this available in a separate PR which mimics the behaviour in this PR.

[JeanChristopheMorinPerso]

@nerdvegas @fnaum The way I see the dev mode is just to avoid some typing. So instead of doing rez-build -ci && rez-test packageA==x.x.x, you could just do rez-build -ci && rez-test .. It would make things simpler to use IMO. So it's just a "shorthand".

[nerdvegas]

Yeah I don't mind that functionality, but in https://github.com/nerdvegas/rez/pull/759 'dev mode' is not quite that simple, as it does things like automatically re-install the package if its tests change. I am not so sure about that, because it feels a bit arbitrary as to whether a package is reinstalled (ie yes if tests change, no if anything else changes).

In #759, test byproducts are also written to a specific directory when in dev mode. So the question is also whether this should happen and if so, what that path should be.

A

On Wed, Dec 4, 2019 at 2:58 PM Jean-Christophe Morin < [email protected]> wrote:

@nerdvegas https://github.com/nerdvegas @fnaum https://github.com/fnaum The way I see the dev mode is just to avoid some typing. So instead of doing rez-build -ci && rez-test packageA==x.x.x, you could just do rez-build -ci && rez-test .. It would make things simpler to use IMO. So it's just a "shorthand".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/pull/759?email_source=notifications&email_token=AAMOUSVZUO6SGH3FUDLYKZ3QW4TIHA5CNFSM4I6XOR42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF3UWJA#issuecomment-561466148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSSJGRR7JWFQNGQ3RE3QW4TIHANCNFSM4I6XOR4Q .

[JeanChristopheMorinPerso]

in #759 'dev mode' is not quite that simple, as it does things like automatically re-install the package if its tests change.

Yeah, I wouldn't go with the auto-install. Hence why I was giving rez-build -ci && rez-test . as an example. I'd almost prefer if the user was manually building (as he probably knows more what he needs to do for his tests to run then us).

In #759, test byproducts are also written to a specific directory when in dev mode. So the question is also whether this should happen and if so, what that path should be.

I still haven't made my mind on this. To be frank, the first time I reviewed this PR I didn't like the idea that rez would know where my tests artifacts are. And as of today I have the same feeling. You could have 5 different test target that each run different tools that will all generate tests artifacts in a different way (and sometimes if would even be impossible to choose where to put them). That means each test would have to define it's artifact directory. Though a possible case, it might not be a super common one. It would be easy to implement though. I think the hardest part will be to agree on how to handle that...

[nerdvegas]

Marking as ON HOLD - these features are desired (interactive mode, dev mode), however the code has changed too much for this PR to be viable. Keeping for reference until an equivalent is merged.

[fnaum]

Sorry, I was not active when you reviewed this, we lost the only resource that was working with rez and all the other tools and we have not found a replacement yet. :( Feel free to close it at any time