mxnet [FEATURE] Add feature of attach_grad to nonleaf variables in HybridizedBlock.

Description

This is the same as this https://github.com/apache/incubator-mxnet/pull/20559. The PR adds the support for fetching the gradients of intermediate variables in a gluon HybridizedBlock. This applies uniformly to both when block.hybridize() is on and off. This generates the attach_grad implemented in implemented in PR#20500.

The motivation of this feature comes from this issue#11865.

Checklist

Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented

Changes

[x] block.py where mark_vars and get_mark_vars are added along with MXNDArrayMarkDCVariables.
[x] cached_op.invoke in cpp backend and CachedOp.__call__ have been editted to include the pass of marked nonleaf ndarrays.
[x] set_nleafs method is added into CachedOp class to store the marked nonleaf ndarrays.
[x] Inside void RunGraph, marked nonleaf ndarrays are linked to the marked computational node for autograd computation.

Comments

This feature is built on top of PR#20500. The modification here is mainly in the invoke of CachedOp computation.

Jul 08 '22 16:07 KexinFeng

Hey @KexinFeng , Thanks for submitting the PR All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [website, clang, miscellaneous, sanity, windows-cpu, edge, windows-gpu, centos-cpu, unix-cpu, unix-gpu, centos-gpu]

Note: Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. All CI tests must pass before the PR can be merged.

Jul 08 '22 16:07 mxnet-bot

@mxnet-bot run ci [unix-cpu, unix-gpu, windows-gpu]

Jul 08 '22 17:07 KexinFeng

Jenkins CI successfully triggered : [windows-gpu, unix-cpu, unix-gpu]

Jul 08 '22 17:07 mxnet-bot

@mxnet-bot run ci [centos-gpu, windows-gpu]

Jul 09 '22 06:07 KexinFeng

Jenkins CI successfully triggered : [centos-gpu, windows-gpu]

Jul 09 '22 06:07 mxnet-bot

@mxnet-bot run ci [centos-gpu, windows-gpu]

Jul 09 '22 15:07 KexinFeng

Jenkins CI successfully triggered : [windows-gpu, centos-gpu]

Jul 09 '22 15:07 mxnet-bot

@mxnet-bot run ci [unix-gpu]

Jul 09 '22 15:07 KexinFeng

Jenkins CI successfully triggered : [unix-gpu]

Jul 09 '22 15:07 mxnet-bot

@mxnet-bot run ci [macosx-x86_64]

Jul 09 '22 15:07 KexinFeng

Undefined action detected. Permissible actions are : run ci [all], run ci [job1, job2] Example : @mxnet-bot run ci [all] Example : @mxnet-bot run ci [centos-cpu, clang]

Jul 09 '22 15:07 mxnet-bot

None of the jobs entered are supported. Jobs entered by user: [macosx-x86_64] CI supported Jobs: [unix-cpu, website, sanity, unix-gpu, clang, centos-gpu, windows-cpu, windows-gpu, miscellaneous, edge, centos-cpu]

Jul 09 '22 15:07 mxnet-bot

@mxnet-bot run ci [centos-gpu ]

Jul 09 '22 23:07 KexinFeng

Jenkins CI successfully triggered : [centos-gpu]

Jul 09 '22 23:07 mxnet-bot

Jenkins CI successfully triggered : [centos-gpu]

Jul 10 '22 15:07 mxnet-bot

@mxnet-bot run ci [centos-gpu ]

Jul 10 '22 15:07 KexinFeng

Jenkins CI successfully triggered : [centos-gpu]

Jul 10 '22 15:07 mxnet-bot

Jenkins CI successfully triggered : [centos-gpu]

Jul 11 '22 07:07 mxnet-bot

@mxnet-bot run ci [macosx-x86_64]

Jul 11 '22 16:07 KexinFeng

None of the jobs entered are supported. Jobs entered by user: [macosx-x86_64] CI supported Jobs: [sanity, centos-cpu, clang, unix-cpu, windows-cpu, windows-gpu, miscellaneous, centos-gpu, unix-gpu, edge, website]

Jul 11 '22 16:07 mxnet-bot

@mxnet-bot run ci [macosx-x86_64]

Jul 11 '22 16:07 KexinFeng

None of the jobs entered are supported. Jobs entered by user: [macosx-x86_64] CI supported Jobs: [sanity, centos-cpu, clang, unix-cpu, windows-cpu, windows-gpu, miscellaneous, centos-gpu, unix-gpu, edge, website]

Jul 11 '22 16:07 mxnet-bot

@mxnet-bot run ci [edge, unix-gpu]

Sep 22 '22 19:09 KexinFeng

Jenkins CI successfully triggered : [unix-gpu, edge]

Sep 22 '22 19:09 mxnet-bot

@mxnet-bot run ci [unix-gpu]

Oct 08 '22 20:10 KexinFeng

Jenkins CI successfully triggered : [unix-gpu]

Oct 08 '22 20:10 mxnet-bot

@mxnet-bot run ci [unix-gpu]

Oct 11 '22 22:10 KexinFeng

Jenkins CI successfully triggered : [unix-gpu]

Oct 11 '22 22:10 mxnet-bot