Project-MONAI
[Early WIP] - Lazy resampling
Lazy Resampling - deep refactor (see #4855)
The goal is to provide less memory-intensive, faster and higher quality spatial resampling by adopting the best practice of traditional graphics pipelines.
This change impacts on most of the spatial transform code, and so is also an opportunity to refactor said code to detangle what have become quite complex spatial transforms. While #4911 demonstrates that #4855 is possible, this PR is trying to find a way to decouple elements of this solution from one another and dramatically simplify the task of implementing new transforms in future and by third parties.
Transform refactoring
Due to the complexity of existing transforms and the need to avoid adding more complexity, this refactor looks at the transforms from first principles, attempting to decouple the definition of the transform from its application. This can be seen in the three layers of transform implementation, functional, array and dictionary:
Functional
Functional transforms now have a single responsibility, to generate the definition of the transform.
Array
Array transforms now wrap the Functional transforms, and then decide whether to push the transform to a pending queue or apply it immediately. They do this by descending from LazyTransform, which has a flag lazy_evaluation. This could be taken further; the pattern of whether to push to pending or apply immediately could be part of LazyTransform functionality.
Dictionary
Dictionary transforms wrap Array transforms, applying them once for each key. There is potential here to further reduce the code by having a standard 'apply to each specified key' function that can be called by subclasses.
The amount of code is dramatically reduced as a result, at least in the transforms reimplemented so far.
Compose compilers
As part of looking at this refactor, it is becoming obvious that there are a few places where the list of transforms to be executed are manipulated as they are being executed. Each place in the code that does this does so with the assumption that no-one else is doing it. As such, one compose manipulator will "win" and the others do not appear to be applied at all.
One way around this is to specify "compose compilers". These are functions that will take an input list of transforms and compile it to a new list of transforms. The benefit of compose compilers is twofold:
- They reduce the number of places where handling of the compose is customised
- They can be chained in order to meet the demands of multiple transform list customisations (lazy resampling and cached datasets, for example)
Types of changes
- [x] Non-breaking change (fix or new feature that would not break existing functionality).
- [x] Breaking change (fix or new feature that would cause existing functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running
./runtests.sh -f -u --net --coverage. - [ ] Quick tests passed locally by running
./runtests.sh --quick --unittests --disttests. - [ ] In-line docstrings updated.
- [ ] Documentation updated, tested
make htmlcommand in thedocs/folder.
TODO:
Spatial transform implementation
See Table: Implementation by Spatial Transform for detailed progress:
- [ ] Implement functional versions of all spatial transforms
- [ ] Implement array versions of all spatial transforms
- [ ] Implement dictionary versions of all spatial transforms
LazyTransform and RandomTransform interfaces for 3rd party use
Interfaces that indicate capability without needing to adopt implementations:
- [ ] Implement LazyTransform interface
- [ ] Optionally reimplement RandomTransform interface for 3rd party use
Compose compilation
This is done if we go with compose compilers as a unified mechanism for lazy resampling, smart datasets, etc.:
- [ ] Compose compiler implementation
- [ ] Compose accepts compose compilers
- Compose compiler transforms
- [ ] Cached / Persistent dataset transforms
- [ ] Multi-sample transforms
MetaTensor updates
Optionally make MetaTensors able to share read only access to underlying Tensor for multi-sampling transforms:
- [ ] MetaTensor sharing underlying tensor
Lazy resampling compose
This is done if we go with augmenting compose and leaving smart datasets alone:
- [ ] Compose performs lazy resampling
Table: Implementation by spatial transform
# R
# a
# n
# S R d
# p e R R G G
# a s a a R R r r R
# t a R R n n a a i i a
# i m O a a d d n n d d n
# a p r n R n A A D R d d D D d
# l l i d a d f f e a 2 3 o i G G G T
# R e e R R n R A R f f f R n D D s s r r r r
# e T S n o o d a x a i i o e d E E t t i i i a
# s o p t R R t t R n i n n n r s A A l l o o d d d n
# a M a a e o a a o d s d e e m a f f a a r r S P S s
# m a c t F s t Z t t t F F Z G G G m f f s s t t p a p l
# p t i i l i a o e e a l l o r r r p i i t t i i l t l a
# l c n o i z t o 9 9 t i i o i i i l n n i i o o i c i t
# e h g n p e e m 0 0 e p p m d d d e e e c c n n t h t e
Functional X . X X X X X n n n n n n n n n . n n n . n . . n X
Array forward X . X X X X X X X X . . . . . n . . . . . . . . . X
Dictionary forward X . . X X X . . . . . . . . . n . . . . . . . . . X
Test forward f . . f f f . . . . . . . . . . . . . . . . . . . .
Array inverse . . . . . . . . . . . . . . . . . . . . . . . . . .
Dictionary inverse . . . . . . . . . . . . . . . . . . . . . . . . . .
Test inverse . . . . . . . . . . . . . . . . . . . . . . . . . .
Table: Implementation by Crop / Pad transform
# R R
# a R a
# n a n
# d n d R
# S d C e
# C p C r s
# e a R r o i
# n C R t a o p z
# t e a i C n p B e
# e n n R a r d B y W
# D r t d a l o W y L i B
# i S S e S n C p e P a t o
# S v p p r p d r F i o b h u
# p B i a a S a S o o g s e P n
# a o s t t c t c p r h N l a d
# t r i i i a i a S e t e C d i C
# i d b a a l a l a g e g l O n r
# a e l l l e l e m r d L a r g o
# l r e C C C C C C p o C a s C R p
# P P P P r r r r r r l u r b s r e P
# a a a a o o o o o o e n o e e o c a
# d d d d p p p p p p s d p l s p t d
Functional . . . . . . . . n n n . n n n . . X
Array forward . . . . . . . . . . . . . . . . . X
Dictionary forward . . . . . . . . . . . . . . . . . X
Test forward . . . . . . . . . . . . . . . . . .
Array inverse . . . . . . . . . . . . . . . . . .
Dictionary inverse . . . . . . . . . . . . . . . . . .
Test inverse . . . . . . . . . . . . . . . . . .
