Consistent shape and types

[nwittler]

Describe the missing feature

Shapes and types of tensorflow variables needs to be consistent and correct across the whole codebase.

Describe the solution you'd like

The current workaround are casts and reshapes to fix inconsistencies. This is performance poison and needs to be corrected.

Describe alternatives you've considered

Die the slow death of garbage performance.