Add ability to use custom nodes as layer or mask

[ucupumar]

Using custom nodes should be possible with these rules:

• Should be in the form of a node group
• At least has a single output
• Vector input and alpha output can be optional
• Node outputs can be assigned to layer channel inputs
• Shader/BSDF socket type won't be supported