azure.synapse.tools icon indicating copy to clipboard operation
azure.synapse.tools copied to clipboard

Unsupported Type: SparkConfigurations

Open jreitz43 opened this issue 1 year ago • 0 comments

Deployment throws error "ASWT0029: Unknown object type: SparkConfiguration" when deploying a notebook (example shown below) that has a reference to a custom spark configuration. I believe this may be a very similar issue to #11 where SparkConfiguration would need to be added to the allowed valid types in private/!SynapseObject.class.ps1 as well as private/Get-SynapseObjectByName.ps1 so that it can pretend that the object referenced is valid and exists. Notebook:

{
	"name": "nb_example",
	"properties": {
		"folder": {
			"name": "utility"
		},
		"nbformat": 4,
		"nbformat_minor": 2,
		"bigDataPool": {
			"referenceName": "sparkSm",
			"type": "BigDataPoolReference"
		},
		"targetSparkConfiguration": {
			"referenceName": "CustomSparkConfig",
			"type": "SparkConfigurationReference"
		},

Failure:

**mVERBOSE: A***lyzing notebook dependencies...
**mVERBOSE: Folder: D:\a\***\drop\***\notebook
**mVERBOSE: - nb_example.json
Failure occurred while publishing artifacts to ***
##[debug]Error record:
##[debug]Exception: D:\a\_temp\a560ab36-63de-480f-888a-bc877***f23f0a.ps***:38
##[debug]     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
##[debug]     | ASWT0029: Unknown object type: SparkConfiguration.

azure.synapse.tools Version 0.23.000

jreitz43 avatar Apr 03 '24 17:04 jreitz43