dbutils.notebook.exit("anytext") throws an exception

[brian-pickens]

Description

Admittedly, I'm pretty new to databricks so go easy on me if I get anything wrong here. We are using vscode, and databricks-connect to develop a project with notebooks that will be executed via synapse. We are using dbutils.notebook.exit(intended_result) to return data back to the synapse pipeline.

The issue is that when executing locally in vscode, passing any string value to exit() results in an exception. This is not an issue when executing in databricks.

Reproduction

Create any jupyter notebook with a single cell with the following line:

dbutils.notebook.exit("anytext")

Results

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Cell In[13], [line 1](vscode-notebook-cell:?execution_count=13&line=1)
----> [1](vscode-notebook-cell:?execution_count=13&line=1) dbutils.notebook.exit("anytext")

File c:\my_project\.conda\Lib\site-packages\databricks\sdk\dbutils.py:416, in _ProxyCall.__call__(self, *args, **kwargs)
    [414](file:///C:/my_project/.conda/Lib/site-packages/databricks/sdk/dbutils.py:414)     self._raise_if_failed(result.results)
    [415](file:///C:/my_project/.conda/Lib/site-packages/databricks/sdk/dbutils.py:415)     raw = result.results.data
--> [416](file:///C:/my_project/.conda/Lib/site-packages/databricks/sdk/dbutils.py:416)     return json.loads(raw)
    [417](file:///C:/my_project/.conda/Lib/site-packages/databricks/sdk/dbutils.py:417) else:
    [418](file:///C:/my_project/.conda/Lib/site-packages/databricks/sdk/dbutils.py:418)     raise Exception(result.results.summary)

File c:\my_project\.conda\Lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    [341](file:///C:/my_project/.conda/Lib/json/__init__.py:341)     s = s.decode(detect_encoding(s), 'surrogatepass')
    [343](file:///C:/my_project/.conda/Lib/json/__init__.py:343) if (cls is None and object_hook is None and
    [344](file:///C:/my_project/.conda/Lib/json/__init__.py:344)         parse_int is None and parse_float is None and
    [345](file:///C:/my_project/.conda/Lib/json/__init__.py:345)         parse_constant is None and object_pairs_hook is None and not kw):
--> [346](file:///C:/my_project/.conda/Lib/json/__init__.py:346)     return _default_decoder.decode(s)
    [347](file:///C:/my_project/.conda/Lib/json/__init__.py:347) if cls is None:
    [348](file:///C:/my_project/.conda/Lib/json/__init__.py:348)     cls = JSONDecoder

File c:\my_project\.conda\Lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    [332](file:///C:/my_project/.conda/Lib/json/decoder.py:332) def decode(self, s, _w=WHITESPACE.match):
    [333](file:///C:/my_project/.conda/Lib/json/decoder.py:333)     """Return the Python representation of ``s`` (a ``str`` instance
    [334](file:///C:/my_project/.conda/Lib/json/decoder.py:334)     containing a JSON document).
...
    [354](file:///C:/my_project/.conda/Lib/json/decoder.py:354) except StopIteration as err:
--> [355](file:///C:/my_project/.conda/Lib/json/decoder.py:355)     raise JSONDecodeError("Expecting value", s, err.value) from None
    [356](file:///C:/my_project/.conda/Lib/json/decoder.py:356) return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Expected behavior I expect this to not throw an exception, and display the exit result like databricks does.

Is it a regression? I have not tried any previous versions

Debug Logs The SDK logs helpful debugging information when debug logging is enabled. Set the log level to debug by adding logging.basicConfig(level=logging.DEBUG) to your program, and include the logs here.

Other Information OS: Windows 10 VS Code: 1.96.0 Databricks plugin: v2.4.8

pip show databricks-sdk Name: databricks-sdk Version: 0.38.0 Summary: Databricks SDK for Python (Beta) Home-page: https://databricks-sdk-py.readthedocs.io Author: Serge Smertin Author-email: [email protected] License: Location: c:\my-project.conda\Lib\site-packages Requires: google-auth, requests Required-by: databricks-connect

Additional context Add any other context about the problem here.

[brian-pickens]

Having dug into the sdk code, it appears that the sdk implementation requires a json string be passed to the exit() method. And actually calls dbutils.notebook.exit() in the code passed to the databricks cluster. So you end up with double exit calls. Not sure what the effect of that is.

    def __call__(self, *args, **kwargs):
        raw = json.dumps((args, kwargs))
        code = f'''
        import json
        (args, kwargs) = json.loads('{raw}')
        result = dbutils.{self._util}.{self._method}(*args, **kwargs)
        dbutils.notebook.exit(json.dumps(result))
        '''
        ctx = self._context_factory()
        result = self._commands.execute(cluster_id=self._cluster_id,
                                        language=compute.Language.PYTHON,
                                        context_id=ctx.id,
                                        command=code).result()
        if result.status == compute.CommandStatus.FINISHED:
            self._raise_if_failed(result.results)
            raw = result.results.data
--->        return json.loads(raw)
        else:
            raise Exception(result.results.summary)