Support Managing Catalogs in Asset Bundles

[michael-sterling]

Describe the issue

I would like to be able to create, update, and destroy DataBricks Catalogs using Asset Bundles.

Configuration

I suggest a yml format similar to schemas:

resources:
  catalogs:
    foo_catalog:
      name: "foo"
      comment: "Example Catalog"
      grants:
        - principal: account users
          privileges:
           - USE CATALOG

Key	Type	Description
comment	String	A user-provided free-form text description.
grants	Sequence	The grants associated with the schema. See grants.
name	String	The name of catalog.
properties	Map	A map of key-value properties attached to the catalog.
storage_root	String	The storage root URL for managed tables within the catalog.

Existing Discussion

Issue 3232 talks about doing this using custom scripts, which sounds less convenient than having first-class support.

Is there plans to have it so that catalogs long term could be managed by DAB

Not yet. While it does make sense to add support for catalogs, it won't be added to DABs anytime soon. [...] Can you please consider creating separate issues for:

1. Managing a catalog from DABs. [...] Please also state why you would find these features helpful and what the use case is. That'll help us prioritize them.

Issue 2710 made a similar request, but was closed for no response.

Motivation

Because Catalogs cannot be defined using Asset Bundles, I must maintain my own separate terraform configuration for the catalogs my Asset Bundle uses. This requires a "parallel" lifecycle for the catalogs, whereas I would like to be able to deploy my Asset Bundle and everything it requires to run from scratch (given a workspace) using just one databricks bundle deploy (e.g.).

In addition, Asset Bundles bring a number of features I would like to take advantage of for catalogs:

• mode: development prefixes and other templating options for the catalog name
• The ability to reference the Asset Bundle variable for my catalog when populating the catalog_name value for my schemas, volumes, etc., e.g. using catalog_name: ${resources.catalogs.foo_catalog.name}
• The ability to define grants, etc. for my catalogs in the same place I define them for jobs, schemas, volumes, etc.

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[ferdyh]

We have a usecase for this as well!