databricks cli cannot access '/dbfs/databricks-datasets/'
In the notebook, I can use
%fs ls dbfs:/
or
%sh ls /dbfs/
to see the directory: databricks-datasets
However in cli, when I type databricks fs ls or dbfs ls, I cannot see the above directory. I also tried dbfs ls dbfs:/databricks-datasets/, it prompted an error message:
Error: b'{"error_code":"RESOURCE_DOES_NOT_EXIST","message":"No file or directory exists on path dbfs:/databricks-datasets/."}'
Is is due to some restrictions of CLI or did I get something wrong during the configuration?
FYI: cluster mode: standard pool: None databricks runtime version: 5.5LTS python version:3 worker type: m4.large driver type: m4.large
Is there any fix for this?
@bogdanghita-db is this a misconfigured cross-account policy or something else?