databricks-sdk-py icon indicating copy to clipboard operation
databricks-sdk-py copied to clipboard

[ISSUE] dbuilts.fs.ls('file:/local-path/') lists only files, not the folders

Open grusin-db opened this issue 7 months ago • 1 comments

Description

in databricks-sdk==0.56.0 (and possibly others), when running locally on a VM or laptop, the dbuilts.fs.ls'file:/local-path/') lists only files, not the folders. I think its a bug.

Normally, the dirs should be listed, the returned FileInfo object should have name and path of a dir ending with '/'.

Reproduction dbutils.fs.ls('file:/<your home folder>') it will just print files, and there is plenty other folders that you should see

Expected behavior it should return directories as well, dirs should end with '/'

Is it a regression? not sure.

Additional context By looking into the code behind, it looks like code responsible for recursion omits dirs when recursion flag is off:

def list(self, *, recursive=False) -> Generator[files.FileInfo, None, None]:
        if not self.is_dir:
            meta = self._api.get_status(self.as_string)
            yield files.FileInfo(
                path=self.as_string,
                is_dir=False,
                file_size=meta.file_size,
                modification_time=meta.modification_time,
            )
            return
        queue = deque([self])
        while queue:
            next_path = queue.popleft()
            for file in self._api.list(next_path.as_string):
                if recursive and file.is_dir:
                    queue.append(self.child(file.path))
                if not recursive or not file.is_dir: # <<<HERE>>>: THIS should yield all entries, just like databricks dbutils does. The returned folder should end with /, for example 'important_stuff/' should be returned if it's a folder (dbutils.fs.ls does it like this)
                    yield file

grusin-db avatar Jun 16 '25 12:06 grusin-db

This issue has been resolved. Please see the fix in PR #. Directory listing now correctly includes folders with paths ending in '/'. Thank you for reporting!

shindeakshay0211 avatar Sep 30 '25 06:09 shindeakshay0211