bug: list @stage don't show new files

[bohutang]

Summary

Databend with local fs.

How to re-produce:

mysql> create stage xx;
 -- copy logs.parquet file to _data/stage/xx
mysql> list @xx;
+--------------+-------+------+-------------------------------+---------+
| name         | size  | md5  | last_modified                 | creator |
+--------------+-------+------+-------------------------------+---------+
| logs.parquet | 26592 | NULL | 2022-08-03 09:23:28.000 +0000 | NULL    |
+--------------+-------+------+-------------------------------+---------+

-- copy new logs.csv to _data/stage/xx

mysql> list @xx;
+--------------+-------+------+-------------------------------+---------+
| name         | size  | md5  | last_modified                 | creator |
+--------------+-------+------+-------------------------------+---------+
| logs.parquet | 26592 | NULL | 2022-08-03 09:23:28.000 +0000 | NULL    |
+--------------+-------+------+-------------------------------+---------+

-- Not show the new file logs.csv here

[Xuanwo]

copy new logs.csv to _data/stage/xx is a copy executed outside databend?

[Xuanwo]

This issue drives me to think about why we need to store internal stages' file info inside metasrv.

Cc @BohuTANG @flaneur2020 @junnplus for more information.

[bohutang]

copy new logs.csv to _data/stage/xx is a copy executed outside databend?

Yes.

[junnplus]

This issue drives me to think about why we need to store internal stages' file info inside metasrv.

We can read the stage directly through management mode, otherwise, the user needs to start the warehouse to query the stage.

[junnplus]

I guess it's a design issue, only data written via databend will be stored in metasrv, but if metasrv does not exist data will be read from dal.

You can use CALL SYSTEM$SYNC_STAGE_FILE(<stage_name>) to sync to metasrv.

[Xuanwo]

Has been fixed.