feat: Migrate filebrowser into storage proxy
Migration to Mono Repository. Moves file browsers to the storage proxy. This will greatly reduce the burden of manager and webserver to mediate large file transfers.
- [x] Implement filebrowser container mgmt (with a configurable maximum number of containers)
- [x] Add new manager-facing storage-proxy APIs to create/destroy filebrowser sessions
- [x] manager: Add new vfolder APIs to map the new manager-facing storage-proxy APIs
- [x] client-py: Add new functional wrappers and
vfoldersubcommands to use the filebrowser session APIs - [x] webui: Update the current filebrowser UIs to use the new filebrowser session mgmt APIs
- Implemented main feature such as the communication between Client, Manager and Storage Proxy to support FileBrowser session where user can attach multiple folders and copy/move data using UI.
- Implemented the CLI commands on client side to start filebrowser with attached folders.
- Implemented Manager API interface to redirect requests from Client to Storage Proxy and receive the response from Storage Proxy about the url, port number and container id.
- The user using CLI can destroy the filebrowser session using the container id information printed out in cli.
While implementing the filebrowser, I decoupled filebrowser from Vfolder as a separate app. This will allow having filebrowser its own routing management for more complex commands. Decoupling will help to improve maintenance and advanced feature implementation of Filebrowser without affecting Vfolder.
Previously, did the implementation on the server of preventing multiple monitors run with common.FileLock.
Implemented UID and GID settings in a startup.sh script after the containers are set. The input variables are defined in the .toml settings file.
Refactoring, in order to deal with multiple volumes and their path in the settings file.
Added a host:volume option in client-py. This is needed for the following things: to specify the volume on which folders are located. The same name folder can occur on multiple volumes. Then based on given argument options such as volume and name the vfid of vfolder is infered. Then filebrowser based on volume path and vfid is able to mount those directories into the container.
On client-py command execution example: backend.ai filebrowser create -host local:volume1 -vf mydata1 -vf mydata2
Recent commit changes introduce the following updates:
- Before creating a container, added a feature to check if the virtual folder exists and if its status is set to Readable.
- Refactored the database handling by switching from SQLAlchemy to aiosqlite.
- The database is now initialized through the async ainit method.
- Implemented a new feature to ensure that when the Storage Proxy shuts down, all Filebrowser sessions and associated containers are removed accordingly.
Now we are going to merge this PR with the SFTP resource group running on storage proxy nodes. (related: #1246, #1250)