File System Watcher

[damen-dotcms]

Parent Issue

https://github.com/dotCMS/core/issues/28378

Task

We need some sort of tool to monitor file systems for changes, and trigger a push when a change is detected.

The idea is to include a new parameter --watch to the existing push commands (the global push and the specific ones).

After a brainstorming session, we agreed the following:

• Use Java nio watch service where possible to be notified of changes rather than polling and scanning the folder every time. The watch service only works if the OS supports it and on linux does not work on separate disk, so the code should fallback to polling if the Watch Service is not available.
• When a new notification is received, we should wait 2 seconds by default before running the push logic to make sure there are no further changes over a minimum time. This would prevent the case where there are logs of changes and we end up having to run the sync multiple times where it could be covered with the once if we waited for a few seconds. The waiting time can be set using a parameter as well, and if not defined, it will default to 2 secs.
• No explicit path will be provided, the service will use the same path being used by the push command (it includes the case when the --workspace is set)

Proposed Objective

Same as Parent Issue

Proposed Priority

Same as Parent Issue

Acceptance Criteria

Essentially we're aiming for automated deployment capabilities using the CLI.

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

[wezell]

The watcher sync should probably be Debounced - here is the Debouncer code, not sure if it runs in Quarkus:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotcms/concurrent/Debouncer.java

[nollymar]

Internal QA: Passed

Tested by @spbolton

[fabrizzio-dotCMS]

Note to QA watch mode is meant to be available only for push commands. So when the option is turned on the CLI sits and waits for any changes to happen in the watched path before it can send the files up the connected instance don't forget you can always get help by doing

 dot push -h
dotCMS global push
Usage: dotCLI push [-h] [-dau] [--dry-run] [-ff] [-fse] [-ra] [-rct] [-rf]
                   [-rl] [-rs] [-w[=watch]] [--dotcms-url=<remoteURL>]
                   [--retry-attempts=<retryAttempts>] [-tk=<token>] [path]
 This command push Sites, Content Types, Languages and Files to the server.

      [path]                 local directory or file to push
....
  -w, --watch[=watch]        When this option is enabled the tool observes
                               changes in the file system within the push path
                             If a change is detected the push command being
                               executed gets triggered.
                             The auto-update feature is disabled when watch
                               mode is on
                             The default watch interval is 2 seconds, but it
                               can be specified passing an integer value with
                               this option.
                             e.g. --watch 5

Again the watch option will be available for any push sub command if you do specify a path that path that will be watched, if no path is specified it will be inferred from the current workspace here a few examples:

run global push in watch mode over the the current workspace dotCLI push --watch run global push on specific path dotCLI push /Users/lalal/myWorkspace/ --watch

run site push over the current workspace dotCLI sites push /Users/lalal/myWorkspace/ --watch

[bryanboza]

Blocked by #28915

[bryanboza]

We need some work here:

• I'm encountering a weird error when I run the watch under a specific folder and make some changes. We are getting a permissions error in a different folder that is completely unrelated to the one I'm using in the watch.

Here is a video with the explanation

https://github.com/dotCMS/core/assets/2641437/f355242a-6a08-4f0c-a8ae-063ceadf6770

[fabrizzio-dotCMS]

A few things to notice here : The path passed on the video isn't valid it starts with files/ it isn't resolved as a relative path If I do

dot push files/live/en-us/demo.dotcms.com/lol -e I will get the same error executing any of the stand-alone commands

[ERROR] ❗  Error in command [push] with message:
  Unable to access the path [/Users/fabrizzioaraya/local/languages/lol] check that it does exist and that you have read permissions on it.
java.io.IOException: Unable to access the path [/Users/fabrizzioaraya/local/languages/lol] check that it does exist and that you have read permissions on it.
	at com.dotcms.cli.command.language.LanguagePush.call(LanguagePush.java:114)
	at com.dotcms.cli.command.language.LanguagePush_Subclass.call$$superforward(Unknown Source)

The cli does not report the error immediately when running in watch mode cause it waits for a file-system event. Of course, the resolution of the path /Users/fabrizzioaraya/local/languages/lol needs to improve.

When you do

dotCLI push

the CLI watches all subdirectories sites files content-types languages etc..

Cause we are performing a watch through the Global push In this case, whatever path we pass has to be resolved to the workspace, cause it affects all sub-commands as we're running a GLOBAL Push

Now if you only want to watch for changes in files you should use

dotCLI files /Users/fabrizzioaraya/local/files/.. --watch

again any path passed here should be resolved as a valid sub-folder of the workspace/files/ directory

[spbolton]

Tested cli against nightly with npm i -g @dotcms/[email protected] dotcms server : just docker-ext-run nightly (dotcms/dotcms:nightly image id: f1396833e532) native image: m1

Tested across filesystem boundary also

[bryanboza]

Tested again with the latest changes, and now I got a different error.

If I try with the global push using dotcli push --watch, it works as expected and waits for changes at the workspace level.

Now, If I try the watch in some specific folder, I'm getting an error when some change happens: Trying using: dotcli push content-types --watch and dotcli push /Users/bryanboza/Desktop/dotCLI/content-types --watch In both cases I got the same error

[fabrizzio-dotCMS]

@bryanboza it is not clear what you did here you ran dotcli push content-types --watch and dotcli push /Users/bryanboza/Desktop/dotCLI/content-types --watch and then what ? where did you drop the files ? under what folder ?

Can you please add that bit of info ?

[bryanboza]

@fabrizzio-dotCMS I just made a modification to one of the content-types files to ensure we are triggering the job after any change inside the content-types folder.

[spbolton]

Internal QA. Tested global push and pull changing site with and without watch option as well as running from sites folder and specifying path. No error messages and changes were pushed as expected

[bryanboza]

Fixed and tested on the latest CLI version, and now it is working as expected. A new error regarding the exceptions has been opened here: #29629

We can close this one for now.