docker-sync support

[bsormagec]

I think docker-sync support will be good for mac because docker has poor performance on osx. Can you add this to devilbox ?

[cytopia]

@bsormagec thanks for pointing it out. Can you direct me to some more documentation about this.

[bsormagec]

Sure,

https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration

[vzlatev]

Hello,

Any plans to include it in the upcoming releases - on OSX without docker-sync web apps are awfully slow.

Regards

[coredevel]

+1, without docker-sync there are a couple of options for os x when mounting shares using bind mounts:

• consistent or default: The default setting with full consistency, as described above.
• delegated: The container runtime’s view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.
• cached: The macOS host’s view of the mount is authoritative. There may be delays before updates made on the host are visible within a container.

(Source: https://docs.docker.com/engine/admin/volumes/bind-mounts/#configure-mount-consistency-for-macos)

Unfortunately though none of them appear to resolve the bi-directional sync without huge performance overheads (in the case of the default consistent setting) or persistence issues (e.g. if uploading images to a cms and building up a site's content locally with the docker container instance). Using vagrant in the past for large Magento builds I tried both unison and gatling-rsync, the latter being more performant and the final solution chosen which worked wonders. It ended up being a v similar setup to the docker-sync native_osx implementation, more reading material here:

• http://docker-sync.io/
• https://github.com/EugenMayer/docker-sync
• https://github.com/EugenMayer/docker-sync/wiki/How-does-native_osx-work-with-OSXFS-and-is-still-native-speed-fast%3F

As an aside a short term workaround to resolve performance issues at the moment is to use memcache/redis caching which works really well, managed to get a basic WordPress instance sub 80ms fully rendered in the browser!

[mad99]

Any news about a well usable and reasonably fast system under osx?

Have tried to extend the Makefile on my own (process to include docker-sync-stack). But it does not work well... Maybe someone else is also to try to do this way. Would great to share or hear from news in this directions!

[cytopia]

This is still on the agenda, however the roadmap is currently really filled and me alone working on that takes some time. Probably somebody wants to jump in on this issue and submit a PR.

[johnwbaxter]

I tried setting up docker-sync but ended up somehow making docker use 400% cpu...will try again though as it's almost unusable on OSX as it stands.

[johnwbaxter]

@cytopia What would it cost to sponsor you to move this to the top of the list?

[cytopia]

@johnwbaxter I'd rather have someone helping me on this issue, as I dont' have a Mac at the moment. So I cannot get sponsored on this one and make an unreliable promises/guess. But thank you for bringing this up.

[cytopia]

There seems to be a native approach by Docker for volume mounts with the following options:

• consistent
• cached
• delegated

How would one of those strategy perform against the current mount options:

https://docs.docker.com/docker-for-mac/osxfs-caching/#tuning-with-consistent-cached-and-delegated-configurations

Might give it a try. If this does not work out, I will have alook into docker-sync

[cytopia]

@bsormagec @vzlatev @richdel @mad99 @johnwbaxter

The above suggestion should now be easily achievable via global mount options: https://devilbox.readthedocs.io/en/latest/configuration-files/env-file.html#mount-options

[ericgerhardy]

Has anyone solved this problem by using docker-sync or the global mount options? If yes, could you please tell us your configuration? Thanks in advance!

[cytopia]

@ericfynnmaier have you already tried one of these:

MOUNT_OPTIONS=,consistent
MOUNT_OPTIONS=,cached
MOUNT_OPTIONS=,delegated

Probably ,cached will give you some performance boost. I have no OSX anymore to try. Can you give it a shot and report how well it performs in terms of the empty default

[ericgerhardy]

@cytopia thank you! I added ,cached to the .env and e.g. Wordpress loads way faster!

[vielhuber]

,cached only seems to improve performance on mac, but not windows. how do windows people work with devilbox in a performant manner?

[cytopia]

@vielhuber I haven't looked into Windows yet for those kinds of performance improvements. Any idea what alternatives are offered for Windows

[vielhuber]

The only tool everybody is mentionning is "docker-sync" (http://docker-sync.io/). This seems to solve the problems everywhere. But it seems quite complex to install on Windows.

[cytopia]

OK, didn't know that this also works on Windows.

[vielhuber]

On which OS do you work? Do you also suffer from performance problems like everybody else when spinning up a normal Laravel project?

[cytopia]

I started this project on OSX and am now on Debian Linux. So far no performance issues (on Linux).

Update: I even tried Magento and Shopware, both very heavy projects and they have a decent speed on my machine.

[vielhuber]

Nice; Linux seems to have zero problems.

One temporary solution: Can we disable all filemounts in devilbox and to the sync on our own (e.g. with GoodSync via SSH)?

[jeffwhitfield]

I would love to know how to get docker-sync working with Devilbox. Running Magento on my Mac is absolutely atrocious! Tried everything but docker-sync which, based on what I'm reading, will result in a serious performance bump.

[anselmdk]

Just adding myself here. I spent the day yesterday getting up to speed with Devilbox (I absolutely LOVE it -great work!), however I'm also experiencing performance issues on the Mac...

[krebbi]

What about using SSHFS to access the dockers document root from the host?

https://github.com/libfuse/sshfs

[chiribuc]

Hi guys,

I think I successfully installed docker sync on my Mac.
This are the steps to make it work on your Mac OS machine:

Step 1.

Install docker-sync:

gem install docker-sync

Install unison:

brew install unison
brew install eugenmayer/dockersync/unox

Step 2.

Go to your devilbox directory, where you have the docker-compose.yml file and create two new files, docker-sync.yml and docker-compose-dev.yml.

In the docker-sync.yml file paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'

options:
    # the path to devilbox docker-compose.yml file
    compose-file-path: 'docker-compose.yml'
    
    # the path to your custom docker-compose-dev.yml file
    compose-dev-file-path: 'docker-compose-dev.yml'

syncs:
    # this name has to be unique and is used in docker-compose-dev.yml as well
    wordpress-sync:
        # which folder to watch / sync from - you can use tilde (~), it will get expanded. Be aware that the trailing slash makes a difference
        # if you add them, only the inner parts of the folder gets synced, otherwise the parent folder will be synced as top-level folder
        src: '~/data/www/wordpress/'
        
        # this does not user groupmap but rather configures the server to map
        # optional: usually if you map users you want to set the user id of your application container here
        sync_userid: '1000'
        
        # optional, default can be either rsync or unison See Strategies in the wiki for explanation
        sync_strategy: 'unison'
        
        # optional, a list of excludes. These patterns will not be synced
        # see http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#ignore for the possible syntax and see sync_excludes_type below
        sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']
       
    # other app here 
    other-app-sync:
        src: '~/data/www/other-app/'
        sync_userid: '1000'
        sync_strategy: 'unison'
        sync_excludes: ['.gitignore', '.git/', '.idea/']
        
    # and so on...

In the docker-compose-dev.yml paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'
services:

bind:
    volumes: 
        # set the apps that you want to sync
        - wordpress-sync:/shared/httpd/wordpress:nocopy
        - other-app-sync:/shared/httpd/other-app:nocopy

volumes:
    # set each app volume as external
    wordpress-sync:
        external: true
    other-app-sync:
        external: true

Step 3.

Run docker with docker sync using:

docker-sync-stack start

To stop the devilbox ctr+c like always and docker-sync stop. I have an alias for this in my zsh.

It's working for me so I hope will work for you.

[chiribuc]

P.S. - It seems that for me it works even better if I use native_osx as sync_strategt in the docker-sync.yml file.

[cytopia]

@chiribuc This is really great news. I've also linked it here in case someone stumbles across this issue via the forum: https://devilbox.discourse.group/t/fix-macos-performance-issues-via-docker-sync-mvp-available/72

Btw, if you do have bigger applications, can you tell something on the performance gain with docker-sync in place vs without.

[chiribuc]

@chiribuc This is really great news. I've also linked it here in case someone stumbles across this issue via the forum: https://devilbox.discourse.group/t/fix-macos-performance-issues-via-docker-sync-mvp-available/72

Btw, if you do have bigger applications, can you tell something on the performance gain with docker-sync in place vs without.

I will try these days but at a simple look, the speed increased with more than 1 second.

[cytopia]

In the docker-compose-dev.yml paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'
services:

bind:
    volumes: 
        # set the apps that you want to sync
        - wordpress-sync:/shared/httpd/wordpress:nocopy
        - other-app-sync:/shared/httpd/other-app:nocopy

volumes:
    # set each app volume as external
    wordpress-sync:
        external: true
    other-app-sync:
        external: true

@chiribuc why are you overwriting bind in docker-compose-dev.yml and not the httpd and php container which actually specifiy the mounts? bind does not have any volumes anyway.

So I was thinking of something like this instead:

version: '2.1'

services:
  php:
    volumes:
      - devilbox-intranet-sync:/var/www/default:nocopy
      - devilbox-data-www-sync:/shared/httpd:nocopy
  httpd:
    volumes:
      - devilbox-intranet-sync:/var/www/default:nocopy
      - devilbox-data-www-sync:/shared/httpd:nocopy

volumes:
  # set each app volume as external
  devilbox-intranet-sync:
    external: true
  devilbox-data-www-sync:
    external: true

With the following docker-sync.yml:

version: '2'

options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose-dev.yml'

syncs:
  devilbox-data-www-sync:
    src: 'data/www/'
    sync_userid: '1000'
    sync_strategy: 'unison'
    sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']

  devilbox-intranet-sync:
    src: '.devilbox/www:/var/www/default/'
    sync_userid: '1000'
    sync_strategy: 'unison'
    sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']

Also note that docker-sync.yml did not allow me to specify version: '2.1', it asks for version: '2'

[cytopia]

I am currently trying to make a quick PR for testing, so it can still go into v1.0.0 next week. A few issues I already stumbled across:

• https://github.com/EugenMayer/docker-sync/issues/647
• https://github.com/EugenMayer/docker-sync/issues/648
• https://github.com/EugenMayer/docker-sync/issues/649

[chiribuc]

I am currently trying to make a quick PR for testing, so it can still go into v1.0.0 next week. A few issues I already stumbled across:

• EugenMayer/docker-sync#647
• EugenMayer/docker-sync#648
• EugenMayer/docker-sync#649

It is better to use sync_strategy for each volume instead of the global default_strategy.

[cytopia]

It is better to use sync_strategy for each volume instead of the global default_strategy.

Good point, but this does not change the default_strategy issue on a per volume base either.

[chiribuc]

@cytopia you asked me why are you overwriting bind in docker-compose-dev.yml and not the httpd and PHP container which actually specify the mounts? bind does not have any volumes anyway.

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

[chiribuc]

The purpose, I think, it's not necessary to overwrite the containers but to tell docker that docker-sync will use those volumes to sync data.

[cytopia]

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

Can you check if the following workaround would make sense:

1. Set your project directory to somewhere outside the Devilbox (where also the other word-press plugins are)
   • https://devilbox.readthedocs.io/en/latest/howto/devilbox/move-projects-to-different-directory.html#howto-move-projects-to-a-different-directory
2. Disable the wordpress plugins from being shown in the intranet/ not have vhosts created for them:
   • https://github.com/cytopia/devilbox/pull/497

This would then not require you to have something outside the Devilbox (which is related to the Devilbox) and still have a clean WebUI and only vhosts that are actually vhosts.

[cytopia]

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

This really sounds strange :-)

I would assume that when not set on httpd and php that you will probably run a mix of docker-sync and Devilbox volumes.

[chiribuc]

I'm in the root of my user ~/ and my structure is like this:

- devilbox
- data
  - www
     - wordpress-site
         - wp-content
           - plugins
     
- repositories
  - wordpress
     - plugins
        - some-plugin
           -- ***plugin-name***
           -- src
           -- webpack.conf.js
           -- composer.json
           -- package.json
           -- etc.
        - some-other-plugin
          -- ***some-other-plugin-name***
          -- src
          -- etc.

So in this case, I will symlink the plugin-name which contains all the plugin built files, to the wordpress-site/wp-content/plugins directory. I tried different scenarios but only the one that I use seems to work without any problems.

[cytopia]

@chiribuc can you create a separate issue for that including how you've altered docker-compose.yml in order to achieve such a symlinking magic.

[chiribuc]

@chiribuc can you create a separate issue for that including how you've altered docker-compose.yml in order to achieve such symlinking magic.

I haven't altered it directly. On the docker-compose.overwrite.yml file, I added

version: '2.1'

services:
    php:
        volumes:
            - $HOME:$HOME
    
    httpd:
        volumes:
            - $HOME:$HOME

[cytopia]

I guess this is something that you probably also want to have in docker-sync or? Can imagine that performance on this will also degrade otherwise.

[cytopia]

Docker Sync Support (PR open)

@bsormagec @vzlatev @richdel @mad99 @johnwbaxter @ericfynnmaier @vielhuber @jeffwhitfield @anselmdk @krebbi @chiribuc @OksanaRomaniv @akuehn @lesteak @Yame- @ohrlab

I have created a first working draft including instructions how to get this working.

• PR: https://github.com/cytopia/devilbox/pull/535

Thanks again to @chiribuc jumping in and providing a working example.

[buderus199]

I have tested docker-sync on my updated windows 10 machine with latest stable docker for desktop. Mainly because of slow magento 2 project.

I'm using latest devilbox from master branch and I have added docker-sync.yml & docker-compose-dev.yml file (from docker-sync branch) to devilbox root folder. I did run docker-sync and get "success Starting Docker-Sync in the background", but can't see any improvements in loading times or deployment (php bin/magento setup:static-content:deploy) for magento 2 project. Probably I'm missing something?

[cytopia]

@buderus199 Did you adjust (uncomment) the sync_strategy according to what is required for Windows?

https://github.com/cytopia/devilbox/pull/535/files#diff-09bf18a28f8967be55bc552a00f967dfR32

[buderus199]

@cytopia yes I have used sync_strategy: 'unison'

This is the output when running docker-sync start from Ubuntu WSL:

/var/lib/gems/2.5.0/gems/docker-sync-0.5.11/lib/docker-sync/update_check.rb:41: warning: Insecure world writable dir /mnt/c in PATH, mode 040777 ok Starting unison for sync devilbox-intranet-sync doing initial sync with unison Unison 2.51.2 (ocaml 4.06.1): Contacting server... Looking for changes Reconciling changes Propagating updates UNISON 2.51.2 (OCAML 4.06.1) started propagating changes at 14:30:48.56 on 29 Apr 2019 [BGN] Copying from /app_sync to /host_sync [END] Copying UNISON 2.51.2 (OCAML 4.06.1) finished propagating changes at 14:30:48.56 on 29 Apr 2019 Saving synchronizer state Synchronization complete at 14:30:48 (1 item transferred, 0 skipped, 0 failed) real 0m 0.04s user 0m 0.04s sys 0m 0.00s chown ing file to uid 1000

And it stays like this for hours. Not sure if it is normal to stay like this...

[cytopia]

@buderus199 If the docker-sync command stales, its probably something to ask these guys: https://github.com/EugenMayer/docker-sync/

[ClearMarble]

I'm using the PR for this and noticing a definite performance improvement on MacOS. CPU usage seems significantly lighter on page loads.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ClearMarble]

Hey, do you imagine this would still work if I updated to the latest version? Been holding off updating due to this but don't know if anything relevant has really changed.

[pirex360]

Still at this date, the last version of devilbox on mac , using laravel...very slow.

[krebbi]

Same here. It's incredible slow on my Macbook Pro. I have to restart the Docker Images every few hours, otherwise they are slowing down over time.

[cytopia]

The performance is still an issue with MacOS itself - they haven't improved it yet (since years). The workaround using docker-sync is still blocked by this missing feature: https://github.com/EugenMayer/docker-sync/issues/649

[science695]

@cytopia Since they seem to have found an intense debate on this setting, would it not be possible to default it to native, and in instructions for macos say uncomment this line from the .env sample?

syncs:
  default-sync:
    sync_strategy: '${SYNC_STRATEGY:native}'

env-sample

# For MacOS using docker-sync, uncomment this line:
# SYNC_STRATEGY=rsync

It seems like the extra step to add 1 line to .env file is much better than being without this feature to fix the MacOS performance problem.

[cube-dan]

@cytopia,

Has your mac situation changed since 2018? You said "... as I dont' have a Mac at the moment."

If you have you developed a devilbox specific docker-sync.yml file for PR #535, I'd love to give it a spin and report back my test results here.

Simply put, I'm volunteering time to help, any way I can, to help get the docker-sync feature off the ground.

Thanks @cytopia for all your amazing work on DevilBox!

-Dan

[FJLessing]

@cytopia or anyone really has there been any movement on this front?

[Lenitr]

Docker on Mac has an experimental feature now that allows directory sharing using a Monterey only function called VirtioFS. I'm currently experimenting with it to see if everything runs ok using this. So far it speeds up the containers significantly, especially on requests with a lot of file reads.

Last week i had some issues with composer packages, giving it another go now.

[Lenitr]

So far so good, haven't encountered any issues anymore since switching. Running composer, installing npm packages etc. all done in devilbox php container ran without issues (at least for my usage).

I'm running MacOS Montery 12.3.1 on an Intel Macbook with Docker Desktop 4.7.1 (4.7.0 last week).

The feature can be enabled here:

Hopefully more people can try it out and post their feedback, I think it's worth a try (but keep in mind that it's an experimental feature).

[anselmdk]

@Lenitr that sounds awesome. Is that all you needed to do, no settings withing devilbox?

[Lenitr]

@anselmdk Yes, correct. The slowness was not a Devilbox issue, but rather Docker on Mac issue. Sharing the filesystem was very slow, but this new technique seems to be a bit faster. Only changing the setting in Docker desktop sped up the containers for me.

To manage your expectations though, it's still not as fast as running any Docker containers on a Linux host, but it's a nice improvement.