A while ago, I signed up for the Docker for Mac Beta and was very excited once I got access to it. Mostly because I hoped one can get finally rid of the overhead and issues coming with VirtualBox and docker-machine. While the native integration is pretty sweet and just feels right, one issue which has not been solved is the slow sync with the host file system. With the release of docker-sync by Eugen Mayer, there’s finally a very promising tool available which helps you overcome this issue. In this post, I want to show you my simple setup with a two-way-sync.
Performance issues with syncing data
One problem with Docker Toolbox & VirtualBox is the terrible performance when sharing container data with the host filesystem, especially if you’re working on a project with a huge amount of files (10k+, in my case the awesome Neos CMS). Unfortunately, once I set up a shared volume with docker for mac, the macbook’s fan started rising up super-fast because the sync was still very resource-hungry and brought the CPU to its limits.
You can check this thread on the docker forum to see that many other people noticed this issue, too. While there are several workarounds mentioned using nfs, unison or rsync, all of them we’re quite complex to set up imho. Luckily, Eugen Mayer developed a an easy to use tool called
docker-sync, which bundles syncing with unison (two-way-sync) and rsync (one-way-sync).
Firstly, you should know that this tool is available for Mac only (for now?). Secondly, docker-sync does not rely on Docker for Mac, you can use it with Docker Toolbox, Virtual Box, etc., as well! Thirdly, I’ve only done the installation on a Mac with
brew installed, so other ways of installing it could be more complicated.
That being said, this is how you install docker-sync:
# install unison or rsync brew install unison brew install rsync # install fswatch to check for file changes brew install fswatch # install docker-sync gem install docker-sync
Since this tool is very new, the configuration could change quickly (see the wiki). Therefore instead of going through the config in detail, I will show you a simple config for my use case with Neos. In case you haven’t heard what this Neos thingy is, you should definitely check out http://neos.io.
The sync configuration is put into a separate file, so a possible structure could look like this:
├── docker-compose.yml # docker container conf ├── docker-sync.yml # sync conf └── neos # directory with the shared data
When configuring the
docker-sync.yml file, keep in mind that the name of the sync container (docker-sync uses docker container to sync docker container data 🤔) has to be unique.
# docker-sync.yml syncs: # this name has to be unique and is used in docker-compose.yml as well neos-data-sync: src: './neos' # the directory on the host system dest: '/data' # the directory on the container sync_host_ip: 'localhost' # localhost for docker for mac sync_host_port: 10872 # unique port sync_strategy: 'unison'
docker-compose.yml file there’s not much to configure. We just make the neos container use an external container as the volume with read/write-access. This container should be the sync-container which we configured above, so its name needs to be entered here as well.
# docker-compose.yml version: "2" services: web: image: million12/typo3-neos ports: - '9010:80' links: - db:db environment: T3APP_VHOST_NAMES: neos dev.neos behat.dev.neos T3APP_DB_PASS: 1234 T3APP_DB_USER: admin # container name from before volumes_from: - container:neos-data-sync:rw # will be mounted on /data db: image: million12/mariadb:latest ports: - 3306 environment: MARIADB_PASS: 1234 # container name from before volumes: neos-data-sync: external: true
Update: Keep your config portable!
If you look at the example above you might noticed that the
docker-compose.yml file is extended with a docker-sync related configuration. This could be a problem if you want to use it in production, but luckily there’s a way to overcome this problem.
The trick is to put all the sync specific configuration into a
docker-compose-dev.yml file, which will override the original configuration while you develop. You simply put this file right next to the other config files:
├── docker-compose.yml # clean(!) docker container conf ├── docker-compose-dev.yml # docker-sync overrides ├── docker-sync.yml # sync conf └── neos # directory with the shared data
docker-compose.yml can be cleaned up like this:
version: "2" services: web: image: million12/typo3-neos ports: - '9010:80' links: - db:db environment: T3APP_VHOST_NAMES: neos dev.neos behat.dev.neos T3APP_DB_PASS: 1234 T3APP_DB_USER: admin db: image: million12/mariadb:latest ports: - 3306 environment: MARIADB_PASS: 1234
And the sync override goes in
version: "2" services: web: # container name from docker-sync.yml volumes_from: - container:neos-data-sync:rw # will be mounted on /data # container name from docker-sync.yml volumes: neos-data-sync: external: true
To start this configuration including the overrides, enter
docker-compose -f docker-compose.yml -f docker-compose-dev.yml up. For more information, please take a look at the corresponding wiki page.
3) Starting the container
Open a terminal and navigate to the directory where you keep the configuration files from above.To start the sync process along with the containers you have two options:
The first option is to use
docker-sync-stack start. This command will start up the sync process and docker-compose as a single command. On top of that, it will automatically add the settings from a
docker-compose-dev.yamlfile when starting the container, so you can keep your original
docker-compose.ymlfree from docker-sync specific stuff! (Take look at the previous section)
The second option is to use
docker-sync start. This command will initialize and start the sync container along with the watchers. In another terminal, simply start the actual docker containers with
As soon as docker-compose has ‘finished’, you should see your Neos environment on
In my opinion docker-sync is a great option to bring adequate sync performance to docker for mac today. Though it has to be said that I can only speak for one personal use case with Neos, so there could be some caveats I did not encounter yet. If you’re interested just try it for yourself and leave a star on github for Eugen’s work. In case you need some more examples or want to see how other sync strategies work, check out the boilerplate repository on github.