A while ago, I signed up for the Docker for Mac Beta and was very excited once I got access to it. Mostly because I hoped one can get finally rid of the overhead and issues coming with VirtualBox and docker-machine. While the native integration is pretty sweet and just feels right, one issue which has not been solved is the slow sync with the host file system. With the release of docker-sync by Eugen Mayer, there’s finally a very promising tool available which helps you overcome this issue. In this post, I want to show you my simple setup with a two-way-sync.

Performance issues with syncing data

One problem with Docker Toolbox & VirtualBox is the terrible performance when sharing container data with the host filesystem, especially if you’re working on a project with a huge amount of files (10k+, in my case the awesome Neos CMS). Unfortunately, once I set up a shared volume with docker for mac, the macbook’s fan started rising up super-fast because the sync was still very resource-hungry and brought the CPU to its limits.

You can check this thread on the docker forum to see that many other people noticed this issue, too. While there are several workarounds mentioned using nfs, unison or rsync, all of them we’re quite complex to set up imho. Luckily, Eugen Mayer developed a an easy to use tool called docker-sync, which bundles syncing with unison (two-way-sync) and rsync (one-way-sync).

Docker-sync

Firstly, you should know that this tool is available for Mac only (for now?). Secondly, docker-sync does not rely on Docker for Mac, you can use it with Docker Toolbox, Virtual Box, etc., as well! Thirdly, I’ve only done the installation on a Mac with brew installed, so other ways of installing it could be more complicated.

1) Installation

That being said, this is how you install docker-sync:

# install unison or rsync
brew install unison
brew install rsync

# install fswatch to check for file changes
brew install fswatch

# install docker-sync
gem install docker-sync

2) Configuration

Since this tool is very new, the configuration could change quickly (see the wiki). Therefore instead of going through the config in detail, I will show you a simple config for my use case with Neos. In case you haven’t heard what this Neos thingy is, you should definitely check out http://neos.io.

The sync configuration is put into a separate file, so a possible structure could look like this:

├── docker-compose.yml # docker container conf
├── docker-sync.yml # sync conf
└── neos # directory with the shared data

When configuring the docker-sync.yml file, keep in mind that the name of the sync container (docker-sync uses docker container to sync docker container data 🤔) has to be unique.

# docker-sync.yml
syncs:
# this name has to be unique and is used in docker-compose.yml as well
  neos-data-sync: 
    src: './neos' # the directory on the host system
    dest: '/data' # the directory on the container
    sync_host_ip: 'localhost' # localhost for docker for mac
    sync_host_port: 10872 # unique port
    sync_strategy: 'unison'

In the docker-compose.yml file there’s not much to configure. We just make the neos container use an external container as the volume with read/write-access. This container should be the sync-container which we configured above, so its name needs to be entered here as well.

# docker-compose.yml
version: "2"
services:
  web:
    image: million12/typo3-neos
    ports:
      - '9010:80'
    links:
      - db:db
    environment:
      T3APP_VHOST_NAMES: neos dev.neos behat.dev.neos
      T3APP_DB_PASS: 1234
      T3APP_DB_USER: admin
    # container name from before
    volumes_from:
      - container:neos-data-sync:rw # will be mounted on /data
  db:
    image: million12/mariadb:latest
    ports:
      - 3306
    environment:
      MARIADB_PASS: 1234

# container name from before
volumes:
  neos-data-sync:
    external: true

Update: Keep your config portable!

If you look at the example above you might noticed that the docker-compose.yml file is extended with a docker-sync related configuration. This could be a problem if you want to use it in production, but luckily there’s a way to overcome this problem. The trick is to put all the sync specific configuration into a docker-compose-dev.yml file, which will override the original configuration while you develop. You simply put this file right next to the other config files:

├── docker-compose.yml # clean(!) docker container conf
├── docker-compose-dev.yml # docker-sync overrides
├── docker-sync.yml # sync conf
└── neos # directory with the shared data

The docker-compose.yml can be cleaned up like this:

version: "2"
services:
  web:
    image: million12/typo3-neos
    ports:
      - '9010:80'
    links:
      - db:db
    environment:
      T3APP_VHOST_NAMES: neos dev.neos behat.dev.neos
      T3APP_DB_PASS: 1234
      T3APP_DB_USER: admin
  db:
    image: million12/mariadb:latest
    ports:
      - 3306
    environment:
      MARIADB_PASS: 1234

And the sync override goes in docker-compose-dev.yml.

version: "2"
services:
  web:
    # container name from docker-sync.yml
    volumes_from:
      - container:neos-data-sync:rw # will be mounted on /data

# container name from docker-sync.yml
volumes:
  neos-data-sync:
    external: true

To start this configuration including the overrides, enter docker-compose -f docker-compose.yml -f docker-compose-dev.yml up. For more information, please take a look at the corresponding wiki page.

3) Starting the container

Open a terminal and navigate to the directory where you keep the configuration files from above.To start the sync process along with the containers you have two options:

  • The first option is to use docker-sync-stack start. This command will start up the sync process and docker-compose as a single command. On top of that, it will automatically add the settings from a docker-compose-dev.yaml file when starting the container, so you can keep your original docker-compose.yml free from docker-sync specific stuff! (Take look at the previous section)

  • The second option is to use docker-sync start. This command will initialize and start the sync container along with the watchers. In another terminal, simply start the actual docker containers with docker-compose up.

As soon as docker-compose has ‘finished’, you should see your Neos environment on http://dev.localhost:9010.

Conclusion

In my opinion docker-sync is a great option to bring adequate sync performance to docker for mac today. Though it has to be said that I can only speak for one personal use case with Neos, so there could be some caveats I did not encounter yet. If you’re interested just try it for yourself and leave a star on github for Eugen’s work. In case you need some more examples or want to see how other sync strategies work, check out the boilerplate repository on github.

Useful resources