Use lsyncd for Real-Time File Synchronization • TheUnknownBlog

Ever since I became an mjj (server hoarder), I’ve accumulated a lot of VPSs—I just can’t resist buying more. But to keep my blog data synchronized across multiple servers, I’ve put in quite a bit of effort. I got tired of using cron to automatically package my blog directory and then manually back it up. Ultimately, it’s laziness—I want a fully automated solution. Since my Typecho blog is deployed via Docker images, I figured I’d go the extra mile and tackle multi-end data synchronization, so that any change on one site is reflected on all sites.

When it comes to synchronizing files between multiple servers, rsync is a commonly used tool. It achieves efficient directory synchronization through incremental transfers, compression, and deletion operations. However, the classic working mode of rsync is “manual or scheduled trigger,” which falls short for scenarios requiring real-time synchronization.

How rsync Works#

rsync compares the differences between the source and target directories and only transfers changed files or data blocks, reducing bandwidth usage. This method is ideal for backing up and synchronizing large amounts of data, especially in bandwidth-constrained environments. However, rsync usually needs to be triggered manually or via scheduled tasks (like cron). For applications that require real-time updates, this approach leads to data lag and resource waste.

The Shortcomings of rsync + inotify#

To address real-time synchronization, you can use inotify to monitor file system changes and trigger rsync when changes occur. However, this approach has several obvious drawbacks:

inotify requires additional scripts to work with rsync, increasing system complexity.
This solution is usually one-way and cannot achieve multi-source real-time synchronization, which goes against my goals.

The Advantages of lsyncd#

To solve the above problems, lsyncd combines inotify’s real-time monitoring with rsync’s efficient transfer capabilities, providing a simple yet powerful solution for real-time synchronization. The advantages of lsyncd include:

lsyncd can handle complex real-time synchronization tasks with a simple configuration file, eliminating the need for extra scripts.
It supports one-way synchronization between multiple servers, ensuring that data on every server is up to date. Note: lsyncd does not natively support true bidirectional or multi-master sync with conflict resolution. But it doesn’t matter in my case because I only need one-way sync.

Step-by-Step Guide to Configuring lsyncd#

Here’s how to use lsyncd for real-time synchronization:

Install lsyncd and rsync:#

On all servers involved in synchronization, run the following command to install the necessary tools:

sudo apt-get install lsyncd rsync

bash

Configure lsyncd:#

On each server, create the configuration file /etc/lsyncd.conf with the following content:

settings {
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd.status",
    inotifyMode  = "CloseWrite or Modify",
    maxProcesses = 1,
    -- nodaemon = true,
}

sync {
    default.rsyncssh,
    source = "/var/www",
    targetdir = "/var/www",
    host = "45.*.*.*",
    delete = true,
    rsync = {
        binary = "/usr/bin/rsync",
        archive = true,
        compress = true,
        verbose = true,
    },
    delay = 1,
}

lua

Explanation:

source: The local directory to monitor, /var/www/ (replace with your own).
host: The remote target server (excluding itself).
targetdir: The remote target directory, /var/www/ (replace with your own).
delay: Sets the synchronization delay (in seconds) to prevent excessive syncing during frequent changes.
delete: Deletes files on the target server that have been deleted on the source server.

Note: When using rsyncssh, maxProcesses must be 1. If using rsync, you can set a higher value (e.g., 5).

Tip: For troubleshooting, it’s recommended to start with lsyncd /etc/lsyncd.conf to check for errors. Also, make sure to create the log directory first: mkdir -p /var/log/lsyncd.

One more thing: To allow servers to log in to each other without a password, you need to set up SSH key-based authentication.

To automate real-time synchronization, ensure the source server can log in to the target server via SSH without a password.

On the source server, generate an SSH key:

ssh-keygen -t ed25519

bash

Follow the prompts; usually, you don’t set a passphrase.

Copy the public key to the target server:

ssh-copy-id user@target-server

bash

This copies the generated public key to the target server, enabling passwordless login. Note: You need to configure this on both servers if you want mutual access.

Once configured, start verification.

Start lsyncd:#

lsyncd /etc/lsyncd.conf

bash

Verify the Configuration:#

Perform file operations in the /var/www/ directory on any server and check synchronization on the others.

Handling Conflicts#

When multiple servers modify the same file at the same time, conflicts may occur. However, my use case probably won’t encounter conflicts, so I’m leaving it as is :D