The Perfect Sync

In a previous article, I detailed my awkward experience with Windows Home Server and setting up a stable FTP server. In the end, I converted it into an XP server and assigned my clients with NetDrive so that they could mount the FTP server as a drive letter and browse effortlessly.

Problem is, NetDrive’s performance was subpar with large files, such as PDFs. For example, if I were to open a large PDF, instead of caching the entire PDF and reading it locally, NetDrive would proceed to download the PDF in parts much the same way it works when you view a large PDF from within a web browser. Because many of the PDFs on the server are large (100MB+), this resulted in extremely slow performance. Moreover, a client requested that the files be made available offline so that data from the server could be reviewed and edited while on a plane, for example. Dropbox’s impressive core sync engine evidently left quite an impression, and rightfully so.

I began an epic quest for the perfect realtime, bi-directional sync utility that I could use in conjunction with NetDrive to sync between a local copy of the server data and the server itself — in an effort to essentially replicate the Dropbox-like sync experience.

I must have installed over a hundred sync applications. This was an exhausting and defeating endeavor. Many sync software applications are remarkably similar and there is heavy competition among them. At one point, I had a near-perfect sync using this remarkably capable (albeit not-entirely-user-friendly) program called FTP Auto Sync which I could get working about 80% of the time. However, all of the programs were unable to handle multi-client scenarios. My server is intended to handle changes from multiple users simultaneously. So, one user may delete a file while another user adds a 10GB worth of new files somewhere else. What I was essentially asking all of these programs to do was “check whatever was done last on the server and sync it back with whatever I did last locally and figure it out.” A tall order. Problem is, a locally residing program is blind to the changes the remote FTP server is doing while offline and is unable to monitor all folders at all times anyway.

Let’s say I delete four files from the server. How is the local client to know whether to delete the files from the local client, or the sync the local files back to the server? To get around this, some programs maintain a database of changes, however, they still do not work in a multi-client environment unless I were to switch to an entirely different platform, such as iFolder. Many programs simply were unable to handle syncing in realtime between a pseudo-drive such as NetDrive altogether.

I ended uninstalling everything, including NetDrive and installing WebDrive to replace it after discovering that it had advanced caching and offline synchronization features. The main WebDrive interface may look similar to NetDrive, but the underlying technology is much more robust and feature rich.

By default, WebDrive is configured for “Single User” under Cache Settings. This infers that you will be the only one affecting changes on the server. Since my server data will be changed by many users, I wanted mine to be Multi-user which makes WebDrive check the modified date/time on the server files to verify that the cached files are up to date, if they are not, it will re-download the files as necessary.

You’ll also note that on the bottom, there are two additional options for flushing the cache upon connecting. Since this is the default behavior for Multi-user, I selected Custom and unchecked these two so that the cached files would remain.

Once you have connected to your FTP server, WebDrive has a useful little tool called the WebDrive Monitor. You can access it by right clicking on the WebDrive system tray icon and selecting “Show Monitor.” The Monitor displays the raw FTP commands as they are executed between the client and server as well as file transfers, speed, and a handy progress bar. In addition, the WebDrive system tray icon flashes a world icon whenever data is being transferred. Whenever there is any kind of delay, it’s nice to be able to open up the Monitor and see what is slowing things down, or even to troubleshoot an issue on the server if necessary.

Now, to test the offline sync, I created a test folder entitled “server copy”; I placed four PDFs in the folder and selected the drive made available offline. To prepare a system to be ready for offline files unfortunately requires a few steps to be taken. Those steps can be found in complete detail here, so please refer to that PDF and read it entirely if you intend to use WebDrive for this purpose. First, I open the WebDrive program settings and check both “Synchronize Offline files at Connect time” and “Synchronize Offline files at Disconnect time” for full synchronization. With the server in “online mode” I right click on the “server copy” folder and under WebDrive> select “Make Available Offline.” Since I have the Monitor open, I can see that they are “Set offline.”

Once the files are marked for offline mode, I reconnect to the WebDrive in “offline mode” by checking the “Connect Offline” box next to the Connect button, then reconnect. Of course, you must first disconnect from the server in order to reconnect, even in offline mode. Once connected, I then VNC’d to the server unit directly and deleted 4.pdf and added a new text file called “server.txt”; on the local “offline” copy, I deleted 1.pdf and 2.pdf and added a new text file called “local.txt”

WebDrive has several ways to perform a sync: (1) you can right click on the drive letter itself and select WebDrive > Synchronize Now; (2) right click on the system tray icon, select your WebDrive and then choose Synchronize Now; and (3) simply going back into Online mode where WebDrive will automatically perform a sync on connect. I right clicked on the system tray icon and selected “Go Online” to initiate an auto-sync. The sync was near instantaneous since my files sizes were trivial.

I was hoping for a solution that would free my clients from having to right-click and generally be reminded so much that they are using a workaround for Dropbox, but without Dropbox’s core sync engine, this is as elegant as it gets. I am informed that Novell/kablink’s iFolder (server software not available for Windows) and SparkleShare (not yet available) as well as other utilities such as Unison (no longer under active development) could potentially solve my FTP sync issues as well. My quest is over, at least for now, and while it may not be entirely seamless, my clients will be given the closest thing to the perfect sync.

admin@variableghz.com