File serving over the Internet, for one

| Stephen | Software.

For personal notes and development, there’s a lot to be said for using a shared network file system. Dropbox has a collection of notification and sharing features that don’t naturally fit development use (where many files are intermediate products of a build process and not particularly interesting at any point of their lifecycle). In this post, I outline the steps to build a single server-to-multiple client file service configuration that might be suitable for text editing and relatively small software builds. (You always want one or more big servers with fast I/O for large builds.) Technologies used are Wireguard, MUNGE, and 9P; examples are given for Ubuntu.

Dropbox is great for managing files shared with others, and also convenient for some forms of archiving.

The goal is to set up a network share that can be used from various systems. In my initial deployment there will be a stationary workstation and a laptop. Both are running current Ubuntu. The server is running current Ubuntu as well.

In most situations, my network connectivity—even tethered—is better that what we had in university and at work. (MPK17 received hundred megabit Ethernet long after most of the other buildings on campus.) That means, as long as we can put up with some latency, a network file service might be sufficient.

Our network transport is a Wireguard link to the server from each client. This choice means that the authentication and file operations are encrypted as they pass between client and server.

We have set up Wireguard using dsnet for ease of administration. dsnet is a lightweight command line utility for managing a Wireguard server and its clients; in particular, it manages the IP address assignment for the clients. Follow the dsnet author’s instructional post for the initial configuration.

$ sudo apt install wireguard
$ wget https://github.com/naggie/dsnet/releases/download/v0.1/dsnet-linux-amd64
$ sudo cp dsnet-linux-amd64 /usr/local/bin/dsnet
$ sudo dsnet init
[ edit /etc/dsnetconfig.json as appropriate ]
$ sudo dsnet up

dsnet’s add subcommand can be used to generate the configuration for a new client.

$ sudo dsnet add glowy > dsnet-glowy.conf

(glowy is a Hades Canyon NUC with an illuminated logo on its top.)

Copy this configuration to the appropriate workstation. You’ll also need the IP address of the server’s Wireguard interface; we’ll refer to this as SERVERADDR below.

On Ubuntu, wg-quick requires resolvconf. One possible workaround is:

$ sudo ln -s /usr/bin/resolvectl /usr/local/bin/resolvconf

On that workstation, test the configuration using

$ sudo apt install wireguard
$ wg-quick up ./dsnet-glowy.conf

Both authentication and file service will use the Wireguard link for transport. The link is not configured to take all traffic from the client, although that is certainly possible.

Authentication for file service is Munge, which will securely communicate the user ID (UID) and group ID (GID) making each request. Your UIDs (and GIDs) should match across systems.

Munge reminds me somewhat of NIS, in that a shared secret is used to allow a group of machines to trust one another’s user and group IDs. Munge has superior security characteristics to NIS, benefiting from delivering a smaller set of features and using, well, cryptography.

On the server and each client install Munge.

$ sudo apt install munge

The postinstall script for this package creates a weak key to keep the installation time low, so on the server generate a better one:

$ sudo create-munge-key -f -r

You need to place a copy of this key in /etc/munge/munge.key on each client system. This file needs to be owned by the munge user and group. Updating the key will require it to be distributed to each participating system, client and server. (And restarting the service as well, once the key is in place.)

Finally, we use 9P for network file service. We are using the diod implementation.

On the server, we create a /sync directory, under which we will collect the directories we plan to share.

$ apt install diod
$ mkdir /sync/$USER

Then add /sync/$USER to /etc/diod.conf

You can invoke diod directly, with logging off and authentication on:

$ sudo diod -f

(There are command line options for debugging and for deactivating authentication.)

Finally set DIOD_ENABLE to true in /etc/default/diod

$ sudo vim /etc/default/diod

and ensure that the diod service starts successfully.

On the client, create a target directory for the mount

$ mkdir ~/sync

And then use diodmount to make the mount:

$ apt install diod
$ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync
$ ls ~/sync

Since this mount was the first time I had used 9P, I spent a brief period of time copying around ~80MB files, just to get a feel for the performance.

$ time cp $EIGHTY_MB_FILE /tmp
cp $EIGHTY_MB_FILE /tmp  0.00s user 0.14s system 66% cpu 0.222 total
$ time cp $EIGHTY_MB_FILE ~/sync
cp $EIGHTY_MB_FILE sync  0.00s user 0.23s system 0% cpu 2:38.76 total
$ stat $EIGHTY_MB_FILE
  File: $EIGHTY_MB_FILE
  Size: 81224528  	Blocks: 158648     IO Block: 4096   regular file
Device: 802h/2050d	Inode: 37748762    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1001/   $USER)   Gid: (   50/   staff)
Access: 2020-10-27 18:05:41.399117077 -0700
Modify: 2019-06-06 06:58:17.680609774 -0700
Change: 2020-10-27 15:26:29.596553909 -0700
 Birth: 2019-06-06 06:58:13.608637639 -0700

That’s an effective rate of ~511 kilobytes/second. I have a typical Comcast home cable link, so there’s a maximum of ~6 megabits/second available.

Linux has FS-Cache, a built-in caching subsystem for network file systems. We can enable caching for 9P by using the cache mount option:

$ sudo umount ~/sync
$ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync -o cache=fscache

If we rerun our copy operation, we see

$ time cp $EIGHTY_MB_FILE ~/sync
cp $EIGHTY_MB_FILE sync  0.01s user 0.18s system 1% cpu 16.538 total

Which is an effective speed of ~4911 kilobytes/second. I suspect most of this speedup is just making the operations asynchronous from the perspective of our cp(1) invocation.

There isn’t much recent work on 9P performance. Van Hensbergen and Minnich (2005) showed that 9P was generally competitive with NFS on a variety of workloads.

In an attempt to make detaching and reattaching the mount simple, I wrote a brief script, named toggle-sync. You’ll need to make sure HOME, SERVERADDR, and USER are defined.

#!/bin/bash -p

hostname=$(uname -n)
mountpoint=$HOME/sync
server=$SERVERADDR
serverdir=/sync/$USER

if ! which wg-quick; then
	echo "may need apt install wireguard"
	exit 1
fi

if ! which diodmount; then
	echo "may need apt install diod"
	exit 1
fi

if ! pgrep -af munged; then
	echo "munged not running; may need apt install munge and completed key configuration"
fi

if ip addr | grep -q "dsnet-$hostname"; then
	echo "dsnet-$hostname wireguard link present"
else
	wg-quick up "./dsnet-$hostname.conf"
fi

if mount | grep -q /home/sch/sync; then
	echo "$mountpoint mounted; unmounting"
	sudo umount "$mountpoint"
else
	echo "$mountpoint not mounted; mounting"
	sudo diodmount -n "$server:$serverdir" "$mountpoint" -o cache=fscache
fi

So far this setup has been working well for notes and small files. This post (and its Hugo build) were written (and invoked) from ~/sync. It’s nice to be able to switch machines without losing my working state.