File serving over the Internet, for one
For personal notes and development, there’s a lot to be said for using a shared network file system. Dropbox has a collection of notification and sharing features that don’t naturally fit development use (where many files are intermediate products of a build process and not particularly interesting at any point of their lifecycle). In this post, I outline the steps to build a single server-to-multiple client file service configuration that might be suitable for text editing and relatively small software builds. (You always want one or more big servers with fast I/O for large builds.) Technologies used are Wireguard, MUNGE, and 9P; examples are given for Ubuntu.
Dropbox is great for managing files shared with others, and also convenient for some forms of archiving.
The goal is to set up a network share that can be used from various systems. In my initial deployment there will be a stationary workstation and a laptop. Both are running current Ubuntu. The server is running current Ubuntu as well.
In most situations, my network connectivity—even tethered—is better that what we had in university and at work. (MPK17 received hundred megabit Ethernet long after most of the other buildings on campus.) That means, as long as we can put up with some latency, a network file service might be sufficient.
Our network transport is a Wireguard link to the server from each client. This choice means that the authentication and file operations are encrypted as they pass between client and server.
We have set up Wireguard using
dsnet for ease of administration.
dsnet is a lightweight command line utility for managing a Wireguard server
and its clients; in particular, it manages the IP address assignment for the
clients. Follow the
dsnet author’s instructional post for the
$ sudo apt install wireguard $ wget https://github.com/naggie/dsnet/releases/download/v0.1/dsnet-linux-amd64 $ sudo cp dsnet-linux-amd64 /usr/local/bin/dsnet $ sudo dsnet init [ edit /etc/dsnetconfig.json as appropriate ] $ sudo dsnet up
add subcommand can be used to generate the configuration for
a new client.
$ sudo dsnet add glowy > dsnet-glowy.conf
glowy is a Hades Canyon NUC with an illuminated logo on its top.)
Copy this configuration to the appropriate workstation. You’ll also need the
IP address of the server’s Wireguard interface; we’ll refer to this as
resolvconf. One possible
$ sudo ln -s /usr/bin/resolvectl /usr/local/bin/resolvconf
On that workstation, test the configuration using
$ sudo apt install wireguard $ wg-quick up ./dsnet-glowy.conf
Both authentication and file service will use the Wireguard link for transport. The link is not configured to take all traffic from the client, although that is certainly possible.
Authentication for file service is Munge, which will securely communicate the user ID (UID) and group ID (GID) making each request. Your UIDs (and GIDs) should match across systems.
Munge reminds me somewhat of NIS, in that a shared secret is used to allow a group of machines to trust one another’s user and group IDs. Munge has superior security characteristics to NIS, benefiting from delivering a smaller set of features and using, well, cryptography.
On the server and each client install Munge.
$ sudo apt install munge
postinstall script for this package creates a weak key to keep the
installation time low, so on the server generate a better one:
$ sudo create-munge-key -f -r
You need to place a copy of this key in
/etc/munge/munge.key on each client
system. This file needs to be owned by the
munge user and group. Updating the
key will require it to be distributed to each participating system, client and
server. (And restarting the service as well, once the key is in place.)
Finally, we use 9P for network file service. We are using the
On the server, we create a
/sync directory, under which we will collect the
directories we plan to share.
$ apt install diod $ mkdir /sync/$USER
/sync/$USER to /etc/diod.conf
You can invoke
diod directly, with logging off and authentication on:
$ sudo diod -f
(There are command line options for debugging and for deactivating authentication.)
DIOD_ENABLE to true in
$ sudo vim /etc/default/diod
and ensure that the
diod service starts successfully.
On the client, create a target directory for the mount
$ mkdir ~/sync
And then use
diodmount to make the mount:
$ apt install diod $ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync $ ls ~/sync
Since this mount was the first time I had used 9P, I spent a brief period of time copying around ~80MB files, just to get a feel for the performance.
$ time cp $EIGHTY_MB_FILE /tmp cp $EIGHTY_MB_FILE /tmp 0.00s user 0.14s system 66% cpu 0.222 total $ time cp $EIGHTY_MB_FILE ~/sync cp $EIGHTY_MB_FILE sync 0.00s user 0.23s system 0% cpu 2:38.76 total $ stat $EIGHTY_MB_FILE File: $EIGHTY_MB_FILE Size: 81224528 Blocks: 158648 IO Block: 4096 regular file Device: 802h/2050d Inode: 37748762 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1001/ $USER) Gid: ( 50/ staff) Access: 2020-10-27 18:05:41.399117077 -0700 Modify: 2019-06-06 06:58:17.680609774 -0700 Change: 2020-10-27 15:26:29.596553909 -0700 Birth: 2019-06-06 06:58:13.608637639 -0700
That’s an effective rate of ~511 kilobytes/second. I have a typical Comcast home cable link, so there’s a maximum of ~6 megabits/second available.
Linux has FS-Cache, a built-in caching subsystem for network file
systems. We can enable caching for 9P by using the
cache mount option:
$ sudo umount ~/sync $ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync -o cache=fscache
If we rerun our copy operation, we see
$ time cp $EIGHTY_MB_FILE ~/sync cp $EIGHTY_MB_FILE sync 0.01s user 0.18s system 1% cpu 16.538 total
Which is an effective speed of ~4911 kilobytes/second. I suspect most of this
speedup is just making the operations asynchronous from the perspective of our
There isn’t much recent work on 9P performance. Van Hensbergen and Minnich (2005) showed that 9P was generally competitive with NFS on a variety of workloads.
In an attempt to make detaching and reattaching the mount simple, I wrote
a brief script, named
toggle-sync. You’ll need to make sure
USER are defined.
#!/bin/bash -p hostname=$(uname -n) mountpoint=$HOME/sync server=$SERVERADDR serverdir=/sync/$USER if ! which wg-quick; then echo "may need apt install wireguard" exit 1 fi if ! which diodmount; then echo "may need apt install diod" exit 1 fi if ! pgrep -af munged; then echo "munged not running; may need apt install munge and completed key configuration" fi if ip addr | grep -q "dsnet-$hostname"; then echo "dsnet-$hostname wireguard link present" else wg-quick up "./dsnet-$hostname.conf" fi if mount | grep -q /home/sch/sync; then echo "$mountpoint mounted; unmounting" sudo umount "$mountpoint" else echo "$mountpoint not mounted; mounting" sudo diodmount -n "$server:$serverdir" "$mountpoint" -o cache=fscache fi
So far this setup has been working well for notes and small files. This post
(and its Hugo build) were written (and invoked) from
~/sync. It’s nice to be
able to switch machines without losing my working state.