Get the most from your network

BeamDisk reads at over 1 GB/s on 10 GbE. What you see depends on the slowest part of the path: the network, the server’s pool, or the app doing the reading and writing. Measure your own with Test Speed.

What it can do

Measured on 10 GbE to TrueNAS, one disk at a time:

• Reads: about 1.1 GB/s at the standard 1,500-byte MTU, and about 1.2 GB/s with jumbo frames, 97% of what the link can carry, when the server had the data in memory. Two disks read together filled the link.
• Writes: as fast as the server’s pool takes them. A SATA SSD pool with default settings sustained about 830 MB/s, with bursts of 1.0 to 1.2 GB/s; a hard-drive pool about 340 MB/s.
• Reads and writes at the same time: 10 GbE carries data both ways at once, so one disk moved 1,844 MB/s in all to a Linux iSCSI target with SATA SSDs, about 1.5 times what one direction can carry. Reads took the larger share, about 1,150 MB/s against 690 MB/s of writes, so a copy that writes while another app reads runs slower than either alone. The benchmarks.

The network

• Use a wired connection. BeamDisk is tested on wired 10 GbE. After protocol overhead, 10 GbE can carry at most about 1,180 MB/s of data at the standard MTU (about 1,237 MB/s with jumbo frames), 2.5 GbE at most about 290 MB/s, and 1 GbE about 118 MB/s.
• Jumbo frames (MTU 9000) help, if every device on the path supports them: the Mac, every switch, and the server, including a hypervisor’s bridge and virtual network card if the server runs in a VM. In BeamDisk’s tests they made reads 7 to 9% faster, write bursts 12% faster and random writes 6% faster, and took work off the server’s network. On the Mac, set it in System Settings → Network → your Ethernet connection → Details → Hardware: choose Manually, then the MTU. See Set up TrueNAS for the server’s side.
• Keep storage on a trusted network, ideally its own VLAN: iSCSI traffic isn’t encrypted.

The server

• The pool sets the write speed: see the numbers above.
• SSDs make safe writes quick. On a hard-drive pool, each request to store data safely waited about three times longer in BeamDisk’s tests: about 21 ms, against 7 ms on a SATA SSD pool.
• Requests to store data safely wait for the pool. Apps that ask for their data to be safely stored, such as databases, wait for the server each time. On ZFS, a SLOG device is the usual way to shorten that wait.
• ZFS settings matter. 16 KiB zvol blocks, and the default sync and logbias: Set up TrueNAS has the measured differences.

Power-loss protection

Turning a disk’s power-loss protection off made long sequential writes about 9 to 11% faster in tests, with an occasional pause of several seconds, and recent writes can then be lost if the server crashes or loses power. Leave it on unless the disk’s contents don’t matter.

The disk

• Format with BeamDisk for random writes: they ran up to about three times faster in tests than on Disk Utility’s standard APFS format. See Format a new disk.

The Mac

• Screen Sharing in High Performance mode slows other transfers. While a High Performance session is open, macOS holds other network traffic back for a second every 30 seconds, and for the first second of each new connection, which cost about 3% of a long read in tests. Standard mode doesn’t do this.
• A heavily loaded Mac (a big build using every core, for example) can cut network throughput sharply. Check Activity Monitor if numbers drop.

The app

Apps that keep several requests in flight get the most, as Test Speed and Time Machine do. A copy that writes one stream at a time runs slower: a Finder copy of one large file onto a BeamDisk disk ran at 300 to 620 MB/s in tests.