Using development tools on NFS file shares is slow

[cptanalatriste]

✅ Checklist

• I have searched open and closed issues for duplicates.
• This is a problem observed when using a Data Safe Haven.
• I can reproduce this with the latest version.
• I have read through the documentation.
• This isn't an open-ended question (open a discussion if it is).

💻 System information

• Operating System: macOS
• Data Safe Haven version: v5.6.0
• Browser details: Google Chrome

🚫 Describe the problem

Our users reported several performance problems when running development tools (i.e. Codium, Python Virtual Environments) on the TRE. We have found online multiple reports that this happens when coding is performed from NFS file shares (as per the TRE user model):

https://serverfault.com/questions/1127070/very-slow-experience-with-students-nfs-development-tools
https://www.reddit.com/r/sysadmin/comments/120yk9j/students_nfs_development_tools_terribly_slow/

We have run some performance tests on the NFS file shares available from the workspace (like /home/) following Azure guidance. Our tests using fio (using 5G files) show the following:

• Sequential Bandwith:
  • Reads from /home: 74.4 MiB/s
  • Reads from OS disk /mnt/scratch: 158MiB/s
  • Writes to /home: 263MiB/s
  • Writes to /mnt/scratch: 195MiB/s
• Sequential IOPS:
  • Reads from /home: 134 IOPS
  • Reads from /mnt/scratch: 9186 IOPS
  • Writes to /home: 8245 IOPS
  • Writes to /mnt/scratch: 9186 IOPS
• Random IOPS:
  • Reads from /home: 113 IOPS
  • Reads from /mnt/scratch: 9185 IOPS
  • Writes to /home: 15.6K IOPS
  • Writes to /mnt/scratch: 9165 IOPS

These suggest reading from the NFS file share is order of magnitude slower, when compared to reading from the OS disk.

🚂 Workarounds or solutions

According to our tests, and the posts we found online, users can work on disk, and use Git to access their code among workspaces. At the moment, TREs only support user access to the temporary disk of SKUs that support them (see: #2432 ).

[martintoreilly]

Can you add performance numbers for blob storage? I think that's pretty fast, especially for reading. It's not fully POSIX compliant (doesn't work so well when multiple people may be making writes as I recall).

[martintoreilly]

Can you also rerun your data save and render tests with the 3 different storage options?

[craddm]

I don't know if this is really a bug per se - we'd generally expect worse performance with NFS than the local disk (which is why it's actually a surprise to me that write speeds seem higher over NFS). The sequential read bandwidth doesn't sound that bad - at 74MiB/s it's comparable to a typical HDD; but then the file share is supposed to be Premium SSD, and the IOPs do seem ver low

I'd imagine with VSCode and the like it's more likely to be doing lots of small reads that is the problem, so maybe 5G files are not the best comparison? Each read will have a bigger overhead on NFS than on local storage, so lots of small reads == lots more overheads. Trying it with a smaller file size might even make the problem more starkly visible in the fio results

I wonder if the other tweaks here might help: https://learn.microsoft.com/en-us/azure/storage/files/nfs-performance

Our NFS mounts already use nconnect=4 but maybe some of the other stuff ("read ahead buffers"?) might help?

[cptanalatriste]

From "How Linux Works" by Brian Ward (2021):

One thing you need to ask yourself when working with any kind of file sharing system is why you’re doing it in the first place. In traditional Unix-based networks, there were two major reasons: convenience and lack of local storage. One user could log in to one of several machines on a network, each with access to the user’s home directory... This model’s advantages are overshadowed by one major disadvantage that has remained constant over the years: network storage performance is often poor compared to local storage. Some kinds of data access are okay... However, if you’re doing more complex manipulation or accessing many different files at once, you’ll find your CPU waiting on the network more often than not. Latency is one of the primary culprits.... Before sending any data to the client, the server must accept and decipher the request, and then locate and load the data. The first steps are often the slowest, and are done for almost every new file access. The moral of the story is that when you start thinking about network file sharing, ask yourself why you’re doing it... if, for example, you’re editing video or developing a software system of any substantial size, you’ll want to keep all of your files on local storage.

Maybe we should explore giving end-user access to the OS Disk? At the moment, we're only allowing writing to temporary disks (if the VM supports it!)