Non-cached/non-buffered File Operations (System.IO.FileSteam) with .NET Core on Windows and Mac - Creating Cross-platform Disk Benchmark App

If you have the question of executing disc reads/writes directly against the device and avoid file cache of the OS you're welcome to continue reading this post.

Intro

Assume you're creating a disk benchmark app you want to execute on Windows and macOS (and might be any other platform) and in your .NET Core project come up with a solution similar to the below C# code snippet:

var sw = new Stopwatch();
var file = new FileStream("C:\\testfile.dat", FileMode.CreateNew);
var buffer = new byte[1024 * 1024 * 1024];
var rand = new Random();
rand.NextBytes(buffer);

sw.Restart();
file.Write(buffer, 0, blockSize);
file.Flush();
sw.Stop();

- you create a file a get an instance of FileStream, write a randomized byte array to it and measure the time it took to complete the write.

Eventually you get some throughput measure (array.Lenght/Stopwatch.EllapsedMilliseconds), lets say 100 MB/s (typical spinning HDD) or 1000 MB/s (high end SSD). Consequently you want to measure read speed an use FileStream.Read() method on the same file you've just created and used for write test. You'll be surprised to see a read speed over 3000MB/s, expecting it to be not far from write speed (from my observations, on SSDs sequential write speeds are usually greater than  read ones).

The reason is intervention of OS's File Caching subsystem which sits behind system's file API and keeps in memory whatever recent data has been read from file or written to it. In our case the writing to file in the first step cached the array data in memory and on subsequent read from the file OS's API made a simple memcopy, no disk involved.

Windows and macOS caching work in similar way and results observed are the same. In Windows you can see how File Cache grows while writing a big file to the disk by checking "Cache" value in Task Manager/Performance/Memory tab:

Cut to The Chase

First, there's no out-of-the box capability in .NET Core 2.1 to turn of file caching (at least I found none).

Second, there's a way to use FileStream class in non-cached mode.

Third, you have to use different workarounds with code running on Windows and macOS.

The following if statement is a starting point:

if (System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform
(System.Runtime.InteropServices.OSPlatform.Windows)) 
{
}

On Windows you can disable buffering by passing FILE_FLAG_NO_BUFFERING flag in FileOptions options parameter of stream constructor. There's no such value in FileOptions enum and you have to pass it's numerical value (0x2000000 - this value is found in FileStream source codes and is treated as valid).

FileStream writeStream, readStream;

if (RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows)) 
{
  writeStream = new FileStream(
      path, FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite, buffer, FileOptions.WriteThrough);

  readStream = new FileStream(
      path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, buffer, (FileOptions)0x20000000);
}

Please note that 2 streams are created - one for writing (no flag) and another for reading (with FILE_FLAG_NO_BUFFERING). The reason for that is because you most likely will get an exception with writes to the stream open with the flag. If you still need non-buffered writes to the stream, feel free to investigate MS documentation and follow alignments discipline. For the sake of testing disk speed, writing random data to one stream and reading data from another non-buffered stream (pointing to the same file) solves the problem.

In macOS this trick doesn't work since the flag above is a piece of WinAPI and has no effect outside Windows. The approach on Mac is to turn off caching for already created FileStream instance, namely by turning it off for underlying OS file handle. In order to do this fcntl UNIX/POSIX function needs to be called with  F_NOCACHE attribute. There's no standard way in .NET Core to interop with POSIX API and Mono.Posix.NETStandard NuGet package comes to help. Below is the superclass of FileStream implementing the required behavior:

using Mono.Unix.Native;
using System.IO;

namespace Saplin.StorageSpeedMeter
{
    class MacOsUncachedFileStream : FileStream
    {
        public MacOsUncachedFileStream(string path, 
                                       FileMode mode, 
                                       FileAccess access, 
                                       FileShare share, 
                                       int bufferSize, 
                                       FileOptions options
                                      ) : base(path, mode, access, share, bufferSize, options)
        {
            Syscall.fcntl(
              (int)SafeFileHandle.DangerousGetHandle(), 
              FcntlCommand.F_NOCACHE);
        }
    }
}

The code snippet returning write stream and uncached read stream on both OSs will look the following way:

FileStream writeStream, readStream;

if (RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows)) 
{
  writeStream = new FileStream(
      path, FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite, buffer, FileOptions.WriteThrough);

  readStream = new FileStream(
      path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, buffer, (FileOptions)0x20000000);
}
else
{
  writeStream = new MacOsUncachedFileStream(
      path, FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite, buffer, FileOptions.None);

  readStream = new MacOsUncachedFileStream(
      path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, buffer, FileOptions.None);
}

P.S.: Though Linux hasn't been tested, the macOS trick might work there as well.

P.P.S.: You can find source codes of a complete .NET Core disk test class library and console app on GitHub.

P.P.P.S: compiled and ready to use console app can be downloaded from here.