How to change the buffer size of StreamWriter in C#

You can modify StreamWriter’s buffer size by specifying the bufferSize parameter. For example, if you're writing a large file (such as one that’s approximately 500 MB), you can increase the buffer size to improve performance.

Below is an example where we increase the buffer size to 128 KB:

using System.IO;

var filePath = @"C:\LargeFiles\bigFile.txt";

using (var writer = new StreamWriter(filePath, append: true, encoding: Encoding.UTF8, bufferSize: 128 * 1024))
{
    for (int i = 0; i < 50000000; i++)
    {
        writer.WriteLine("Sample text");
    }
}

In this example, increasing the buffer size from the default 1 KB to 128 KB resulted in the file writing process being twice as fast on my system.

For instance, doubling the buffer size from 128 KB to 256 KB did not result in any noticeable improvement and only consumed additional memory. To find the best buffer size for your use case, experiment with different sizes and measure the performance.

1. Using FileStream Buffering

FileStream, which is used by StreamWriter to interact with files, also employs its own buffer. By default, FileStream uses a byte array buffer of size 4096 bytes (4 KB). Data written to a FileStream (either directly or via StreamWriter) is first stored in this buffer, and once it reaches capacity, it is flushed to the disk.

If the data exceeds the buffer capacity, FileStream bypasses its buffer and writes the data directly to disk. Thus, even if you change the StreamWriter buffer size to something larger than the default 4096 bytes, there is no need to adjust the FileStream buffer size unless you are writing directly to FileStream.

For instance, when working directly with a FileStream, you can specify a custom buffer size in the constructor, as shown below:

using System.IO;

var filePath = @"C:\LargeFiles\bigFile.txt";

using (var stream = new FileStream(filePath, 
    FileMode.Append, FileAccess.Write, FileShare.None,
    bufferSize: 128 * 1024))
{
    // Perform write operations here
}

2. Using Flush() and AutoFlush

Flushing ensures that the data in the buffer is written to the underlying stream before it is full. This can be particularly useful when you want to ensure that data is written immediately, such as in the case of logging, where the data must be saved periodically without waiting for the buffer to fill.

You can flush the buffer manually by calling StreamWriter.Flush(), or you can set StreamWriter.AutoFlush to true to automatically flush the buffer after every write operation.

using (var writer = new StreamWriter(filePath))
{
    writer.WriteLine("1 - Initializing");
    writer.WriteLine("2 - Fetching data");
    writer.Flush();

    // Continue writing after flushing
    writer.WriteLine("3 - Processing");
    writer.WriteLine("4 - Saving results");
    writer.Flush();
}

Alternatively, setting AutoFlush to true will automatically flush after every write:

using (var writer = new StreamWriter(filePath))
{
    writer.AutoFlush = true;

    writer.WriteLine("1 - Initializing");
    writer.WriteLine("2 - Fetching data");
    writer.WriteLine("3 - Processing");
    writer.WriteLine("4 - Saving results");
}

Why Avoid AutoFlush for Large Data?

While AutoFlush may seem convenient, it can severely impact performance when writing large datasets. The primary purpose of buffering is to minimize the number of disk I/O operations, and AutoFlush effectively disables this optimization by performing an I/O operation after every write.

For example, using AutoFlush to write a 500 MB file resulted in a significant performance hit—taking over 200 times longer than writing the same data with buffering enabled. Therefore, for large-scale file writing, it’s best to avoid AutoFlush to maintain optimal performance.

using (var writer = new StreamWriter(filePath))
{
    writer.AutoFlush = true; // Disables buffering

    for (int i = 0; i < 50000000; i++)
    {
        writer.WriteLine("Data entry");
    }
}

In contrast, writing the same data with manual flushing or relying on the default buffering behavior (without AutoFlush) will be much more efficient for large files.

In summary, adjusting the buffer size in StreamWriter can improve performance when writing large files, but it’s important to understand the trade-offs involved with AutoFlush. Avoid AutoFlush for large data writes to retain the performance benefits of buffering.