Writing Files Asynchronously with Multiple Threads in .NET Core

You can use ReaderWriterLock, semaphores, and the lock statement to control access to shared resources.

Multiple threads send their data to a ConcurrentQueue, which is a thread-safe collection designed for scenarios like this. A background task continuously monitors the queue and performs the actual file writing, ensuring that only one thread accesses the file at any given time.

using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace MyApp
{
    public class MultiThreadFileWriter
    {
        private static ConcurrentQueue<string> _textToWrite = new ConcurrentQueue<string>();
        private CancellationTokenSource _source = new CancellationTokenSource();
        private CancellationToken _token;

        public MultiThreadFileWriter()
        {
            _token = _source.Token;
            // Start the background task that will handle file writing
            Task.Run(WriteToFileAsync, _token);
        }

        /// <summary>
        /// Public method for threads to request writing a line.
        /// </summary>
        public void WriteLine(string line)
        {
            _textToWrite.Enqueue(line);
        }

        /// <summary>
        /// Background task that performs the file writing asynchronously.
        /// </summary>
        private async Task WriteToFileAsync()
        {
            using (StreamWriter writer = new StreamWriter("c:\\myfile.txt", append: true))
            {
                while (!_token.IsCancellationRequested)
                {
                    // Write all available lines in the queue
                    while (_textToWrite.TryDequeue(out string textLine))
                    {
                        await writer.WriteLineAsync(textLine);
                    }

                    // Asynchronously wait before trying again
                    if (_textToWrite.IsEmpty)
                    {
                        await Task.Delay(100, _token);  // Non-blocking delay
                    }
                }
            }
        }

        public void Stop()
        {
            _source.Cancel();
        }
    }
}

In this example:

• ConcurrentQueue: The queue is used to safely enqueue strings from multiple threads. Since ConcurrentQueue is thread-safe, no additional synchronization is necessary when adding data.
• Background Task: A Task.Run() is used to launch the WriteToFile method on a separate thread. This method continuously dequeues data from the ConcurrentQueue and writes it to a file.
• StreamWriter: A StreamWriter is used for appending to the file. The WriteLineAsync method ensures that the file write is done asynchronously.

To use the MultiThreadFileWriter class in your application, you can register it as a singleton in your Startup.cs or Program.cs (depending on the application type): 

// In Startup.cs or Program.cs
services.AddSingleton<MultiThreadFileWriter>();

Then, inject the MultiThreadFileWriter into any class that needs to write to the file:

public class MyService
{
    private readonly MultiThreadFileWriter _fileWriter;

    public MyService(MultiThreadFileWriter fileWriter)
    {
        _fileWriter = fileWriter;
    }

    public void AddDataToWrite(string data)
    {
        _fileWriter.WriteLine(data);
    }
}

The MultiThreadFileWriter class ensures that multiple threads can safely queue their data for writing without worrying about file access conflicts.