Writing Files Asynchronously with Multiple Threads in .NET Core
By FoxLearn 1/10/2025 8:29:52 AM 287
You can use ReaderWriterLock, semaphores, and the lock statement to control access to shared resources.
Multiple threads send their data to a ConcurrentQueue, which is a thread-safe collection designed for scenarios like this. A background task continuously monitors the queue and performs the actual file writing, ensuring that only one thread accesses the file at any given time.
using System.Collections.Concurrent; using System.IO; using System.Threading; using System.Threading.Tasks; namespace MyApp { public class MultiThreadFileWriter { private static ConcurrentQueue<string> _textToWrite = new ConcurrentQueue<string>(); private CancellationTokenSource _source = new CancellationTokenSource(); private CancellationToken _token; public MultiThreadFileWriter() { _token = _source.Token; // Start the background task that will handle file writing Task.Run(WriteToFileAsync, _token); } /// <summary> /// Public method for threads to request writing a line. /// </summary> public void WriteLine(string line) { _textToWrite.Enqueue(line); } /// <summary> /// Background task that performs the file writing asynchronously. /// </summary> private async Task WriteToFileAsync() { using (StreamWriter writer = new StreamWriter("c:\\myfile.txt", append: true)) { while (!_token.IsCancellationRequested) { // Write all available lines in the queue while (_textToWrite.TryDequeue(out string textLine)) { await writer.WriteLineAsync(textLine); } // Asynchronously wait before trying again if (_textToWrite.IsEmpty) { await Task.Delay(100, _token); // Non-blocking delay } } } } public void Stop() { _source.Cancel(); } } }
In this example:
- ConcurrentQueue: The queue is used to safely enqueue strings from multiple threads. Since
ConcurrentQueue
is thread-safe, no additional synchronization is necessary when adding data. - Background Task: A
Task.Run()
is used to launch theWriteToFile
method on a separate thread. This method continuously dequeues data from theConcurrentQueue
and writes it to a file. - StreamWriter: A
StreamWriter
is used for appending to the file. TheWriteLineAsync
method ensures that the file write is done asynchronously.
To use the MultiThreadFileWriter
class in your application, you can register it as a singleton in your Startup.cs
or Program.cs
(depending on the application type):
// In Startup.cs or Program.cs services.AddSingleton<MultiThreadFileWriter>();
Then, inject the MultiThreadFileWriter
into any class that needs to write to the file:
public class MyService { private readonly MultiThreadFileWriter _fileWriter; public MyService(MultiThreadFileWriter fileWriter) { _fileWriter = fileWriter; } public void AddDataToWrite(string data) { _fileWriter.WriteLine(data); } }
The MultiThreadFileWriter
class ensures that multiple threads can safely queue their data for writing without worrying about file access conflicts.
- How to use JsonConverterFactory in C#
- How to serialize non-public properties using System.Text.Json
- The JSON value could not be converted to System.DateTime
- Try/finally with no catch block in C#
- Parsing a DateTime from a string in C#
- Async/Await with a Func delegate in C#
- How to batch read with Threading.ChannelReader in C#
- How to ignore JSON deserialization errors in C#