How to Deserialize JSON as a stream in C#

Using System.Text.Json

This is a modern and built-in library in .NET for JSON serialization and deserialization.

For example, Async Deserialization:

using System;
using System.IO;
using System.Text.Json;
using System.Threading.Tasks;

public class Employee
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int Experience { get; set; }
}

public async Task<Employee> DeserializeJsonAsync(string filePath)
{
    using var fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
    var employee = await JsonSerializer.DeserializeAsync<Employee>(fileStream);
    return employee;
}

For example, Sync Deserialization:

public Employee DeserializeJson(string filePath)
{
    using var fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
    var employee = JsonSerializer.Deserialize<Employee>(fileStream);
    return employee;
}

Using Newtonsoft.Json

This library is widely used and offers additional features for JSON manipulation.

For example, Deserialization

using System;
using System.IO;
using Newtonsoft.Json;

public class Employee
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int Experience { get; set; }
}

public Employee DeserializeJsonWithNewtonsoft(string filePath)
{
    using var fileReader = File.OpenText(filePath);
    using var jsonReader = new JsonTextReader(fileReader);
    var serializer = new JsonSerializer();
    var employee = serializer.Deserialize<Employee>(jsonReader);
    return employee;
}

Usage

public async Task Main()
{
    var employee = await DeserializeJsonAsync(@"D:\employees.json");
    Console.WriteLine($"ID: {employee.Id}, Name: {employee.Name}, Experience: {employee.Experience}");
}

Benefits of Stream Deserialization

1. Memory Efficiency: It uses less memory as it doesn’t need to load the entire JSON into memory as a string.
2. Fail Fast: It detects errors in the JSON data quickly.
3. Cancellation Support: With the async version, you can cancel long-running operations using a CancellationToken.

Performance

You can deserialize JSON in two primary ways:

1. Load it into a string and then deserialize.
2. Deserialize directly from a stream.

Deserializing from a stream consumes significantly less memory because it avoids the allocation of a large string object.

For instance, when testing with a 9 MB JSON file, the performance results were as follows:

The stream method demonstrated superior memory efficiency, which notably boosts overall performance.

Fail Fast

Deserializing as a stream allows for immediate error detection. Consider a scenario where your JSON file contains 100,000 records, and one of them has invalid data:

{
  "Id": 9999,
  "Name": "Alice",
  "Experience": 5
},
{
  "Id": 10000,
  "Name": "Bob",
  "Experience": "Invalid data"
},
{
  "Id": 10001,
  "Name": "Charlie",
  "Experience": 7
}

Attempting to deserialize will yield an exception at the point of invalid data, preventing further processing:

System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.Employees[10000].Experience

This method fails much earlier and uses significantly less memory, which is beneficial.

Can Be Canceled

The DeserializeAsync() method supports a CancellationToken, allowing you to cancel long-running operations.

For example, if you want to limit the deserialization to 10 ms:

using var fileStream = new FileStream(@"D:\employees.json", FileMode.Open, FileAccess.Read);

var timeoutAfter = TimeSpan.FromMilliseconds(10);
using var cancellationTokenSource = new CancellationTokenSource(timeoutAfter);

var employees = await JsonSerializer.DeserializeAsync<Employee>(fileStream,
    cancellationToken: cancellationTokenSource.Token);

After 10 ms, a TaskCanceledException will be thrown, which can be helpful in a UI context, enhancing user experience.

Get Objects as They Are Deserialized

When dealing with large JSON arrays and you don’t need to retain all objects in memory, consider using DeserializeAsyncEnumerable().

For example, if you have an array of Employee objects:

[
  {
    "Id": 0,
    "Name": "Alice",
    "Experience": 5
  },
  {
    "Id": 1,
    "Name": "Bob",
    "Experience": 2
  },
  ...
]

You can use DeserializeAsyncEnumerable() to process one object at a time:

using System.Text.Json;

using var fileStream = new FileStream(@"D:\employees.json", FileMode.Open, FileAccess.Read);

await foreach (var employee in JsonSerializer.DeserializeAsyncEnumerable<Employee>(fileStream))
{
    ProcessEmployee(employee);
}

This method is extremely memory-efficient, making it ideal for scenarios where you don't need to keep all deserialized objects in memory.