How to Remove Duplicates from a List with LINQ in C#

The Distinct() method works well for simple types, but for more complex objects, you'll need to use a custom equality comparison. 

Removing Duplicates from Simple Types

Here’s an example using a string with non-unique elements. The Distinct() method will help us easily eliminate duplicates.

// A string with non-unique elements
string s = "a|b|c|d|a";

// Split the string and take the distinct elements
var distinctList = s.Split('|').Distinct().ToArray();

// Re-join the list into a string
var distinctString = string.Join("|", distinctList);

// Output: "a|b|c|d"
Console.WriteLine(distinctString);

In this case, the string is split by a delimiter (|), and LINQ’s Distinct() is used to get the unique values. Finally, the distinct values are joined back together with string.Join().

Removing Duplicates from Complex Types

For more complex data types like classes, Distinct() doesn’t automatically work because it compares object references, not the properties. You have two main options for removing duplicates in such cases: using an IEqualityComparer or grouping by a property.

Implement an IEqualityComparer

To remove duplicates from a list of complex objects, we can create a custom equality comparer that compares the Title property of a class, for example.

public class MyClass
{
    public string Title { get; set; }
    public string Text { get; set; }
}

Next, create the custom IEqualityComparer:

public class MyClassDistinctComparer : IEqualityComparer<MyClass>
{
    public bool Equals(MyClass x, MyClass y) 
    {
        return x.Title == y.Title; // Compare by Title
    }

    public int GetHashCode(MyClass obj) 
    {
        return obj.Title.GetHashCode(); // Hash code based on Title
    }       
}

Now, use this comparer with LINQ’s Distinct():

// Create a list of non-unique titles
List<MyClass> list = new List<MyClass>();
list.Add(new MyClass() { Title = "A", Text = "Text" });
list.Add(new MyClass() { Title = "B", Text = "Text" });
list.Add(new MyClass() { Title = "A", Text = "Text" });

// Get the distinct elements using the custom comparer
var distinctList = list.Distinct(new MyClassDistinctComparer()).ToList();

// Output the unique titles
foreach (var myClass in distinctList)
    Console.WriteLine(myClass.Title); // Output: "A B"

This approach works well when you need fine-grained control over how duplicates are determined, based on one or more properties.

Group by Property and Take First

If you want a simpler solution and don’t want to implement a custom comparer, you can group the elements by a property (e.g., Title) and then select the first item from each group.

// Create a list of non-unique titles
List<MyClass> list = new List<MyClass>();
list.Add(new MyClass() { Title = "A", Text = "Text" });
list.Add(new MyClass() { Title = "B", Text = "Text" });
list.Add(new MyClass() { Title = "A", Text = "Text" });

// Group by title and take the first element of each group
var distinctList = list.GroupBy(s => s.Title)
                       .Select(s => s.First())
                       .ToArray();

// Output the unique titles
foreach (var myClass in distinctList)
    Console.WriteLine(myClass.Title); // Output: "A B"

In this case, GroupBy(s => s.Title) groups the list by the Title property. Then, Select(s => s.First()) selects the first element from each group, effectively removing duplicates.

Both methods are efficient and flexible, making it easy to handle duplicates in your C# collections.