CsQuery - JQuery for .NET

I'm currently working on a personal use utility application which requires some web scraping to extract data (from HTML) which can be locally processed by the application. Getting the raw HTML was straightforward enough using the HttpWebRequest and HttpWebResponse classes in the System.Net namespace.

I then reached the point where I had some raw HTML in a string that I needed to parse. After doing a quick search I found CsQuery (available as a NuGet package) which is an open source JQuery port for .NET. I was able to easily extract the data I required from the HTML using the familiar JQuery-like selectors. There is an example code snippet below which shows just how easy it is to use CsQuery.

var html = new StringBuilder();
html.Append("<html><body>");
html.Append("<h1>Hello, world!</h1>");
html.Append("<p class='intro'>This program is using CsQuery.</p>");
html.Append("<p id='author'>CsQuery is a library written by James Treworgy.</p>");
html.Append("</body></html>");

var dom = CsQuery.CQ.Create(html.ToString());

// Get the inner text of an element by element name selector
Console.WriteLine(dom["h1"].Text());

// Get the inner text of an element by class name selector
Console.WriteLine(dom[".intro"].Text());

// Get the inner text of an element by id selector
Console.WriteLine(dom["#author"].Text());

// Add a class to an element
dom["h1"].AddClass("title");
            
// Update the title text using new class in selector
dom[".title"].Text("CSQuery - a JQuery port for .NET");

// Now retrieve the new title by a class selector
Console.WriteLine(dom[".title"].Text());

// Pause console
Console.ReadLine();

The example source code is available in a C# console application project on GitHub - https://github.com/rsingh85/CsQueryExample.

C# MongoDB Repository Implementation

I recently started a small project with a friend and we opted for MongoDB as our data store. One of the initial tasks was to write a repository class that allowed us to store, retrieve, update and search for entities in MongoDB. In the past, I've worked on codebases where there was one repository per entity, it didn't take long to discover that this resulted in a lot of code duplication across the data layer. So the goal here was to write the repository generically enough so that we only have one implementation that can be reused for any entity within our application. This meant that each entity would have its own MongoDB collection (if you're not familiar with MongoDB, a collection can be thought of as a table in the relational world - see this page for a comparison).

One of the first steps in this task was to write an interface for the repository - this is in case we decide to use a different data store in the future. A while back, I found a repository interface on the Redmondo blog which covers everything I'd want from a repository, so we'll use a slightly modified version of that interface (I've removed comments for brevity, you can download the code using a link further below and that'll contain the comments):

public interface IRepository<TEntity> where TEntity : EntityBase
{
    bool Insert(TEntity entity);
    bool Update(TEntity entity);
    bool Delete(TEntity entity);
    IList<TEntity>
        SearchFor(Expression<Func<TEntity, bool>> predicate);
    IList<TEntity> GetAll();
    TEntity GetById(Guid id);
}

To paraphrase the code, this is a generic repository interface for an entity of type TEntity that must derive from a base class called EntityBase. The EntityBase class is a very simple abstract class and contains just one property, the identifier property:

/// <summary>
/// A non-instantiable base entity which defines 
/// members available across all entities.
/// </summary>
public abstract class EntityBase
{
    public Guid Id { get; set; }
}

The idea is for any entity that we want to manage in our data store, that entity must derive from EntityBase. So the infrastructure is all in place for our "MongoDbRepository". I used the official 10gen MongoDB C# driver (available as a NuGet package) and arrived at the following repository implementation:

/// <summary>
/// A MongoDB repository. Maps to a collection with the same name
/// as type TEntity.
/// </summary>
/// <typeparam name="T">Entity type for this repository</typeparam>
public class MongoDbRepository<TEntity> :
    IRepository<TEntity> where
        TEntity : EntityBase
{
    private MongoDatabase database;
    private MongoCollection<TEntity> collection;

    public MongoDbRepository()
    {
        GetDatabase();
        GetCollection();
    }

    public bool Insert(TEntity entity)
    {
        entity.Id = Guid.NewGuid();
        return collection.Insert(entity).Ok;
    }

    public bool Update(TEntity entity)
    {
        if (entity.Id == null)
            return Insert(entity);
       
        return collection
            .Save(entity)
                .DocumentsAffected > 0;
    }

    public bool Delete(TEntity entity)
    {
        return collection
            .Remove(Query.EQ("_id", entity.Id))
                .DocumentsAffected > 0;
    }

    public IList<TEntity>
        SearchFor(Expression<Func<TEntity, bool>> predicate)
    {
        return collection
            .AsQueryable<TEntity>()
                .Where(predicate.Compile())
                    .ToList();
    }

    public IList<TEntity> GetAll()
    {
        return collection.FindAllAs<TEntity>().ToList();
    }

    public TEntity GetById(Guid id)
    {
        return collection.FindOneByIdAs<TEntity>(id);
    }

    #region Private Helper Methods
    private void GetDatabase()
    {
        var client = new MongoClient(GetConnectionString());
        var server = client.GetServer();

        database = server.GetDatabase(GetDatabaseName());
    }

    private string GetConnectionString()
    {
        return ConfigurationManager
            .AppSettings
                .Get("MongoDbConnectionString")
                    .Replace("{DB_NAME}", GetDatabaseName());
    }

    private string GetDatabaseName()
    {
        return ConfigurationManager
            .AppSettings
                .Get("MongoDbDatabaseName");
    }

    private void GetCollection()
    {
        collection = database
            .GetCollection<TEntity>(typeof(TEntity).Name);
    }
    #endregion
}

In case you're interested, a while ago I wrote a separate blog post on how to perform CRUD operations against MongoDB using the C# driver. To use the repository implementation, you'll need two application configuration settings defined - one that stores the name of the MongoDB database and the other that contains the MongoDB connection string (with a placeholder for the database name). You should have something like:

<appSettings>
  <add key="MongoDbDatabaseName" value="MyCarsDatabase" />
  <add key="MongoDbConnectionString"
         value="mongodb://localhost/{DB_NAME}?safe=true" />
</appSettings>

Hopefully this repository is useful to someone else who is working with MongoDB in C#. Any questions or suggestions for improvements are always welcome via comments. The source code is available on GitHub - https://github.com/rsingh85/MongoDbRepository.

C# Set Implementation (pre-.NET 3.5)

I was recently working on a .NET 3.0 project and found that this version of the framework didn't have any support for a mathematical Set collection. As of .NET 3.5 there is the System.Collections.Generic.HashSet implementation, however, I was working on a WinForms project and updating the framework on the client machines was out of scope for this piece of work.

I had some spare time available and thought it would be a good exercise to write a custom generic Set collection. The first step was to define an interface:

using System.Collections.Generic;

...

public interface ISet<T> : ICollection<T>
{
    void UnionWith(ISet<T> set);
    void IntersectWith(ISet<T> set);
}

By inheriting from ICollection, we get all of the generally useful collection-based methods included in the ISet interface (e.g. Add, Clear, Contains etc.). I just had two methods to add to those, these were the set union and set intersection operations.

The next step was the implementation of the set. To keep things simple, I opted for a list-based implementation. I used the adapter pattern to map the ISet interface to a list interface. Therefore, the internal representation of the Set was actually an instance of System.Collections.Generic.List. By using this approach, I could delegate a lot of the method logic to use the list method implementations (i.e. Clear, Contains, CopyTo and Remove). The only custom logic I needed to write was the Add method (to avoid adding items already within the set), UnionWith method, IntersectWith method, ToString method and the readonly indexer. The implementation is below:

/// <summary>
/// A list-based implementation of a mathematical set.
/// </summary>
public sealed class ListBasedSet<T> : ISet<T>
{
    private IList<T> _thisSet = new List<T>();

    /// <summary>
    /// Gets the number of elements contained in this set.
    /// </summary>
    public int Count { get { return _thisSet.Count; } }

    /// <summary>
    /// Gets a value indicating whether this set is readonly.
    /// </summary>
    public bool IsReadOnly { get { return false; } }

    /// <summary>
    /// Gets the item at the specified index in the set.
    /// </summary>
    /// <param name="i">
    /// Index of the item to retrieve from the set.
    /// </param>
    /// <returns>Item at the specified index</returns>
    public T this[int i] { get { return _thisSet[i]; } }

    /// <summary>
    /// Adds item into the set if this set does not already
    /// contain the item.
    /// </summary>
    /// <param name="item">Item to add to the set.</param>
    public void Add(T item)
    {
        if (!Contains(item))
            _thisSet.Add(item);
    }

    /// <summary>
    /// Clears the items from the set resulting in an empty set.
    /// </summary>
    public void Clear()
    {
        _thisSet.Clear();
    }

    /// <summary>
    /// Determines if the specified item is within the set.
    /// </summary>
    /// <param name="item">
    /// The object to locate in the set.
    /// </param>
    /// <returns>
    /// true if item is found in the set; false otherwise.
    /// </returns>
    public bool Contains(T item)
    {
        return _thisSet.Contains(item);
    }

    /// <summary>
    /// Copies the entire set to a one-dimensional array, 
    /// starting at the specified target index of the target 
    /// array.
    /// </summary>
    /// <param name="array">
    /// The one-dimensional array to copy to.
    /// </param>
    /// <param name="arrayIndex">
    /// The zero-based index in array at which copying begins.
    /// </param>
    public void CopyTo(T[] array, int arrayIndex)
    {
        _thisSet.CopyTo(array, arrayIndex);
    }

    /// <summary>
    /// Removes the specified item from the set.
    /// </summary>
    /// <param name="item">Item to remove.</param>
    /// <returns>
    /// true if item is successfully removed; otherwise, false. 
    /// This method also returns false if item was not found 
    /// in the set.
    /// </returns>
    public bool Remove(T item)
    {
        return _thisSet.Remove(item);
    }

    /// <summary>
    /// Returns an enumerator that iterates through the set.
    /// </summary>
    /// <returns>
    /// An IEnumerator object that can be used to iterate 
    /// through the set.
    /// </returns>
    public IEnumerator<T> GetEnumerator()
    {
        return _thisSet.GetEnumerator();
    }

    /// <summary>
    /// Returns an enumerator that iterates through the set.
    /// </summary>
    /// <returns>
    /// An IEnumerator object that can be used to iterate 
    /// through the set.
    /// </returns>
    System.Collections.IEnumerator 
        System.Collections.IEnumerable.GetEnumerator()
    {
        return _thisSet.GetEnumerator();
    }

    /// <summary>
    /// Modifies this set to contain all elements that are 
    /// present in itself, the specified collection, or both.
    /// </summary>
    /// <param name="otherSet">The set to union with.</param>
    public void UnionWith(ISet<T> otherSet)
    {
        foreach (var item in otherSet)
            Add(item);
    }

    /// <summary>
    /// Modifies this set to contain only the elements that are 
    /// present in it and in the specified collection.
    /// </summary>
    /// <param name="otherSet">The set to intersect with.</param>
    public void IntersectWith(ISet<T> otherSet)
    {
        var itemsToRemove = new List<T>();

        for (int i  = 0; i < Count; i++)
        {
            T item = this[i];

            if (!otherSet.Contains(item))
                itemsToRemove.Add(item);
        }

        foreach (var itemToRemove in itemsToRemove)
            Remove(itemToRemove);
    }

    /// <summary>
    /// Returns a string representation of this set
    /// </summary>
    /// <returns>A string representation of this set</returns>
    public override string ToString()
    {
        var setStringBuilder = new StringBuilder();
        setStringBuilder.Append("{");

        foreach (var item in _thisSet)
            setStringBuilder.Append(
                string.Format(" {0},", item));
            
        return string.Format("{0} }}", 
            setStringBuilder.ToString().TrimEnd(','));
    }
}

Example usage:

ISet<string> set = new ListBasedSet<string> 
    { "Mars", "Earth", "Pluto" };
ISet<string> otherSet = new ListBasedSet<string> 
    { "Jupiter", "Earth", "Pluto", "Venus" };
ISet<string> earthSet = new ListBasedSet<string> 
    { "Earth" };
ISet<string> emptySet = new ListBasedSet<string>();

set.UnionWith(otherSet);
Console.WriteLine("set UNION otherset = {0}", set);

Console.WriteLine("|set| (cardinality of set) = {0}", set.Count);

set.IntersectWith(earthSet);
Console.WriteLine("set INTERSECTION earthSet = {0}", set);

set.UnionWith(emptySet);
Console.WriteLine("set UNION emptySet = {0}", set);

set.IntersectWith(emptySet);
Console.WriteLine("set INTERSECTION emptySet = {0}", set);

You can download the source files for the ISet interface and the ListBasedSet class from here.

NDataFlow - Open Source .NET Dataflow Library

Last year, a colleague of mine showed me an open source .NET extract, transform and load (ETL) helper library that he's working on. It is called NDataFlow and allows you to annotate methods with attributes to create a dataflow in your application. It's a nice lightweight library that you can use whenever you are developing simple or complex ETL programs. The example below simulates a very simple ETL scenario where a set of people (hard-coded in the example) are filtered based on their location and then output to a file.

class Program : DataflowComponent
{
  static void Main(string[] args)
  {
    new Program().Run();
  }

  //First method in the flow
  [Operation]
  public void DataSource(Output output)
  {
    //Imagine retrieving people from a real data source
    //e.g. database, xml file, csv file, etc.
    output.Emit(new Person() { Name = "Alice", City = "London" });
    output.Emit(new Person() { Name = "Bob", City = "New York" });
    output.Emit(new Person() { Name = "Foo", City = "London" });
    output.Emit(new Person() { Name = "Bar", City = "Sydney" });
  }

  [Operation]
  public IEnumerable FilterForLondonPeople
    ([Link("DataSource")] IEnumerable input)
  {
    return input.Where
      (p => p.City.Equals("London", 
        StringComparison.InvariantCultureIgnoreCase));
  }

  [Operation]
  public void OutputResults
    ([Link("FilterForLondonPeople")] IEnumerable results)
  {
    using (var sw = new StreamWriter(@"C:\LondonPeople.txt", false)
    {
      foreach (var p in results)
        sw.WriteLine("{0} {1}", p.Name, p.City);
    }
  }
}

The example shows that there is little work needed to get a simple dataflow setup and running. You inherit the Run method by deriving from the NDataFlow.DataflowComponent class. Then, if you've setup your method and attributes correctly using the LinkAttribute it's a simple case of calling Run to start your dataflow. In this case, the first method in the dataflow would be DataSource, whose output is sent to FilterForLondonPeople and finally whose output is sent to the OutputResults method.