Roundtrip Serialization of DateTimes in C#

Say you need to serialize a DateTime object to string (perhaps for an XML or JSON file), and want to ensure that you get back exactly the same time you serialized. Well, you might think that you can just use DateTime.ToString and DateTime.Parse:

var testDate = DateTime.Now;
var serialized = testDate.ToString();
Console.WriteLine("Serialized: {0}",serialized);
var deserialized = DateTime.Parse(serialized);
Console.WriteLine("Deserialized: {0}",deserialized);

// prints:
Serialized: 14/04/2014 16:32:00
Deserialized: 14/04/2014 16:32:00

At first glance it seems to work, but actually this is a very bad idea. What’s wrong with this approach? Well two things. For starters, the millisecond component of the time has been lost. More seriously, you are entirely dependent on the deserializing system’s short date format being the same as the serializing system’s. For example, if you serialize in a UK culture with a short date format of dd/mm/yyyy and then deserialize in a US culture with a short date format of mm/dd/yyyy you can end up with completely the wrong date (e.g. 1st of March becomes the 3rd of January).

So how can you ensure you get exactly the same DateTime object back that you serialized? Well the answer is to use ToString with the “O” format specifier which is called the “roundtrip format specifier”. This will not only include the milliseconds, but also ensures that the DateTime.Kind property is represented in the string (under the hood a .NET DateTime is essentially a tick count plus a DateTimeKind).

Here’s three dates formatted with the roundtrip format specifier, but each with a different DateTime.Kind value:

Console.WriteLine("Local: {0}", DateTime.Now.ToString("O"));
Console.WriteLine("UTC: {0}", DateTime.UtcNow.ToString("O"));
var unspecified = new DateTime(2014,3,11,16,13,56,123,DateTimeKind.Unspecified);
Console.WriteLine("Unspecified: {0}", unspecified.ToString("O"));

// prints:
Local: 2014-04-14T16:36:01.5305961+01:00
UTC: 2014-04-14T15:36:01.5345961Z
Unspecified: 2014-03-11T16:13:56.1230000

As you can see, UTC times end in Z, local times include the current offset from UTC, and unspecified times have no suffix. You can also see that we have fractions of a second to 7 decimal places. If you’ve serialized your DateTime objects like this, you can then get them back to exactly what they were before by using DateTime.Parse, and passing in the DateTimeStyles.RoundTripKind argument:

DateTime.Parse("2014-03-11T16:14:27.8660648+01:00", CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);

The only thing to note here is that the serialization of a local time includes the offset from UTC. Which means that if you deserialize it in a different timezone, you will get the correct local time for that timezone. Which will of course be a different local time to the one originally serialized This is almost certainly what you want, but it does mean that technically, the binary representation of the deserialized DateTime object is not identical to the one that was serialized (different number of ticks).

Python Equivalents of LINQ Methods

In my last post, I looked at how Python’s list comprehensions and generators allow you to achieve many of the same tasks that you would use LINQ for in C#. In this post, we’ll look at Python equivalents for some of the most popular LINQ extension methods. We’ll mostly be looking at Python’s built-in functions and itertools module.

For these examples, our test data will be a list of fruit. But all of these techniques work with any interable, including the output of generator functions. Here’s our Python test data

fruit = ['apple', 'orange', 'banana', 'pear', 
         'raspberry', 'peach', 'plum']

Which of course in C# is

var fruit = new List<string>() { "apple", "orange",
 "banana", "pear", "raspberry", "peach", "plum" };

Any & All

LINQ’s Any method allows you to test whether any of the items in a sequence fulfil a certain requirement, while All checks if all of them do. Python’s built-in functions are named the same, so it’s really straightforward. Let’s see if any of our fruit contain the letter “e”, then see if all of them do:

>>> any("e" in f for f in fruit)
True
>>> all("e" in f for f in fruit)
False

in LINQ:

fruit.Any(f => f.Contains("e"));
fruit.All(f => f.Contains("e"));

Min & Max

Again, Python has built-in functions similarly named to LINQ. Let’s find the minimum and maximum fruit lengths:

>>> max(len(f) for f in fruit)
9
>>> min(len(f) for f in fruit)
4

which are the equivalents of:

fruit.Max(f => f.Length);
fruit.Min(f => f.Length);

Take, Skip, TakeWhile & SkipWhile

LINQ’s Take and Skip methods are very useful for paging data, or limiting the amount you process, and TakeWhile and SkipWhile come in handy from time to time as well (TakeWhile can be a good way of checking for user cancellation).

Take and Skip can be implemented using the itertools islice function. We can specify an end index, or a start and end index. If the end index is None, that means keep going to the end of the iterable. I’d prefer methods actually called “skip” and “take” as I think that makes for more readable code, but they could be easily created if needed.

Here’s Take(2) and Skip(2) implemented with Python. Since islice returns a generator function, I turn it into a list for debugging purposes:

>>> from itertools import islice
>>> list(islice(fruit, 2))
['apple', 'orange']
>>> list(islice(fruit, 2, None))
['banana', 'pear', 'raspberry', 'peach', 'plum']

islice does have the benefit though of letting you combine a skip and a take into one step rather than chaining them like you would in C#:

fruit.Skip(2).Take(2);

with islice:

>>> list(islice(fruit, 2, 4))
['banana', 'pear']

The itertools module does include a “takewhile” method and for LINQ’s SkipWhile, it’s “dropwhile”. With these functions, you might want to use Python’s lambda syntax, which is a rare example of where the Python is less succinct than C#.

>>> from itertools import takewhile
>>> list(takewhile(lambda c: len(c) < 7, fruit))
['apple', 'orange', 'banana', 'pear']
>>> from itertools import dropwhile
>>> list(dropwhile(lambda c: len(c) < 7, fruit))
['raspberry', 'peach', 'plum']

Here’s the same TakeWhile and SkipWhile in C#:

fruit.TakeWhile (f => f.Length < 7);
fruit.SkipWhile (f => f.Length < 7);

First, FirstOrDefault, & Last

With LINQ you can easily get the first item from an IEnumerable. This throws an exception if the sequence is empty, so FirstOrDefault can be used alternatively. With Python, the “next” method can be used on an iterable (but not on a list). Let’s use Python to get the first fruit starting with “p” and to return a default value when our generator looking for the first fruit starting with “q” doesn’t find any elements.

>>> next(f for f in fruit if f.startswith("p"))
'pear'
>>> next((f for f in fruit if f.startswith("q")), "none")
'none'

There does not seem to be any built-in Python function to implement LINQ’s “Last” or “LastOrDefault” methods, but you could quite easily create one. Here’s a fairly rudimentary one:

>>> def lastOrDefault(sequence, default=None):
...     lastItem = default
...     for s in sequence:
...         lastItem = s
...     return lastItem
...
>>> lastOrDefault((f for f in fruit if f.endswith("e")))
'orange'
>>> lastOrDefault((f for f in fruit if f.startswith("x")), "no fruit found")
'no fruit found'

You could do the same if you really needed the LINQ “Single” or “SingleOrDefault” methods, which also have no direct equivalent.

Count

The LINQ Count extension method lets you count how many items are in a sequence. For example, how many fruit begin with ”p”?

fruit.Count(f => f.StartsWith("p"))

Probably the most logical expectation would be that Python’s “len” function would do the same, but you can’t call len on an iterable. There is a neat trick though you can use with the “sum” built-in function.

>>> sum(1 for f in fruit if f.startswith("p"))
3

Select & Where

We saw in the last blog post that a list comprehension already includes the capabilities of LINQ’s Select and Where, but there may be times you want to them to be available as functions. Python’s “map” and “filter” function take an iterable and a lamba and return an iterator (this is Python 3 only – in Python 2 they returned lists). Here’s a couple of simple examples of them in action, with the output turned into a list for debug purposes:

>>> list(map(lambda x: x.upper(), fruit))
['APPLE', 'ORANGE', 'BANANA', 'PEAR', 'RASPBERRY', 'PEACH', 'PLUM']
>>> list(filter(lambda x: "n" in x, fruit))
['orange', 'banana']

 

GroupBy

At first glance it might appear that itertools groupby method behaves the same as LINQ’s GroupBy, but there is a gotcha. Python’s groupby expects the incoming data to be sorted by the key, so you have to call sorted first. This example shows us first trying to group without sorting (resulting in two “p” groups), and then doing it the right way. We’re grouping by first letter of the fruit, and I’m using a helper method to print out the contents of the grouped data:

>>> def printGroupedData(groupedData):
...     for k, v in groupedData:
...         print("Group {} {}".format(k, list(v)))
...
>>> from itertools import groupby
>>> keyFunc = lambda f: f[0]
>>> printGroupedData(groupby(fruit, keyFunc))
Group a ['apple']
Group o ['orange']
Group b ['banana']
Group p ['pear']
Group r ['raspberry']
Group p ['peach', 'plum']
>>> sortedFruit = sorted(fruit, key=keyFunc)
>>> printGroupedData(groupby(sortedFruit, keyFunc))
Group a ['apple']
Group b ['banana']
Group o ['orange']
Group p ['pear', 'peach', 'plum']
Group r ['raspberry']

OrderBy

As we saw above, the “sorted” built-in function in Python can be used to order a sequence. It returns a list, but this is understandable since to implement OrderBy it must iterate through the entire sequence first. Here we sort the fruit by their string length:

>>> sorted(fruit, key=lambda x:len(x))
['pear', 'plum', 'apple', 'peach', 'orange', 'banana', 'raspberry']

Distinct

As far as I can tell there isn’t a built-in function in Python to emit a distinct iterable sequence, but the easiest way is probably to just construct a set. If you wanted to create a generator function, allowing you to abort early before reaching the end of a sequence, you could create your own helper method:

def distinct(sequence):
    seen = set()
    for s in sequence:
        if not s in seen:
            seen.add(s)
            yield s

Zip

The last example I’ll look at is the Zip method. In Python there is an equivalent zip function, and it is actually a little simpler as it assumes you want a tuple, rather than LINQ’s where you need to explicitly create a result selector function. It actually supports zipping more than two sequences together which is nice. As with LINQ’s Zip, the resulting sequence is the length of the shortest. Here’s a quick example of the Python zip function in action:

>>> recipes = ['pie','juice','milkshake']
>>> list(zip(fruit,recipes))
[('apple', 'pie'), ('orange', 'juice'), ('banana', 'milkshake')]
>>> list(f + " " + r for f,r in zip(fruit,recipes))
['apple pie', 'orange juice', 'banana milkshake']

Conclusion

As can be seen, most of the main LINQ extension methods have fairly close Python equivalents, and those that don’t could be quite easily recreated. I don’t pretend to be an expert on Python, so if I’ve missed any cool tricks, let me know in the comments.

Understanding and Avoiding Memory Leaks with Event Handlers and Event Aggregators

If you subscribe to an event in C# and forget to unsubscribe, does it cause a memory leak? Always? Never? Or only in special circumstances? Maybe we should make it our practice to always unsubscribe just in case there is a problem. But then again, the Visual Studio designer generated code doesn’t bother to unsubscribe, so surely that means it doesn’t matter?.

updater.Finished += new EventHandler(OnUpdaterFinished);
updater.Begin();

...

// is this important? do we have to unsubscribe?
updater.Finished -= new EventHandler(OnUpdaterFinished);

Fortunately it is quite easy to see for ourselves whether any memory is leaked when we forget to unsubscribe. Let’s create a simple Windows Forms application that creates lots of objects, and subscribe to an event on each of the objects, without bothering to unsubscribe. To make life easier for ourselves, we’ll keep count of how many get created, and how many get deleted by the garbage collector, by reducing a count in their finalizer, which the garbage collector will call.

Here’s the object we’ll be creating lots of instances of:

public class ShortLivedEventRaiser
{
    public static int Count;
    
    public event EventHandler OnSomething;

    public ShortLivedEventRaiser()
    {
        Interlocked.Increment(ref Count);
    }

    protected void RaiseOnSomething(EventArgs e)
    {
        EventHandler handler = OnSomething;
        if (handler != null) handler(this, e);
    }

    ~ShortLivedEventRaiser()
    {
        Interlocked.Decrement(ref Count);
    }
}

and here’s the code we’ll use to test it:

private void OnSubscribeToShortlivedObjectsClick(object sender, EventArgs e)
{
    int count = 10000;
    for (int n = 0; n < count; n++)
    {
        var shortlived = new ShortLivedEventRaiser();
        shortlived.OnSomething += ShortlivedOnOnSomething;
    }
    shortlivedEventRaiserCreated += count;
}

private void ShortlivedOnOnSomething(object sender, EventArgs eventArgs)
{
    // just to prove that there is no smoke and mirrors, our event handler will do something involving the form
    Text = "Got an event from a short-lived event raiser";
}

I’ve added a background timer on the form, which reports every second how many instances are still in memory. I also added a garbage collect button, to force the garbage collector to do a full collect on demand.

So we click our button a few times to create 80,000 objects, and quite soon after we see the garbage collector run and reduce the object count. It doesn’t delete all of them, but this is not because we have a memory leak. It is simply that the garbage collector doesn’t always do a full collection. If we press our garbage collect button, we’ll see that the number of objects we created drops down to 0. So no memory leaks! We didn’t unsubscribe and there was nothing to worry about.

But let’s try something different. Instead of subscribing to an event on my 80,000 objects, I’ll let them subscribe to an event on my Form. Now when we click our button eight times to create 80,000 of these objects, we see that the number in memory stays at 80,000. We can click the Garbage Collect button as many times as we want, and the number won’t go down. We’ve got a memory leak!

Here’s the second class:

public class ShortLivedEventSubscriber
{
    public static int Count;

    public string LatestText { get; private set; }

    public ShortLivedEventSubscriber(Control c)
    {
        Interlocked.Increment(ref Count);
        c.TextChanged += OnTextChanged;
    }

    private void OnTextChanged(object sender, EventArgs eventArgs)
    {
        LatestText = ((Control) sender).Text;
    }

    ~ShortLivedEventSubscriber()
    {
        Interlocked.Decrement(ref Count);
    }
}

and the code that creates instances of it:

private void OnShortlivedEventSubscribersClick(object sender, EventArgs e)
{
    int count = 10000;
    for (int n = 0; n < count; n++)
    {
        var shortlived2 = new ShortLivedEventSubscriber(this);
    }
    shortlivedEventSubscriberCreated += count;
}

and here’s the result:

So why does this leak, when the first doesn’t? The answer is that event publishers keep their subscribers alive. If the publisher is short-lived compared to the subscriber, this doesn’t matter. But if the publisher lives on for the life-time of the application, then every subscriber will also be kept alive. In our first example, the 80,000 objects were the publishers, and they were keeping the main form alive. But it didn’t matter because our main form was supposed to be still alive. But in the second example, the main form was the publisher, and it kept all 80,000 of its subscribers alive, long after we stopped caring about them.

The reason for this is that under the hood, the .NET events model is simply an implementation of the observer pattern. In the observer pattern, anyone who wants to “observe” an event registers with the class that raises the event. It keeps hold of a list of observers, allowing it to call each one in turn when the event occurs. So the observed class holds references to all its observers.

What does this mean?

The good news is that in a lot of cases, you are subscribing to an event raised by an object whose lifetime is equal or shorter than that of the subscribing class. That’s why a Windows Forms or WPF control can subscribe to events raised by child controls without the need to unsubscribe, since those child controls will not live beyond the lifetime of their container.

When it goes wrong is when you have a class that will exist for the lifetime of your application, raising events whose subscribers were supposed to be transitory. Imagine your application has a order service which allows you to submit new orders and also has an event that is raised whenever an order’s status changes.

orderService.SubmitOrder(order);
// get notified if an order status is changed
orderService.OrderStatusChanged += OnOrderStatusChanged;

Now this could well cause a memory leak, as whatever class contains the OnOrderStatusChanged event handler will be kept alive for the duration of the application run. And it will also keep alive any objects it holds references to, resulting in a potentially large memory leak. This means that if you subscribe to an event raised by a long-lived service, you must remember to unsubscribe.

What about Event Aggregators?

Event aggregators offer an alternative to traditional C# events, with the additional benefit of completely decoupling the publisher and subscribers of events. Anyone who can access the event aggregator can publish an event onto it, and it can be subscribed to from anyone else with access to the event aggregator.

But are event aggregators subject to memory leaks? Do they leak in the same way that regular event handlers do, or do the rules change? We can test this out for ourselves, using the same approach as before.

For this example, I’ll be using an extremely elegant event aggregator built by José Romaniello using Reactive Extensions. The whole thing is implemented in about a dozen of code thanks to the power of the Rx framework.

First, we’ll simulate many short-lived publishers with a single long-lived subscriber (our main form). Here’s our short-lived publisher object:

public class ShortLivedEventPublisher
{
    public static int Count;
    private readonly IEventPublisher publisher;

    public ShortLivedEventPublisher(IEventPublisher publisher)
    {
        this.publisher = publisher;
        Interlocked.Increment(ref Count);
    }

    public void PublishSomething()
    {
        publisher.Publish("Hello world");
    }

    ~ShortLivedEventPublisher()
    {
        Interlocked.Decrement(ref Count);
    }
}

And we’ll also try many short-lived subscribers with a single long-lived publisher (our main form):

public class ShortLivedEventBusSubscriber
{
    public static int Count;
    public string LatestMessage { get; private set; }

    public ShortLivedEventBusSubscriber(IEventPublisher publisher)
    {
        Interlocked.Increment(ref Count);
        publisher.GetEvent<string>().Subscribe(s => LatestMessage = s);
    }

    ~ShortLivedEventBusSubscriber()
    {
        Interlocked.Decrement(ref Count);
    }
}

What happens when we create thousands of each of these objects?

We have exactly the same memory leak again – publishers can be garbage collected, but subscribers are kept alive. Using an event aggregator hasn’t made the problem any better or worse. Event aggregators should be chosen for the architectural benefits they offer rather than as a way to fix your memory management problems (although as we shall see shortly, they encapsulate one possible fix).

How can I avoid memory leaks?

So how can we write event-driven code in a way that will never leak memory? There are two main approaches you can take.

1. Always remember to unsubscribe if you are a short-lived object subscribing to an event from a long-lived object. The C# language support for events is less than ideal. The C# language offers the += and -= operators for subscribing and unsubscribing, but this can be quite confusing.Here’s how you would unsubscribe from a button click handler…

button.Clicked += new EventHandler(OnButtonClicked)
...
button.Clicked –= new EventHandler(OnButtonClicked)

It’s confusing because the object we unsubscribe with is clearly a different object to the one we subscribed with, but under the hood .NET works out the right thing to do. But if you are using the lambda syntax, it is a lot less clear what goes on the right hand side of the –= (see this stack overflow question for more info). You don’t exactly want to keep trying to replicate the same lambda statement in two places.

button.Clicked += (sender, args) => MessageBox.Show(“Button was clicked”);
// how to unsubscribe?

This is where event aggregators can offer a slightly nicer experience. They will typically have an “unregister” or an “unsubscribe” method. The Rx version I used above returns an IDisposable object when you call subscribe. I like this approach as it means you can either use it in a using block, or store the returned value as a class member, and make your class Disposable too, implementing the standard .NET practice for resource cleanup and flagging up to users of your class that it needs to be disposed.

2. Use weak references. But what if you don’t trust yourself, or your fellow developers to always remember to unsubscribe? Is there another solution? The answer is yes, you can use weak references. A weak reference holds a reference to a .NET object, but allows the garbage collector to delete it if there are no other regular references to it.

The trouble is, how do you attach a weak event handler to a regular .NET event? The answer is, with great difficulty, although some clever people have come up with ingenious ways of doing this. Event aggregators have an advantage here in that they can offer weak references as a feature if wanted, hiding the complexity of working with weak references from the end user. For example, the “Messenger” class that comes with MVVM Light uses weak references.

So for my final test, I’ll make an event aggregator that uses weak references. I could try to update the Rx version, but to keep things simple, I’ll just make my own basic (and not threadsafe) event aggregator using weak references. Here’s the code:

public class WeakEventAggregator
{
    class WeakAction
    {
        private WeakReference weakReference;
        public WeakAction(object action)
        {
            weakReference = new WeakReference(action);
        }

        public bool IsAlive
        {
            get { return weakReference.IsAlive; }
        }

        public void Execute<TEvent>(TEvent param)
        {
            var action = (Action<TEvent>) weakReference.Target;
            action.Invoke(param);
        }
    }

    private readonly ConcurrentDictionary<Type, List<WeakAction>> subscriptions
        = new ConcurrentDictionary<Type, List<WeakAction>>();

    public void Subscribe<TEvent>(Action<TEvent> action)
    {
        var subscribers = subscriptions.GetOrAdd(typeof (TEvent), t => new List<WeakAction>());
        subscribers.Add(new WeakAction(action));
    }

    public void Publish<TEvent>(TEvent sampleEvent)
    {
        List<WeakAction> subscribers;
        if (subscriptions.TryGetValue(typeof(TEvent), out subscribers))
        {
            subscribers.RemoveAll(x => !x.IsAlive);
            subscribers.ForEach(x => x.Execute<TEvent>(sampleEvent));
        }
    }
}

Now let’s see if it works by creating some short-lived subscribers that subscribe to events on the WeakEventAggregator. Here are the objects, we’ll be using in this last example:

public class ShortLivedWeakEventSubscriber
{
    public static int Count;
    public string LatestMessage { get; private set; }

    public ShortLivedWeakEventSubscriber(WeakEventAggregator weakEventAggregator)
    {
        Interlocked.Increment(ref Count);
        weakEventAggregator.Subscribe<string>(OnMessageReceived);
    }

    private void OnMessageReceived(string s)
    {
        LatestMessage = s;
    }

    ~ShortLivedWeakEventSubscriber()
    {
        Interlocked.Decrement(ref Count);
    }
}

And we create another 80,000, do a garbage collect, and finally we can have event subscribers that don’t leak memory:

My example application is available for download on BitBucket if you want

Conclusion

Although many (possibly most) use cases of events do not leak memory, it is important for all .NET developers to understand the circumstances in which they might leak memory. I’m not sure there is a single “best practice” for avoiding memory leaks. In many cases, simply remembering to unsubscribe when you are finished wanting to receive messages is the right thing to do. But if you are using an event aggregator you’ll be able to take advantage of the benefits of weak references quite easily.

Screen-Scraping in C# using LINQPad and HTML Agility Pack

I am a big fan of LINQPad for prototyping small bits of code, but every now and then you find you need to reference another library. LINQPad, does allow this by editing the Query Properties and adding a reference to a dll in the GAC or at a specific path. But the new LINQPad beta release makes life even easier by allowing you to reference NuGet packages.

I recently wanted to do a simple bit of screen-scraping, to extract the results from a web page containing football scores. By examining the HTML with FireBug I was quickly able to see that each month’s fixtures were in a series of tables each with a header row and then one row per result. The HTML looked something like this:

<table class="fixtures" width="502" border="0">
<tbody>
    <tr class="fixture-header">
        <th class="first" scope="col" colspan="6">November</th>
        <th class="goals-for" title="Goals For" scope="col">F</th>
        <th class="goals-against" title="Goals Against" scope="col">A</th>
        <th class="tv-channel" scope="col">
        <th class="last" scope="col"> </th>
    </tr>
    <tr class="home">
        <td class="first ">04</td>
        <td class="month"> Wed </td>
        <td class="fixture-icon">
        <td class="competition">UEFA Champions League</td>
        <td class="home-away">H</td>
        <td class="bold opposition ">
        <td class="goals-for"> 4 </td>
        <td class="goals-against"> 1 </td>
        <td class="tv-channel"> </td>
        <td class="menu-button" valign="middle">
    </tr>

To be able to navigate around HTML in .NET, by far the best library I have found is the HTML Agility Pack. Adding this to your LINQPad Query is very simple with the new beta. Press F4 to bring up Query Properties, then click Add NuGet, find the Html Agility Pack in the list and click Add To Query.

Now we are ready to load the document and find all the tables with the class of “fixtures”. You can use a special XPath syntax to do this in one step:

var web = new HtmlAgilityPack.HtmlWeb();
var doc = web.Load("http://www.arsenal.com/fixtures/fixtures-reports?season=2009-2010&x=11&y=15");
foreach(var fixturesTable in doc.DocumentNode.SelectNodes("//table[@class='fixtures']"))
{
    // ...
}

Having got each fixture table, I then ignore the top row (which has a class of “fixture-header”), and use the classes on each of the table columns to pull out the information I am interested in. Finally, I use the handy Dump extension method in LINQPad to output my information to the results window:

foreach(var fixture in fixturesTable.SelectNodes("tr"))
{
    var fixtureClass = fixture.Attributes["class"];
    // header rows have class of fixture-header
    if(fixtureClass == null || !fixtureClass.Value.Contains("fixture-header"))
    {
        var day = fixture.SelectSingleNode("td[@class='first ']").InnerText.Trim();
        var month = fixture.SelectSingleNode("td[@class='month']").InnerText.Trim();
        var venue = fixture.SelectSingleNode("td[@class='home-away']").InnerText.Trim();
        var oppositionNode = fixture.SelectNodes("td").FirstOrDefault(n => n.Attributes["class"].Value.Contains("opposition"));
        var opposition = oppositionNode.SelectSingleNode("a").InnerText.Trim();
        var matchReportUrl = oppositionNode.SelectSingleNode("a").Attributes["href"].Value.Trim();
        var goalsFor = fixture.SelectSingleNode("td[@class='goals-for']").InnerText.Trim();
        var goalsAgainst = fixture.SelectSingleNode("td[@class='goals-against']").InnerText.Trim();
        string.Format("{0} {1} {2} {3} {4}-{5}", day, month, venue, opposition, goalsFor, goalsAgainst).Dump();
    }
}

This gives me the data I am after, and from here it is easy to convert it into any other format I want such as XML or insert it into a database (something that LINQPad also makes very easy).

04 Sun H Blackburn Rovers 6-2
17 Sat H Birmingham City 3-1
20 Tue A AZ Alkmaar 1-1
25 Sun A West Ham United 2-2
28 Wed H Liverpool 2-1
31 Sat H Tottenham Hotspur 3-0
04 Wed H AZ Alkmaar 4-1
07 Sat A Wolverhampton W. 4-1

And the really nice thing about using LINQPad for this rather than creating a Visual Studio project is that the entire thing is stored in a single compact .linq file without all the extraneous noise of sln, csproj, AssemblyInfo files etc.

10 C# keywords you should be using

Most developers who learn C# pick up the basic keywords quite quickly. Within a few weeks of working with a typical codebase you’ll have come across around a third of the C# keywords, and understand roughly what they do. You should have no trouble explaining what the following keywords mean:

public, private, protected, internal, class, namespace, interface, get, set, for, foreach .. in, while, do, if, else, switch, break, continue, new, null, var, void, int, bool, double, string, true, false, try, catch

However, while doing code reviews I have noticed that some developers get stuck with a limited vocabulary of keywords and never really get to grips with some of the less common ones, and so miss out on their benefits. So here’s a list, in no particular order, of some keywords that you should not just understand, but be using on a semi-regular basis in your own code.

is & as

Sometimes I come across a variation of the following code, where we want to cast a variable to a different type but would like to check first if that cast is valid:

if (sender.GetType() == typeof(TextBox))
{
   TextBox t = (TextBox)sender;
   ...
}

While this works fine, the is keyword could be used to simplify the if clause:

if (sender is TextBox)
{
   TextBox t = (TextBox)sender;
   ...
}

We can improve things further by using the as keyword, which is like a cast, but doesn’t throw an exception if the conversion is not valid – it just returns null instead. This means we can write code in a way that doesn’t require the .NET framework to check the type of our sender variable twice:

TextBox t = sender as TextBox;
if (t != null)
{
   ...
}

I feel obliged to add that if your code contains a lot of casts, you are probably doing something wrong, but that is a discussion for another day.

using

Most developers are familiar with the using keyword for importing namespaces, but a surprising number do not make regular use of it for dealing with objects that implement IDisposable. For example, consider the following code:

var writer = new StreamWriter("test.txt");
writer.WriteLine("Hello World");
writer.Dispose();

What we have here is a potential resource leak if there is an exception thrown between opening the file and closing it. The using keyword ensures that Dispose will always be called if the writer object was successfully created.

using (var writer = new StreamWriter("test.txt"))
{
   writer.WriteLine("Hello World");
}

Make it a habit to check whether the classes you create implement IDisposable, and if so, make use of the using keyword.

finally

Which brings us onto our next keyword, finally. Even developers who know and use using often miss appropriate scenarios for using a finally block. Here’s a classic example:

public void Update()
{   
    if (this.updateInProgress)
    {    
        log.WriteWarning("Already updating");    
        return;
    } 
    this.updateInProgress = true;
    ...
    DoUpdate();
    ...
    this.updateInProgress = false;

}

The code is trying to protect us from some kind of re-entrant or multithreaded scenario where an Update can be called while one is still in progress (please ignore the potential race condition for the purposes of this example). But what happens if there is an exception thrown within DoUpdate? Now we are never able to call Update again because our updateInProgress flag never got unset. A finally block ensures we can’t get into this invalid state:

public void Update()
{   
    if (this.updateInProgress)
    {    
        log.WriteWarning("Already updating");    
        return;
    } 
    try
    {
        this.updateInProgress = true;
        ...
        DoUpdate();
        ...
    }
    finally
    {
        this.updateInProgress = false;
    }
}

readonly

OK, this one is a fairly simple one, and you could argue that code works just fine without it. The readonly keyword says that a field can only be written to from within the constructor. It’s handy from a code readability point of view, since you can immediately see that this is a field whose value will never change during the lifetime of the class. It also becomes a more important keyword as you begin to appreciate the benefits of immutable classes. Consider the following class:

public class Person
{
    public string FirstName { get; private set; }
    public string Surname { get; private set; }

    public Person(string firstName, string surname)
    { 
        this.FirstName = firstName;
        this.Surname = surname;
    }
}

Person is certainly immutable from the outside – no one can change the FirstName or Surname properties. But nothing stops me from modifying those properties within the class. In other words, my code doesn’t advertise that I intend this to be an immutable class. Using the readonly keyword, we can express our intent better:

public class Person
{
    private readonly string firstName;
    private readonly string surname;

    public string FirstName { get { return firstName; } }
    public string Surname { get { return surname; } }

    public Person(string firstName, string surname)
    { 
        this.firstName = firstName;
        this.surname = surname;
    }
}

Yes, it’s a shame that this second version is a little more verbose than the first, but it makes it more explicit that we don’t want firstName or surname to be modified during the lifetime of the class. (Sadly C# doesn’t allow the readonly keyword on properties).

yield

This is a very powerful and yet rarely used keyword. Suppose we have a class that searches our hard disk for all MP3 files and returns their paths. Often we might see it written like this:

public List<string> FindAllMp3s()
{
   var mp3Paths = List<string>();
   ... 
   // fill the list
   return mp3Paths;
}

Now we might use that method to help us search for a particular MP3 file we had lost:

foreach(string mp3File in FindAllMp3s())
{
   if (mp3File.Contains("elvis"))
   {
       Console.WriteLine("Found it at: {0}", mp3File);
       break;
   }  
}

Although this code seems to work just fine, it’s performance is sub-optimal, since we first find every MP3 file on the disk, and then search through that list. We could save ourselves a lot of time if we checked after each file we found and aborted the search at that point.

The yield keyword allows us to fix this without changing our calling code at all. We modify FindAllMp3s to return an IEnumerable<string> instead of a List. And now every time it finds a file, we return it using the yield keyword. So with some rather contrived example helper functions (.NET 4 has already added a method that does exactly this) our FindAllMp3s method looks like this:

public IEnumerable<string> FindAllMp3s()
{
   var mp3Paths = List<string>();
   
   for (var dir in GetDirs())
   {
       for (var file in GetFiles(dir))
       {
           if (file.EndsWith(".mp3")
           {
               yield return file;
           }
       }
   }      
}

This not only saves us time, but it saves memory too, since we now don’t need to store the entire collection of mp3 files in a List.

It can take a little while to get used to debugging this type of code since you jump in and out of a function that uses yield repeatedly as you walk through the sequence, but it has the power to greatly improve the design and performance of the code you write and is worth mastering.

select

OK, this one is cheating since this is a whole family of related keywords. I won’t attempt to explain LINQ here, but it is one of the best features of the C# language, and you owe it to yourself to learn it. It will revolutionise the way you write code. Download LINQPad and start working through the tutorials it provides.

interface

So you already know about this keyword. But you probably aren’t using it nearly enough. The more you write code that is testable and adheres to the Dependency Inversion Principle, the more you will need it. In fact at some point you will grow to hate how much you are using it and wish you were using a dynamic language instead. (dynamic is itself another very interesting new C# keyword, but I feel that the C# community is only just beginning to discover how we can best put it to use).

throw

You do know you are allowed to throw as well as catch exceptions right? Some developers seem to think that a function should never let any exceptions get away, and so contain a generic catch block which writes an error to the log and returns giving the caller no indication that things went wrong.

This is almost always wrong. Most of the time your methods should simply allow exceptions to propagate up to the caller. If you are using the using keyword correctly, you are probably already doing all the cleanup you need to.

But you can and should sometimes throw exceptions. An exception thrown at the point you realise something is wrong with a good error message can save hours of debugging time.

Oh, and if you really do need to catch an exception and re-throw it, make sure you use do it the correct way.

goto

Only joking, pretend you didn’t see this one. Just because a keyword is in the language, doesn’t mean it is a good idea to use it. out and ref usually fall into this category too – there are better ways to write your code.

Merging MP3 Files with NAudio in C# and IronPython

If you would like to concatenate MP3 files using NAudio, it is quite simple to do. I recommend getting the very latest source code and building your own copy of NAudio, as this will work best with some of the changes that are in preparation for NAudio 1.4.

Here’s the C# code for a function that takes MP3 filenames, and writes a combined MP3 to the output stream:

public static void Combine(string[] inputFiles, Stream output)
{
    foreach (string file in inputFiles)
    {
        Mp3FileReader reader = new Mp3FileReader(file);
        if ((output.Position == 0) && (reader.Id3v2Tag != null))
        {
            output.Write(reader.Id3v2Tag.RawData, 0, reader.Id3v2Tag.RawData.Length);
        }
        Mp3Frame frame;
        while ((frame = reader.ReadNextFrame()) != null)
        {
            output.Write(frame.RawData, 0, frame.RawData.Length);
        }
    }
}

And here’s an IronPython script (just put NAudio.dll in the same folder as the mp3merge.py script):

import clr
clr.AddReference('NAudio.dll')

import sys
from NAudio.Wave import Mp3FileReader
from System.IO import File

def GetAllFrames(reader):
    while True:
        frame = reader.ReadNextFrame()
        if frame:
            yield frame
        else:
            return

def Merge(files, outputStream):
    for file in files:
        with Mp3FileReader(file) as reader:
            if reader.XingHeader:
                print 'discarding a Xing header'
            if not outputStream.Position and reader.Id3v2Tag:
                outputStream.Write(reader.Id3v2Tag.RawData, 0, reader.Id3v2Tag.RawData.Length)                
            for frame in GetAllFrames(reader):
                outputStream.Write(frame.RawData, 0, frame.RawData.Length);
            
if __name__ == '__main__':
    if len(sys.argv) < 3:
        print "Usage: ipy mp3merge.py output.mp3 File1.mp3 File2.mp3"
    else:
        with File.OpenWrite(sys.argv[1]) as outStream:
            Merge(sys.argv[2:],outStream)

Notes:

I simply copy across the ID3v2 tag from the first MP3 file if present. All other ID3v2 tags are discarded (as are ID3v1 tags). Also, I discard the Xing frame from VBR files. It could easily be re-included if desired, although it’s information will not necessarily be valid about the combined MP3 file. One final thing, I wouldn’t recommend merging MP3 files of different sample rates, or mixing mono with stereo, as it could cause various players issues.

MVVM – Is it Worth the Pain?

After finding it very easy to get MVVM working in WPF with IronPython, I thought it would be trivial to achieve the same thing in Silverlight. Unfortunately, my bindings didn’t work at all after porting a simple game to Silverlight. The problem may have something to do with the issues described in this blog post, but since I needed to demo my code for a local user group event, I abandoned MVVM and went back to just talking directly to the controls from within my ViewModel, which worked perfectly, and simplified my code considerably. It made me wonder why I was using MVVM in the first place.

The Pain of MVVM

I jumped on the bandwagon fairly early and have been writing the majority of my WPF and Silverlight code using MVVM. But to be honest, it has not been plain sailing. There is a real elegance to it that I like, and I love the testability, but there have been plenty of pain points.

For example, triggering animations and discovering when they have finished from the ViewModel has been a nightmare. I’ve tried various solutions, finding one that works with Silverlight and one with WPF but none that work with both, and have managed to crash Visual Studio dozens of times in the process.

Even simple things like setting the focus to the appropriate controls requires a rather convoluted system of “behaviours”, to do something that is a single line of code with access to the controls themselves. And every time I create a behaviour I seem to spend ages debugging my dependency properties, trying to work out why they aren’t doing what I expected.

The Purpose of MVVM

So if MVVM is so painful for non-trivial applications, why use it at all? What’s the point of MVVM? I think there are three answers.

1. Testability

The first driver for using MVVM is testability. If we put our logic in the C# “code-behind” of a XAML class, then running unit tests on that logic becomes very difficult. But with IronPython, that consideration is irrelevant. Since it is a dynamic language, and we are not using “code-behind” in any case, my non-MVVM version of my ViewModel is just as testable as the one that used data binding exclusively.

2. Data Binding

The second motivation for using MVVM is that the pattern encourages the use of data binding, and simplifies it considerably by removing the need for data converters or complicated binding expressions. But why is data-binding to a TextBox better than just getting or setting its Text property?

One of the key benefits of data-binding comes when we want to bind the same thing to more than one property. A Command object is a good example. One Command object could be bound to a drop-down menu and a toolbar button. So setting the Command’s IsEnabled state can be used to enable and disable both views. But most of the time, binding is a one-to-one relationship.

Data binding also saves a bunch of code in CRUD type applications where you are binding directly to properties on an existing model type, but I don’t tend to write applications like that anyway. I think you can use it to make magical things happen with input validation too, but again, it’s not something I’ve ever needed.

3. Designability

The third driver behind MVVM is the goal of designer / developer separation. The idea is that a designer can create a XAML file and bind it to test data, and then the developer can come along and create the real DataContext at a later date. The idea certainly sound impressive, but I can’t help but think that the designer will run into all the problems mentioned above if he actually wants to give his XAML GUI a real workout. And the truth is that I’ve never worked with a designer writing XAML and will not be doing so in the foreseeable future. The graphic designers we use send us bitmaps and Flash mockups.

A Hybrid Approach

I am now wondering whether I should think of my ViewModel more as a “Presenter”. It could still be the DataContext of the object defined in XAML to enable data-binding where required, but it would also have access directly to any properties on that XAML object. For testability in statically typed languages, you could define an interface with methods like StartAnimation or SetFocusToTextBox – a little cumbersome perhaps, but probably no more so than the hoops you have to jump through to make these things happen from your ViewModel using nothing but data binding.

Anyway, looks like I’m not alone in questioning MVVM. Here’s a good post from K Scott Allen. What do you think? Does MVVM fail a cost-benefit analysis for the types of application you work on?

Convert 16 bit PCM to IEEE float

NAudio has had the Wave32Stream for quite some time which converts a 16 bit PCM stream into a stereo IEEE floating point stream, with optional panning and volume. However, it could do with something simpler, that doesn’t automatically convert to stereo. So here is a preliminary implementation of an IWaveProvider that converts 16 bit PCM to IEEE float. It keeps the Volume property as that is always useful to have available. It keeps the code nice and clean by making use of the WaveBuffer class. I’ll probably add this to NAudio in the near future.

/// <summary>
/// Converts 16 bit PCM to IEEE float, optionally adjusting volume along the way
/// </summary>
public class Wave16toIeeeProvider : IWaveProvider
{
    private IWaveProvider sourceProvider;
    private readonly WaveFormat waveFormat;
    private volatile float volume;
    private byte[] sourceBuffer;

    /// <summary>
    /// Creates a new Wave16toIeeeProvider
    /// </summary>
    /// <param name="sourceStream">the source stream</param>
    /// <param name="volume">stream volume (1 is 0dB)</param>
    /// <param name="pan">pan control (-1 to 1)</param>
    public Wave16toIeeeProvider(IWaveProvider sourceProvider)
    {
        if (sourceProvider.WaveFormat.Encoding != WaveFormatEncoding.Pcm)
            throw new ApplicationException("Only PCM supported");
        if (sourceProvider.WaveFormat.BitsPerSample != 16)
            throw new ApplicationException("Only 16 bit audio supported");

        waveFormat = WaveFormat.CreateIeeeFloatWaveFormat(sourceProvider.WaveFormat.SampleRate, sourceProvider.WaveFormat.Channels);

        this.sourceProvider = sourceProvider;
        this.volume = 1.0f;
    }

    /// <summary>
    /// Helper function to avoid creating a new buffer every read
    /// </summary>
    byte[] GetSourceBuffer(int bytesRequired)
    {
        if (sourceBuffer == null || sourceBuffer.Length < bytesRequired)
        {
            sourceBuffer = new byte[bytesRequired];
        }
        return sourceBuffer;
    }

    /// <summary>
    /// Reads bytes from this wave stream
    /// </summary>
    /// <param name="destBuffer">The destination buffer</param>
    /// <param name="offset">Offset into the destination buffer</param>
    /// <param name="numBytes">Number of bytes read</param>
    /// <returns>Number of bytes read.</returns>
    public int Read(byte[] destBuffer, int offset, int numBytes)
    {
        int sourceBytesRequired = numBytes / 2;
        byte[] sourceBuffer = GetSourceBuffer(sourceBytesRequired);
        int sourceBytesRead = sourceProvider.Read(sourceBuffer, offset, sourceBytesRequired);
        WaveBuffer sourceWaveBuffer = new WaveBuffer(sourceBuffer);
        WaveBuffer destWaveBuffer = new WaveBuffer(destBuffer);

        int sourceSamples = sourceBytesRead / 2;
        int destOffset = offset / 4;
        for (int sample = 0; sample < sourceSamples; sample++)
        {
            destWaveBuffer.FloatBuffer[destOffset++] = (sourceWaveBuffer.ShortBuffer[sample] / 32768f) * volume;
        }

        return sourceSamples * 4;
    }

    /// <summary>
    /// <see cref="IWaveProvider.WaveFormat"/>
    /// </summary>
    public WaveFormat WaveFormat
    {
        get { return waveFormat; }
    }

    /// <summary>
    /// Volume of this channel. 1.0 = full scale
    /// </summary>
    public float Volume
    {
        get { return volume; }
        set { volume = value; }
    }
}

.NET Gotcha – Static Initialiser Ordering

Last week I had to troubleshoot a strange problem. A developer had broken up the source code for a large class of constants (defining various colours) into separate files using the partial keyword, and all of a sudden our Windows Forms controls were painting themselves black instead of the correct colour. Put them back into one file again and the problem went away.

Eventually we tracked the problem down due to the order in which the fields were being initialised by the static constructor. Consider the following code and unit test (which passes):

static class MyConstants
{
    public static readonly int MyNumber = 5;
    public static readonly int MyOtherNumber = MyNumber;
}

[TestFixture]
public class MyConstantsTests
{
    [Test]
    public void ConstantsAreInitialisedCorrectly()
    {
        Assert.AreEqual(5, MyConstants.MyNumber);
        Assert.AreEqual(5, MyConstants.MyOtherNumber);
    }
}

Now if we simply change the ordering of the statements in MyConstants…

static class MyConstants
{
    public static readonly int MyOtherNumber = MyNumber;
    public static readonly int MyNumber = 5;
}

… the test will fail as MyOtherNumber will be 0. Obviously if the two definitions exist in different source files, this type of problem is much harder to spot. The test does pass if we use the const keyword instead of static readonly, but since we were initialising using the Color.FromArgb function, this was not an option for us.

The moral of the story is to avoid setting static read-only fields to values dependent on other fields. Or at least be aware of the problems that can arise from doing so.

Wanted Language Feature: Reinterpret Cast of Byte Arrays

I am a huge fan of C#, but one of the most frustrating things about it is dealing with byte arrays which actually represent some other type of data. For example, suppose I have an array of bytes that I know actually contains some floating point numbers. What I would like to be able to do is:

byte[] blah = new byte[1024];
float[] flah = (float[])blah;

But of course, this won't compile. There are two options:

1. Create a new array of floats and copy the contents of the byte array into it, using the BitConverter.ToSingle method. I could then access the contents as floats. The disadvantages are obvious. It requires twice the memory, and copying it across is not free. Also if I modify any values, they may need to be copied back into the original byte array.

2. Using the unsafe and fixed keywords, pin the byte array where it is and obtain a float pointer. The disadvantages are obvious. First, pinning objects interferes with the garbage collector, reducing performance (and performance is often exactly what you want when you are dealing with arrays of numbers), and second, as the keyword suggests, pointers are unsafe. Here's some example code from my open source audio library NAudio that shows me using this method to mix some audio:

unsafe void Sum32BitAudio(byte[] destBuffer, int offset, byte[] sourceBuffer, int bytesRead)
{
    fixed (byte* pDestBuffer = &destBuffer[offset],
              pSourceBuffer = &sourceBuffer[0])
    {
        float* pfDestBuffer = (float*)pDestBuffer;
        float* pfReadBuffer = (float*)pSourceBuffer;
        int samplesRead = bytesRead / 4;
        for (int n = 0; n < samplesRead; n++)
        {
            pfDestBuffer[n] += (pfReadBuffer[n] * volume);
        }
    }
}

But does it really need to be this way? Why can't the .NET framework let me consider a byte array to be a float array, without the need for copying, pinning or unsafe code? I've tried to think through whether there would be any showstoppers for a feature like this being added...

1. The garbage collector shouldn't need any extra knowledge. The float array reference would be just like having another byte array reference, and the garbage collector would know not to delete it until all references were gone. It could be moved around in memory if necessary without causing problems.

2. Sizing need not be an issue. If my byte array is not an exact multiple of four bytes in length, then the corresponding float array would simply have a length as large as possible.

3. This would only work for value types which themselves only contained value types. Casting an array of bytes to any type that contained a reference type would of course be unsafe and allow you to corrupt pointers. But there is nothing unsafe about casting say an array of bytes into an array of DateTimes. The worst that could happen would be to create invalid DateTime objects.

The benefits of adding this as a language feature would go beyond simply playing with numbers. It would be ideal for interop scenarios, removing the need for Marshal.PtrToStructure in many cases. Imagine being able to write code like the following:

byte[] blah = new byte[1024];
int x = MyExternalDllFunction(blah);
if (x == 0)
{
    MyStructType myStruct = (MyStructType)blah;
}
else
{
    MyOtherStructType myOtherStruct = (MyOtherStructType)blah;
}

What do you think? Would you use this feature if it was in C#? It needn't be implemented as a cast. It could be a library function. But the key thing would be to create two different struct or array of struct types that provided views onto the same block of managed memory.

Basic Web Service Access From C#

I recently wrote a small plugin for Windows Live Writer, which allowed you to enter a Bible reference, and it would either create a link to Bible Gateway, or insert the text in the English Standard Version.

To get the ESV text, I needed to call a web service in C#. It was the first time I had done this, and it turned out to be surprisingly simple. First I needed to construct the URL to call. This was simply a case of constructing the ESV Web Service API documentation. The only slight complication is the need to add a reference to System.Web so you can call HttpUtility.UrlEncode.

StringBuilder url = new StringBuilder();
url.Append("http://www.gnpcb.org/esv/share/get/");
url.Append("?key=IP");
url.Append("&passage=");
url.Append(System.Web.HttpUtility.UrlEncode(reference));
url.Append("&action=doPassageQuery");
url.Append("&include-passage-references=false");
url.Append("&include-audio-reference=false");
url.Append("&include-footnotes=false");
url.Append("&include-subheadings=false");
url.Append("&include-headings=false");

Once you have the URL with the search parameters, then you simply pass it to a web request, get its ResponseStream() and read it to the end.

WebRequest request = WebRequest.Create(url.ToString());
StreamReader responseStream = new StreamReader(request.GetResponse().GetResponseStream());
String response = responseStream.ReadToEnd();
responseStream.Close();
return response.ToString();

Obviously there are more complicated types of web service than this, which require construction and parsing of XML, but I was still impressed with how few lines of code were actually involved in my first web service call.