Showing posts with label NUnit. Show all posts
Showing posts with label NUnit. Show all posts

Wednesday, 14 July 2010

Running NUnit tests on .NET 4 assemblies with MSBuild Part 2

Yesterday, I posted a way to run NUnit tests on .NET 4 assembles with MSBuild. Although it worked, I was not 100% happy with the way of going about it. Thanks to a comment from Travis Laborde, I can now present a simpler way, making use of the exec task in MSBuild and the /framework switch on nunit-console.exe. Here’s the project file syntax:

<Target Name="Test2" DependsOnTargets="Build">
  <exec 
     command="&quot;C:\Program Files\NUnit 2.5.5\bin\net-2.0\nunit-console.exe&quot; /framework=4.0.30319 @(TestAssembly)">
     WorkingDirectory="."
  </exec>
</Target>

Tuesday, 13 July 2010

Running NUnit tests on .NET 4 assemblies with MSBuild

If you want to run NUnit tests with MSBuild, then you need to download and install the MSBuild Community tasks. This allows you to create a unit test target like this:

<ItemGroup>
  <TestAssembly Include="ClientBin\*Tests.dll" />
</ItemGroup>
<Target Name="Test" DependsOnTargets="Build">
  <NUnit Assemblies="@(TestAssembly)"
         WorkingDirectory="."
         ToolPath="C:\Program Files\NUnit 2.5.5\bin\net-2.0"
         />
</Target>

(Most examples don’t show the need to specify a ToolPath, but I have found I need it, perhaps because it isn’t in my Path?)

The trouble is, if the assemblies containing the unit tests have been build with the .NET 4 framework, you will get the following error:

C:\Program Files\NUnit 2.5.5\bin\net-2.0\nunit-console.exe /nologo ClientBin\
Nice.Inform.Client.Tests.dll
ProcessModel: Default    DomainUsage: Single
Execution Runtime: net-2.0
Unhandled Exception:
System.BadImageFormatException: Could not load file or assembly 'D:\TFS\Trial
\Inform5\ClientBin\Client.Tests.dll' or one of its dependencies.
This assembly is built by a runtime newer than the currently loaded runtime a
nd cannot be loaded.
File name: 'D:\TFS\Trial\Inform5\ClientBin\Client.Tests.dll'

What is needed is to ensure that nunit-console.exe runs against the .NET 4 framework. The way I achieved this was to make a copy of the net-2.0 folder that comes with NUnit 2.5.5 and rename it to net-4.0 (n.b. make sure the lib folder comes along too as nunit-console.exe depends on its contents). Then, I edited the nunit-console.exe.config file to have the following contents (the key bit is to add the supportedRuntime setting):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- Set the level for tracing NUnit itself -->
  <!-- 0=Off 1=Error 2=Warning 3=Info 4=Debug -->
  <system.diagnostics>
    <switches>
       <add name="NTrace" value="0" />
    </switches>
  </system.diagnostics>

  <startup>
    <supportedRuntime version="v4.0"/>
  </startup>

  <runtime>
    <!-- We need this so test exceptions don't crash NUnit -->
    <legacyUnhandledExceptionPolicy enabled="1" />

    <!-- Look for addins in the addins directory for now -->
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="lib;addins"/>
   </assemblyBinding>
  </runtime>
</configuration>

Now all that is needed is to update the ToolPath in the MSBuild script to point to the new location

<Target Name="Test" DependsOnTargets="Build">
  <NUnit Assemblies="@(TestAssembly)"
        WorkingDirectory="."
        ToolPath="C:\Program Files\NUnit 2.5.5\bin\net-4.0"
   />
</Target>

I’d be interested in hearing if there is an easier way of solving this problem though, as I don’t really want to have to require all developers to manually perform these steps on their machine.

Thursday, 24 September 2009

Crowd-Sourced Code Reviews

In a post yesterday, I bemoaned the lack of a clean syntax in NUnit for testing whether events fired from an object under test, and set to work at writing my own helper methods. I posted on the NUnit discussion group, to see whether there was a better way. And there was. A really obvious way. Here’s how my tests could have been written without the need for an Assert.Raises function, or for any code outside the test method:
[Test]
public void TestRaisesClosedEvent_Improved()
{
    Blah blah = new Blah();
    EventArgs savedEventArgs = null;
    blah.Closed += (sender, args) => savedEventArgs = args;
    blah.RaiseClosed(1);
    Assert.IsNotNull(savedEventArgs);
}

[Test]
public void TestCanCheckRaisesMany_Improved()
{
    Blah blah = new Blah();
    var savedEvents = new List<openedeventargs>();
    blah.Opened += (sender, args) => savedEvents.Add(args);
    blah.RaiseOpened(5);
    Assert.AreEqual(5, savedEvents.Count);
    Assert.AreEqual("Message 3", savedEvents[2].Message);
}
Of course, I was kicking myself for missing such an obvious solution, but it left me pondering how often this happens without me realising it.
Earlier today Clint Rutkas tweeted:
i feel like i'm doing such bad things in WPF right now
… and I know exactly how he feels. In a couple of WPF applications I am working on, I am trying to use the MVVM pattern. It often ends up with me writing code that might be a really cool piece of lateral thinking, or it might be a pointlessly overcomplicated hack. For an example, here’s an  attached dependency property (usage here) I created to let me bind to Storyboard completed events.
What I need is another person to look over my shoulder and tell me whether I am missing the obvious, unaware of an API, or ignorant of a best practice. But all too often, that person doesn’t exist.
In a commercial environment, hopefully there is at least some kind of provision for code reviews to take place, or maybe even pair programming. But what about the lone programmer working on open source projects?
Maybe we need some way to “crowd-source” code reviews. Some way of getting lots of eyes on your source code, even if it is only for a couple of minutes, and an easy way of getting hold of that feedback. A bit like fivesecondtest but where you help out an open source developer by code reviewing a changeset. I’ve asked for this as a feature on CodePlex.
What do you think? Could it work? How do you make sure you’re not missing the obvious in the code you write?

Wednesday, 23 September 2009

Asserting events with NUnit

Suppose we want to write unit tests for a class that raises events. We want to check that the right events are raised, the right number are raised, and that they have the correct parameters. Here’s an example of a class that we might want to test:
class Blah
{
    public event EventHandler Closed;
    public event EventHandler<OpenedEventArgs> Opened;

    public void RaiseClosed(int count)
    {
        for(int n = 0; n < count; n++)
        {
            Closed(this, EventArgs.Empty);
        }
    }

    public void RaiseOpened(int count)
    {
        for(int n = 0; n < count; n++)
        {
            Opened(this, new OpenedEventArgs() { Message = String.Format("Message {0}", n + 1) });
        }
    }
}

class OpenedEventArgs : EventArgs
{
    public string Message { get; set; }
}

To write NUnit test cases for this class, I would usually end up writing an event handler, and storing the parameters of the event in a field like follows:
EventArgs blahEventArgs;

[Test]
public void TestRaisesClosedEvent()
{
    Blah blah = new Blah();
    blahEventArgs = null;
    blah.Closed += new EventHandler(blah_Closed);
    blah.RaiseClosed(1);
    Assert.IsNotNull(blahEventArgs);
}

void blah_Closed(object sender, EventArgs e)
{
    blahEventArgs = e;
}
While this approach works, it doesn’t feel quite right to me. It is fragile – forget to set blahEventArgs back to null and you inadvertently break other tests. I was left wondering what it would take to create an Assert.Raises method that removed the need for a private variable and method for handling the event.
My ideal syntax would be the following:
Blah blah = new Blah();
var args = Assert.Raises<OpenedEventArgs>(blah.Opened, () => blah.RaiseOpened(1));
Assert.AreEqual("Message 1", args.Message);

The trouble is, you can’t specify “blah.Opened” as a parameter – this will cause a compile error. So I have to settle for second best and pass the object that raises the event, and the name of the event. So here’s my attempt at creating Assert.Raise, plus an Assert.RaisesMany method that allows you to see how many were raised and examine the EventArgs for each one:
public static EventArgs Raises(object raiser, string eventName, Action function)
{
    return Raises<EventArgs>(raiser, eventName, function);
}

public static T Raises<T>(object raiser, string eventName, Action function) where T:EventArgs
{
    var listener = new EventListener<T>(raiser, eventName);
    function.Invoke();
    Assert.AreEqual(1, listener.SavedArgs.Count);
    return listener.SavedArgs[0];
}

public static IList<T> RaisesMany<T>(object raiser, string eventName, Action function) where T : EventArgs
{
    var listener = new EventListener<T>(raiser, eventName);
    function.Invoke();
    return listener.SavedArgs;
}

class EventListener<T> where T : EventArgs
{
    private List<T> savedArgs = new List<T>();

    public EventListener(object raiser, string eventName)
    {
        EventInfo eventInfo = raiser.GetType().GetEvent(eventName);
        var handler = Delegate.CreateDelegate(eventInfo.EventHandlerType, this, "EventHandler");
        eventInfo.AddEventHandler(raiser, handler);            
    }

    private void EventHandler(object sender, T args)
    {
        savedArgs.Add(args);
    }

    public IList<T> SavedArgs { get { return savedArgs; } }
}
This allows us to dispense with the private field and event handler method in our test cases, and have nice clean test code:
[Test]
public void TestCanCheckRaisesEventArgs()
{
    Blah blah = new Blah();
    AssertExtensions.Raises(blah, "Closed", () => blah.RaiseClosed(1));
}

[Test]
public void TestCanCheckRaisesGenericEventArgs()
{
    Blah blah = new Blah();
    var args = AssertExtensions.Raises<OpenedEventArgs>(blah, "Opened", () => blah.RaiseOpened(1));
    Assert.AreEqual("Message 1", args.Message);
}

[Test]
public void TestCanCheckRaisesMany()
{
    Blah blah = new Blah();
    var args = AssertExtensions.RaisesMany<OpenedEventArgs>(blah, "Opened", () => blah.RaiseOpened(5));
    Assert.AreEqual(5, args.Count);
    Assert.AreEqual("Message 3", args[2].Message);
}

There are a couple of non-optimal features to this solution:
  • Having to specify the event name as a string is ugly, but there doesn’t seem to be a clean way of doing this.
  • It expects that your events are all of type EventHandler or EventHandler<T>. Any other delegate won’t work.
  • It throws away the sender parameter, but you might want to test this.
  • You can only test on one particular event being raised in a single test (e.g. can’t test that a function raises both the Opened and Closed events), but this may not be a bad thing as tests are not really supposed to assert more than one thing.
I would be interested to know if anyone has a better way of testing that objects raise events. Am I missing a trick here? Any suggestions for how I can make my Raises function even better?
Download full source code here

Monday, 14 September 2009

Parameterized Tests with TestCase in NUnit 2.5

For some time, NUnit has had a RowTest attribute in the NUnit.Extensions.dll, but with NUnit 2.5, we have built-in support for parameterized tests. These reduce the verbosity of multiple tests that differ only by a few arguments.

For example, if you had test code like this:

[Test]
public void Adding_1_and_1_should_equal_2()
{
    Assert.AreEqual(2,calculator.Add(1,1));
}

[Test]
public void Adding_1_and_2_should_equal_3()
{
    Assert.AreEqual(3, calculator.Add(1, 2));
}

[Test]
public void Adding_1_and_3_should_equal_4()
{
    Assert.AreEqual(4, calculator.Add(1, 3));
}

You can now simply refactor to:

[TestCase(1, 1, 2)]
[TestCase(1, 2, 3)]
[TestCase(1, 3, 4)]
public void AdditionTest(int a, int b, int expectedResult)
{
    Assert.AreEqual(expectedResult, calculator.Add(a, b));
}

The NUnit GUI handles this very nicely, allowing us to see which one failed, and run them individually:

NUnit Parameterized Tests

TestDriven.NET doesn’t give us the ability to specify an individual test case to run (hopefully a future feature), but it will show us the parameters used for any test failures:

TestCase 'SanityCheck.CalculatorTests.AdditionTest(1,2,3)' failed: 
  Expected: 3
  But was:  2
...\CalculatorTests.cs(19,0): at SanityCheck.CalculatorTests.AdditionTest(Int32 a, Int32 b, Int32 expectedResult)

TestCase 'SanityCheck.CalculatorTests.AdditionTest(1,3,4)' failed: 
  Expected: 4
  But was:  2
...\CalculatorTests.cs(19,0): at SanityCheck.CalculatorTests.AdditionTest(Int32 a, Int32 b, Int32 expectedResult)

Another very cool feature is that you can specify a TestCaseSource function, allowing you to generate test cases on the fly. One way I have used this feature is for some integration tests that examine a folder of legacy test data files and create a test for each file.

There are a few options for how to provide the source data. Here I show using a function that returns an IEnumerable<string>:

[TestCaseSource("GetTestFiles")]
public void ImportXmlTest(string xmlFileName)
{
    xmlImporter.Import(xmlFileName);
}

public IEnumerable<string> GetTestFiles()
{
    return Directory.GetFiles("D:\\Test Data", "*.xml");
}

Now when NUnit loads the test assembly, it runs the GetTestFiles function, to get the test case parameters, allowing you to run them individually if necessary.

TestCaseSource Function

There is one gotcha. If your TestCaseSource function takes a long time to run, you will need to wait every time NUnit (or TestDriven.NET) reloads your unit test DLL. So my example of examining a directory for files could negatively impact performance if there are too many test cases (I discovered this after loading 14000 test cases from a network drive).