This Technology Letter gives an overview of the state of the art in unit testing in .NET. The target audience includes developers, architects, test managers and project leaders. We start by looking at plain old unit testing and give a quick comparison of the available frameworks. In the second part, we introduce the concepts behind mocking and give some advice on how to prevent common mistakes in unit tests. Finally, we present two interesting Microsoft Research projects and discuss their benefits.
Unit tests are automated tests that check the smallest testable units (usually methods) of program in an isolated environment for correct operation. Tests that do more than that, or have dependencies on external components (for example a database, the file system or a network resource), are called integration tests.
Test-Driven Development (TDD) is a programming practice which is tightly coupled with unit testing. In theory, it is pretty simple: Tests are written before the productive code. TDD is an evolutionary way to develop software, where small steps and a lot of refactoring lead to better overall design and code quality.
Unit Testing Frameworks
Currently, there are three major frameworks: Microsoft's MSTest and the open-source frameworks NUnit and MbUnit. Another framework is xUnit, which we will not cover in this article. MSTest is a latecomer in this field and provides fewer assertion and extensibility options than its counterparts, but superior reporting capabilities in combination with TFS (Microsoft Team Foundation Server) and better integration in Microsoft Visual Studio. MSTest contains some useful code generation tools, such as the automatic generation of unit tests for all public members of a class, or the generation of so-called accessors, which enable a developer to access non-public classes and members in other assemblies in a unit test. Note that for internal classes and members, this can also be achieved with friend assemblies. Apart from more assertions and a "fluent API", NUnit - as the former de facto standard - and MbUnit support parameterized unit tests and are easily extensible, for example with new attributes. Have a look at the following examples: A fluent assertion in NUnit:
A parameterized unit test is run multiple times with different input values.
Example: Parameterized unit test using the NUnit framework.
While NUnit is a "pure" unit testing framework, the other frameworks offer features more targeted at integration testing, such as support for data-driven tests which require a DB connection. MSTest even has support for automated UI tests, so-called coded UI tests. Roy Osherove's blog has a comparison of MSTest and NUnit. As mentioned above, unit testing is all about testing a single component in isolation. Usually components have some external dependencies that have to be removed first. This is where mocking frameworks come into play.
Consider the following scenario: We want to implement a graphical calculator using the Model-View-Presenter pattern, because we have heard that this pattern allows us to decouple the logic from the user interface (UI), which would enable us to unit test our code without having a dependency on UI. Our implementation looks as follows:
In our unit test, we want to verify that pressing a key in the view actually updates its display, or more specifically, we want to test that raising a KeyPressed event on the view causes the presenter to call SetText(..) with the pressed key on the view. To do this, we have to create a fake implementation of the view and pass it to the presenter. Even though we could do this manually, mocking frameworks make this task a lot easier. This is how we would verify the behavior of the presenter using the Moq framework:
Example: Behavior verification using the Moq framework
Behavior vs. State Verification
Note that the use of a mocking framework enabled us to do behavior verification in this test; we verified that the expected interaction between the presenter and the view took place. In traditional unit tests, we usually do state verification; we perform all actions and then check if the system produces the expected result. The potential danger with behavior verification is that we test too much of the internal logic of the code under test. So if the code is refactored, it is possible that the tests break, even though the external behavior of the code has not changed and is still correct. Therefore, state verification is preferable to behavior verification whenever possible. In the example above, behavior verification is the best choice.
Mockist vs. Classical TDD
Of course, mocking frameworks are well suited for TDD: They enable a developer to work with an interface for which there is not yet an implementation. Martin Fowler distinguishes between two styles of TDD: classical and mockist. In our test, we used the classical style. While we mocked the view, we used the real calculator implementation. A developer following the mockist style generally uses fake implementations of all the dependencies. Because every additional fake object increases the complexity of a test and reduces its readability, it is recommended to use real implementations if possible. Usually this is also easier and has the additional benefit of increasing the test coverage of the productive code. However, if a calculator implementation is unavailable at the time the test is written, or if the calculator class itself has many external dependencies, using a fake implementation is justified.
The main mocking frameworks in the .NET world are Rhino Mocks, Moq and the commercial TypeMock Isolator. Recently, Microsoft Research published Moles, which we will cover later in this article. Moq is a relatively new framework which brought some new ideas into the world of mocking. The other frameworks followed quickly. Now, tests written with either of the frameworks look similar. Rhino Mocks is still the most complex of the three, mainly because it contains a lot of legacy APIs, which makes it hard for developers to pick the right methods. Because Rhino Mocks and Moq both rely on the dynamic generation of proxies, it is not possible to mock certain things, such as non-virtual or static methods and sealed classes. TypeMock and Moles, on the other hand, make use of the profiling API, which enables them to intercept any .NET framework call, for example one to System.DateTime.Now. Of course this power should not be abused; the preferred solution is always to refactor the code and work with interfaces that can easily be mocked. But sometimes such an isolation framework can be invaluable, for example when working with Microsoft SharePoint, ASP.NET or legacy code. In TypeMock, the static Isolate class is the single point of entry. Our example would look as below. Note that we do not need interfaces anymore; we work directly with the View class.
Stubs and Moles
Moles is a Microsoft Research project which is available for download as a Visual Studio Power Tool. Stubs is a part of Moles. Note that Moles does not depend on MSTest and can be used with other unit testing frameworks.
Stubs is a simple framework that generates stubs for all interfaces and non-sealed classes in an assembly. The behavior of the stubs can be customized by attaching delegates to the provided properties. The example cited in the mocking frameworks section above could be ported to the Stubs framework as shown below. Notice the naming convention that is used for the stubs: To every class or interface, the prefix S is added. So for the IView interface, a SIView stub class is created. The IView.SetText(string) method can be customized by attaching a delegate to the SIView.SetTextString property. Because Stubs is not a mocking framework, it does not provide any behavior verification out of the box, but we can of course easily achieve this with a callback mechanism as shown in the example below:
For a long time, TypeMock Isolator was the only product powerful enough to detour arbitrary .NET framework methods. Now Moles gives you this ability for free. Moles allows a developer to replace any .NET method with a delegate. Note again the naming convention: Mole types are prefixed with a capital M, so to customize the behavior of the static method File.ReadAllText(string), you attach a delegate to MFile.ReadAllTextString as shown in the example below. For moles to detour this call, the test must run under a profiler. This can be achieved with the HostType attribute.
The File class is defined in the mscorlib assembly. In order to create the MFile class, you have to add a new "Moles and Stubs for Testing" item with the name mscorlib.moles to your test project. The contents look as below. After doing this, Moles creates a new assembly mscorlib.Moles.dll which contains all the moled types. This file can also contain more specific instructions for the creation of mole and stub types.
Pex (Program EXploration for .NET) is another Microsoft Research project available as a Visual Studio Power Tool. Note that it is not free for commercial use, but requires an MSDN subscription. The Pex installer includes Moles. Pex automatically generates a parameterized unit test for a method. To do this, right click anywhere in a public method and select "Create Parameterized Unit Test" as shown below. Note that Pex can also generate a test suite for an entire assembly.
By clicking "Run Pex", Pex tries to cover all possible branches in the code by providing interesting input values for the parameterized test. As an example, consider a small tool that automatically generates documentation from a unit test given its class and method name. The parameterized test looks as below. The PexMethod attribute is added to all Pex tests. PexAssumeUnderTest tells Pex that a parameter cannot be null. As the "TODO" in the code below shows, you can actually edit this test and add your own asserts, which does not make sense for our example.
The following screenshot shows the parameters Pex comes up with. Each row stands for one execution of the parameterized test. The columns testClassName and testMethodName contain the input values.
For the first two tests, Pex receives an exception, because a precondition for the method is violated. Since this is the expected behavior, the tests have passed successfully. For the third row, Pex generates the following unit test behind the scenes. These automatically generated tests must not be modified manually.
Pex only tests what is already contained in the productive code. So the task of the programmer is to check if all input and output values make sense. Pex will also try to create instances of classes or even interfaces. In the screenshot below we see that it guessed which concrete types to pass to the constructor of our presenter. All the warnings should be reviewed. By clicking "Add PexUseType" we tell Pex that its guess is actually correct and that we do not want to see any further warnings.
We can also hook into the object creation of our types. The screenshot below shows how Pex creates the view. We can accept or edit this.
Pex is a useful unit testing tool, but it does certainly not replace classical unit tests. Rather it could be used to complement them, for example by looking for corner cases. While its use for TDD is limited, it can be very helpful for quickly generating a large test suite for an existing code base, which can then be used for regression testing. Note that using Pex together with code contracts greatly improves the results. A video demonstrating Moles, Pex and code contracts in action can be found here.
After a comparison of the different unit testing frameworks, we introduced a few important mocking concepts and gave some advice on how to make tests more maintainable and less likely to break if an implementation detail changes. After an overview of the available mocking frameworks, we presented Moles. Finally, we had a quick look at Pex and discussed its use. When it comes to selecting a mocking framework, Moq is probably the best choice because it is free, easy to use and provides a clean API. For the few things that are hard or impossible to mock, Moles is the perfect candidate. If a project contains a lot of "untestable" code, for example legacy code, it is probably worth investing some money in TypeMock Isolator, because it supports behavior verification and has an easier syntax than Moles. It is good to see that Microsoft finally recognized the importance of unit testing and testability. Not only are there interesting research projects, but Visual Studio 2010 now contains support for TDD (for more details see the beginning of this article). ASP.NET has long been criticized for its lack of testability; now you can develop your Web applications using ASP.NET MVC instead. The ADO.NET Entity Framework has made an important step in the right direction as well. Version 4.0 is now able to generate POCOs (Plain Old CLR Objects; entities which have no dependency on the framework) that can easily be used in unit tests. We are expecting to see further innovations in this area. Microsoft seems to be on the right track. Hopefully your projects are as well.