I really like how BDD specs read, especially in the test result window. We don’t use anything like MSpec yet. I’m still trying to get testing to gain momentum and making them look as far from regular code as MSpec does makes it that much more difficult.

We do have a pattern we follow:

public class When_something_happens : Specification{

  Foo somethingUnderTest;
  Bar result;

  override public void Given(){
    somethingUnderTest = new Foo(...);
  }

  override public void When(){
    result = somethingUnderTest.doesSomething();
  }

  [Then] public void the_result_should_not_be_null(){
     result.ShouldNotBeNull();
  }

}

We just created a base Specification, renamed the xUnit FactAttribute to ThenAttribute (probably unnecessary) and then created a load of extensions of object for the ShouldBeWhatevers. There is more noise than there would be with MSpec, but that’s the tradeoff for looking like code.

I’m pretty happy with all this and it works pretty well, but what I’m finding is that the BDD specs I write before I write a class often don’t test what the class does in the end, after that work has been off loaded to a dependency. Take for example the following simple spec. (bits have been left out for simplicity…know it won’t compile)

public class When_creating_an_order_invoice_view_model_from_an_order : Specification {

  override public void Given(){
    factory = new OrderInvoiceViewDataFactory();
    order = new Order{ FirstName = "Homer",
                       LastName  = "Simpson",
                       Items     = new []{
                                     new LineItem{ Product ="Duff 6 pack",
                                                            Price = 4.99m, }
                                     new LineItem{ Product ="Duff can koozie",
                                                            Price = 0.99m, }
                                   }
  }

  override public void When(){
    invoice = factory.BuildFrom(order);
  }

  [Then] public void the_total_should_be_the_sum_of_the_items(){
    invoice.Total.ShouldBe(4.99m + 0.99m);
  }

}

Overlooking the simplicity of the spec, it seems reasonable right?

Here’s my problem. I build my factory to calculate the total, but as soon as I do I realize that totaling the order in the factory that builds the view data violates SRP.

That’s easy enough to fix. I just end up with a factory that looks like this:

public class OrderInvoiceViewDataFactory{
  public OrderInvoiceViewDataFactory(IOrderTotalCalculator calculator){
    //...
  }
}

Now I mock that out in my spec:

public class When_creating_an_order_invoice_view_model_from_an_order : Specification {
  override public void Given(){
    // ..

    mockCalculator = new Mock<(IOrderTotalCalculator>();
    mockCalculator
      .Setup(x=>x.CalculateTotal(order))
      .Returns(3.14m);
    factory = new OrderInvoiceViewDataFactory(mockCalculator.Object);

  }

  [Then] public void the_total_should_be_the_sum_of_the_items(){
    // we only need to test what out mock returns
    invoice.Total.ShouldBe(3.14m);
  }

}

…and abracadabra…my spec tests nothing.

From a spec perspective it’s valid, but from a test perspective, all it tests is that my mocking framework works and I’m making that call to it like I expect.

So what do you do? All the test around the calculations move to another file and the spec  gets simplified to just test the interaction with the dependencies?

[Then] public void the_calculator_was_called_to_calculate_the_total(){
   mockCalculator.Verify(x=>x.CalculateTotal(order),Times.Once);
}

What I loose here is my nice test result that anyone can look at an understand. All my non-techy folks need to look at several different places to see if that order invoice is be calculated correctly.

You could make this an integration test and test the calculator as well, but I don’t like that idea either.

This is probably a sign that I’m not writing my specs correctly or I should identify this is going to be factored out sooner. This hasn’t been the case so far. Especially when we are translating something from a graph of objects to a flat representation  for a model in our MVC app, a lot of the work ends up being factored out or pushed into AutoMapper where we need to write a different tests for it.

When we are done with the refactoring, the original spec for the class is very skinny, which is good, but as I said, seeing the results of how the story is progressing is more difficult. Maybe that is where my focus needs to be. How do you tie the specs to a specific story and see the results in an aggregated manner?

I was explaining my dilemma to a Java friend and the e-mail ended up being fairly interesting, so I thought I would share it.

I’m sure Java has extension methods, but I’ll give it a once over just in case.  You can extend existing classes without sub-classing by extension methods.

public static IEnumerable<SelectListItem>
                            AsSelectListItems( this IEnumerable<Person> listOfPeople){
   IList<SelectListItem> selectListItems = new List
();
   foreach(var person in listOfPeople){
      selecListItems.Add(new SelectListItem(person.ID,
					    string.Format("{0} {1} {2}",
                                                          person.First,
                                                          person.Middle,
                                                          person.Last));
   }
   return selectListItems;
}

Right. So the this operator tells the compiler to basically create this function:

public static IEnumerable<SelectListItem>
                           AsSelectListItems(IEnumerable<Person> listOfPeople);

And invoke it every time you see this:

IEnunerable<Person> p = myService.GetAllPeople();
IEnumerable<SelectListItem> s = p.AsSelectListItems();

You probably knew all this, but it explains how we are using them. We end up calling a lot of extension methods off other extension methods.

myViewData.PersonSelectListItems = myService.GetAllPeople()
                                                   .AsSelectListItems()
                                                   .OrderedByText()
                                                   .StartingWithAnEmptyItem();

In this example we have three extension method calls and I think it really makes the code much more readable, but it makes testing this code a little more difficult. Obviously, the myService is a mocked dependency which will return a fake list of people, but the other extension method calls cannot be mocked, and the scope of the test bleeds a bit.

I like the syntax it gives, but now I’m going to test potentially four classes when I should be testing only one.

This gets pretty pandemic in our code. Especially trying to make Selenium’s API and the assertion API of Nunit and xUnit more fluent.

We’re using the Data Annotations Validators in ASP.Net MVC 2.0, but there wasn’t a really great way to unit test them. I’ve read Brad Wilson’s blog post about how to set them up and unit test them, but I didn’t like how you relied on the name of the property as a string.

// Arrange
var propertyInfo = typeof(Contact).GetProperty("FirstName");

It would be nicer if that used lambda expressions to get the name of the property, so I set out to do just that. Before we begin, I need to remind you, as always, I don’t know what I’m doing. This is my first tiny venture into expression trees so it is possible isn’t the best way to do things, but so far, it’s working for me.

I created a base Specification to wrap up a lot of the messy bits of getting property names by lambdas. I took a look at how the ASP.Net MVC helpers did it and roughly followed their pattern.

public class DataAnnotationSpecification<T> : Specification
{

    public PropertyInfo Property(Expression<Func<T,object>> op)
    {
        return Property(GetInputName(op));
    }

    public PropertyInfo Property(string propertyName)
    {
        return typeof (T).GetProperty(propertyName);
    }

    public static string GetInputName<TProperty>(Expression<Func<T, TProperty>> expression)
    {
        // not sure I totally understand this
        if (expression.Body.NodeType == ExpressionType.Convert)
        {
            var ue = (UnaryExpression)expression.Body;
            return GetInputName((MemberExpression) ue.Operand );
        }

        if (expression.Body.NodeType == ExpressionType.MemberAccess)
        {
            return GetInputName((MemberExpression)expression.Body);
        }
        throw new NotImplementedException();
    }

    private static string GetInputName(MemberExpression memberExpression)
    {
        return memberExpression.Member.Name;
    }

}

I know this probably doesn’t cover an exhaustive list of expressions, but we’re only talking about Property access tests here so we’re probably in the clear.

To make things a bit nicer, I also created some extensions off PropertyInfo to help setup their assertions.

public static class AttributeAssertions
{

    public static void ShouldNotBeRequired(this PropertyInfo @this)
    {
        @this.ShouldNotHaveAttribute<RequiredAttribute>();
    }

    public static void ShouldBeRequired(this PropertyInfo @this)
    {
        @this.ShouldHaveAttribute<RequiredAttribute>();
    }

    private static T PropertyAttributeOn<T>(ICustomAttributeProvider propertyInfo)
    {
        return propertyInfo.GetCustomAttributes(typeof (T), false)
            .Cast<T>()
            .FirstOrDefault();
    }

    public static void ShouldHaveAttribute<T>(this PropertyInfo @this)
    {
        PropertyAttributeOn<T>(@this).ShouldNotBeNull();
    }
    public static void ShouldNotHaveAttribute<T>(this PropertyInfo @this)
    {
        PropertyAttributeOn<T>(@this).ShouldBeNull();
    }
}

We only have the ShouldBeRequired and ShouldNotBeRequired right now, but you could see it will be easy to extend upon them.  The finished spec looks something like this.

public class When_testing_the_state_of_the_CustomerViewData
    : DataAnnotationSpecification<CustomerViewData>
{
    [Then] public void the_customer_first_name_is_required()
    {
        Property(x => x.CustomerFirstName).ShouldBeRequired();
    }

    [Then] public void the_customer_middle_name_is_not_required()
    {
        Property(x => x.CustomerMiddleName).ShouldNotBeRequired();
    }

    [Then] public void the_customer_last_name_is_required()
    {
        Property(x=>x.CustomerLastName).ShouldBeRequired();
    }

    [Then] public void the_email_address_is_required()
    {
        Property(x=>x.CustomerEmailAddress).ShouldBeRequired();
    }

}

That’s not so bad, and it stands up to any refactoring we might do.

We’re just starting down the path of serious and extensive testing with mocks and fakes. Our initial tests were all done using Rhino Mocks, mostly because it had a well established reputation and we had no reason not to give it a try. The syntax is beautiful and did everything we needed, until we tried to make a test similar to this one pass successfully.

public interface IFoo
{
    string Bar();
}

[Fact] public void TestWithRhino()
{
    var mock = MockRepository.GenerateStub<IFoo>();
    mock.Stub(x => x.Bar()).Return("bar");
    mock.Stub(x => x.Bar()).Return("baz");
    Assert.Equal("baz",mock.Bar());
}

We couldn’t, at least we couldn’t using strictly stubs. From what I could gather, we could if we used a more complicated for of mocking besides stubs. We only need stubs, or fakes, because we don’t need to test all the interaction, just the ending state.

Since we don’t have any serious stock in using Rhino, as an alternative I decided to take a look at using Moq. The syntax for Moq is not as nice. It is more wordy, but I do like how it doesn’t rely on extensions methods for everything. But what about the sample test?

[Fact] public void TestWithMoq()
{
    var mock = new Mock<IFoo>();
    mock.Setup(x => x.Bar()).Returns("bar");
    mock.Setup(x => x.Bar()).Returns("baz");
    Assert.Equal("baz", mock.Object.Bar());
}

It passes! Switching wasn’t that difficult, but where is the ugly syntax I‘ve run into so far? Well this in Rhino:

MockOrdersRepository.AssertWasNotCalled(x=>x.StartOrder(null,null,null,null));

…becomes this in Moq:

MockOrdersRepository   .Verify(x=>x.StartOrder( It.IsAny<string>(),                            It.IsAny<string>(),                            It.IsAny<string>(),                            It.IsAny<string>()),              Times.Never());

It might be I am not familiar enough with either to say that is the equivalent assertion of both, but it is working for us. I am probably missing something that is causing Moq to look more verbose.

We recently pushed a feature into production that had a pretty serious flaw. If gone unfound it could have caused headaches with billing the wrong customer the wrong amount and all kinds of other potentially damning effects. Luckily for us, we also had a second bug. The feature that was to send the notification e-mail that data was ready to be corrupted by our bug was misconfigured when it moved to the production environment. Lucky for us, no one was being told they should go execute our buggy code, so no one did and we found it, fixed it and all those actions items that queued up ran through without a hitch.

Now lucky for us or not, we did fail to find two serious bugs. It highlights the need for thorough acceptance testing. Thorough being the key word there.

The first bug existed if action items queued up and were acted upon in a certain order. Testing was done in this area, but the items never fell in the order where we found the bug. Would acceptance tests go so far to say:

• Act on the first item in the list and verify the actions were taken correctly.
• Act on the second item in the list…
• Act on a list with a single item…
• Act on a list with two items…
• Act on a list with n items…

How detailed do you get? 

The first bug was a pretty lame mistake, although I feel SQL should have been alerted when we attempted to do an udate  a single from a statement that returned multiples. I don’t know how it makes sense to choose the first row and go about your business. 

As for the second, testing the production configuration is hard. We do have some junk data laying around in the production DB that we can manipulate, and probably should have to verify the e-mails were sending. Config changes between the two are difficult to manage. Keeping the e-mail address, servers, roles, etc in sync to the correct environment is made simpler by branches, but verification isn’t.

The past few days have been spent getting Selenium setup and running as part of our CI builds. We only have a few proof-of-concept tests right now, but the power and potential is apparent. We have no automated unit testing right now and very little code in our code behinds so functional testing is as good as we’re going to get. Also, with the amount of AJAX that is currently in our pages, and the additional amount we see in them in the near future, Selenium’s method of control the browser helps us make sure our pages do and continue to behave like we expect.

Getting setup with Selenium was incredibly simple. The Selenium IDE Firefox plugin makes creating your first Selenium test a snap. Download it and install it and you’re off. I’m not going to walk you through creating the tests, because the Selenium web site has a pretty decent movie that’ll take care of you.

The output of the test recorder are HTML files that contain the instructions, aka “Selenese”, to re-execute the tests through both the IDE and Selenium core. Selenese is very simple by design. While we could run the Selenese tests through the core in our Continuous Integration server, I’ve decided to use the IDE to export the C# source for the tests I create. This allows my to put the Selenium test into an NUnit test harness, refactor some of the common setup and config, and also run the tests via Selenium Remote Control(RC). I know the refactoring goes against the Selenese simplicity, but I want to run my tests locally and with different credentials without changing every test. Also, running in RC lets me run the tests without having to watch the pages in the browser fly by and, someone day, concurrently run all the browsers I care about.

Here’s an example of what one of those tests might look like 

using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
using NUnit.Framework;
using Selenium;

namespace SeleniumTests{

   [TestFixture] public class NewTest {
	private ISelenium selenium;
	private StringBuilder verificationErrors;

	[SetUp] public void SetupTest() {
		selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://www.google.com/");
		selenium.Start();
		verificationErrors = new StringBuilder();
	}

	[TearDown] public void TeardownTest() {
		try {
			selenium.Stop();
		} catch (Exception) {
			// Ignore errors if unable to close the browser
		}
		Assert.AreEqual("", verificationErrors.ToString());
	}

	[Test] public void TheNewTest() {
		selenium.Open("/");
		selenium.Type("q", "fooberry");
		selenium.Click("btnG");
		selenium.WaitForPageToLoad("30000");
	      Assert.IsTrue(selenium.IsElementPresent("css=a[href=\"http://www.fooberry.com/\"]"));
	}
   }
}

Getting this to run in the Team Build is as easy as getting the regular NUnit tests running. First we would want to parameterize those tests a bit so we can pass different hosts, ports and credentials (if present). That’s not too hard and should be obvious.

We’re trying multiple CI builds. One build just compiles and runs the unit test code, which is currently this single test….baby steps.

[TestFixture] public class TheMostAwesomestUnitTestInTheWorld {
    [Test] public void ValidateOurCoreBeliefs() {
        Assert.That(true, Is.True);
    }
}

The other build compiles and runs both the unit tests and integration tests. We suspect this build will take a while and the sooner we could get feedback about the quality of the build, the better. We could push this build to a separate build agent too, but for right now they are all just backing up in the same queue.

The only missing piece of the puzzle so far is the Remote Control Server. The RC Server is a Java application that has to run on a machine with access to which ever browser you plan to test. Starting the server is simple enough. Just make sure Firefox.exe is in the path. If you forget, you’ll get a nice warning about it.

java -jar selenium-server.jar

It’s up an running and when you run the above unit test, you see the browser window fly by and the server log all the actions it’s performing on the browser.

Great! …if you want to stay logged into your server with the console open, but I didn’t. Luckily, creating a Windows service is pretty easy. There is a great tutorial out there that walks you through how to setup a similar Java application as a service. Just substitute the Selenium RC Server for the FitNesse server. Note that when I tried to install the Windows Server 2003 Resource Kit, on both Vista and Windows Server 2008, I got a compatibility warning. I ignored the warning since I was only using the two files and it worked great.

Now you can start and stop the Selenium RC Server via a Windows service, keep it running when you log out and start automatically when the server starts up.

Have fun.

After the disappointing ending to my previous post, I now have much success to report. Once we had the correct path to the MSTest.exe on the build server, I was able to publish the results of my NUnit tests. There are a few things to note.

1. You still need MSTest.exe on the build server. I guess that is OK since we, at our organization, want to allow for the ability to run MSTests as well. I’m not 100% sure that happens on the build server or the TFS server.
2. As of now, if the NUnit tests fail, it will not run the MSTest. This is because we are executing the NUnit tests in the AfterCompile target as suggested in the NUnit for Team Build example. I’m not really sure why you would do both in the same solution, but if you do, you won’t see failures from both. Only one at a time. I’m looking into getting both to run, but that isn’t going to be much of a priority since we won’t need to run both in the same solution.

I am really excited that we have this running with such little effort. Thanks has to go to the NUnit for Team Build guys.

One to step two when time allows.

I’ve been spending the better part of two days doing this same process over and over. It doesn’t feel like I’m testing anything of value.

I am not doing it in the C# 3.0ish way, but it is better to explain it that way. I have a class and a façade (MyFacade) that I want to extend and test.

class Foo {
     public string Name { get; set; }
}

class MyFacade {
     public IMyDependency Depend { get; set; }
}

interface IMyDependency {
     IList<Foo> GetAllFoos();
}

So I write my test first like a good boy person should. I know my façade should give me back a list of the Name properties from the Foo objects it knows about.

[TestFixture]
public class MyFacadeTests {
    [Test]
    public void GetFooNames_ReturnsOnlyName() {
        FakeDependancy fake = new FakeDependency();
        MyFacade m = new MyFacade() { Depend = fake };
        Assert.That(m.GetFooNames(), Is.EquivalentTo(fake.FakeFoos.Select(x => x.Name)));
    }
}

I need that FakeDependency, so I go ahead and create that. This would be a like my repository that will allow me to initialize the fake data.
class FakeDependency : IMyDependency {

    // My local fake repository
    public IList<Foo> FakeFoos = new List<Foo>{
                new Foo{ Name="A"},
                new Foo{ Name="B"},
                new Foo{ Name="B"}};

    public IList<Foo> GetAllFoos() {
        return FakeFoos;
    }
}

OK. So now that I have that, I add my GetFooNames stub to MyFacade.

class MyFacade {
    public IMyDependency Depend { get; set; }

    public IList<string> GetFooNames()
    {
        return null;
    }
}

Run my test. It fails. Good. Now to implement that method….uhh I just copy my assert statement:

fake.FakeFoos.Select(x => x.Name)

…replace “fake.FakeFoos” with “Depend.GetAllFoos()” and I get:

class MyFacade {
    public IMyDependency Depend { get; set; }

    public IEnumerable<string> GetFooNames() {
        return Depend.GetAllFoos().Select(x => x.Name);
    }
}

Build it. Run it. It passes. What did I really test since I just copied the assert statement into the implementation body and changed the source? Did I test anything? That my copy code does the same thing in two places? Am I wasting time?
I rolled it around with a friend and decided to take out the select in the assert statement and replace it with a declarative list so it would look like this:

Assert.That(m.GetFooNames(), Is.EquivalentTo(new[]{"A","B","C"}));

That make me feel a little less “copy and pastey”, but now I have to know my fake is setup to give me three letters for the names. Let’s add a little bit to the fake to make it easier to configure. I am hesitant to do this since I don’t want to add code to my test classes, but I think in the end it is more readable than a more raw configuration.
So I added this to my test class:

public FakeDependency With(int i)
{
    FakeFoos.Clear();
    for (int j = 0; j < i; j++)
    {
        FakeFoos.Add(new Foo());
    }
    return this;
}

public FakeDependency Named(string name)
{
    for (int i = 0; i < FakeFoos.Count; i++)
    {
        // We want them to be 1 based and not zero based
        FakeFoos[i].Name = string.Format("{0} {1}", name, i + 1);
    }
    return this;
}

And use it in my test like this:

[Test]
public void GetFooNames_ReturnsOnlyName() {
    FakeDependency fake = new FakeDependency().With(3).Named("Fake");
    MyFacade m = new MyFacade() { Depend = fake };
    Assert.That(m.GetFooNames(), Is.EquivalentTo(new[]{"Fake 1","Fake 2","Fake 3"}));
}

I’m a lot happier with that than copying the same code from the test to the implementation classes.

Recently I’ve been able to extend my unit test code coverage as I’ve been introducing more features into our code base. I know this is something that should always be done, but for whatever reason, that’s not the case. What I’ve come to notice is our current method of organizing our unit test makes adding new features and test for new features awkward.

Currently our unit tests align perfectly with the class they test. So for example, if we have a class:

namespace MyCompany.Domain{
     public class Foo{}
}

We have, or should have, a test fixture in a separate assembly that looks like this:

using MyCompany.Domain;
using NUnit.Framework;
namespace MyCompany.Domain.Tests{
    [TestFixture]
    public class FooTests{}
}

This is probably a pretty typical strategy for organization. It is intuitive where to find tests for a type. Were I’m loosing my taste for this method is as the solution grows to several assemblies or layers, adding new unit tests for a specific feature becomes a scattered mess of changes. If I need to add a new field, I add new tests in the UI.Tests, the Domain.Tests, Repository.Tests, Core.Tests, etc.  Makes sense though right? You have to pass that new field through all the layers. I’m ok with the number of tests I need to write, but I don’t like having to go to eight different places to run them when I want to test my feature. It would be nice to align them them to the feature, or tag them in such a way I could find them easily. It could be that we are over layered and that is the real problem. We could be an onion.

Shrek: Example? Okay, er… ogres… are… like onions. 
Donkey: [sniffs onion] They stink? 
Shrek: Yes…NO! 
Donkey: Or they make you cry. 
Shrek: No! 
Donkey: Oh, you leave them out in the sun and they turn brown and start sproutin’ little white hairs.

We are definitely an onion….or an ogre, but that’s a whole different analogy.

There is a nice article in CoDe Magazine by Scott Bellware that talks about BDD. Most of it I didn’t see as valuable to my every day activities until he started talking about SpecUnit .Net. It was, in theory, what I’ve been looking for. Align my tests with my features/defects/requirements / whatever. SpecUnit .Net does some nice human readable report generation that I don’t need, but why would I need a new tool when I could just categorize my tests in folders that line up with my real world problems.

That gets me close, but there is no way to run the entire category without using the command line. Still doable, but not as nice as ReSharper’s unit test runner. It would be nice to get a Run all tests in this category from the context menu, but I’m not sure if that would be easy to do across the different frameworks. Again it would be nice to save my unit test session so I don’t loose it every time I close the IDE.

While we’re dreaming, Maybe we could invert the relationship and make the class under test and attribute and the tests themselves could align directly with the requirements. Something like this for example:

[Test][TypeUnderTest(typeof(Foo))] public void Correctly_Calculates_Bar(){
    //...
}

Or we could even get fancy with generics:

[Test<Foo>]
public void Correctly_Calculates_Bar(){
    //...
}

The IDE, ReSharper or someone could let me run all the tests for Foo if I wanted. 

After Josh’s comments I thought I would add a couple screen shots of using the Category attribute to group the tests as needed. You can apply the Category attribute to the method or to the TestFixture.

    [TestFixture]
    public class MyClassTests {

        [Category("Add the ability to do work")]
        [Test] public void DoWork_WithNoArgs_DoesWork(){  }

        [Category("Add the ability to do additional work ")]
        [Category("Add the ability to do work")]
        [Test]
        public void DoWork_WithAllArgs_DoesWork() {   }
    }

You can select Categories from the unit test runner’s Group by: combo box.

It then will group your tests by their category. I did find the UI a bit buggy. At first it wasn’t applying the grouping, but after a couple attempts they started to show up.

Thanks again Josh. That will keep me content for the time being. Now if I could only inspect the entire solution for tests in the category I would really happy.