On one of my last projects, I started using Fitnesse tests, primarily to give the business the opportunity to specify their test cases. After playing around with these tests for about 4 days, I, however, threw them out. The main reason for this was the fact that I had to write a whole lot of complementary code to get the Fitnesse tests to run properly.
With Fitnesse, you first need to specify input and output data for your test case. This means edditing a web page with data and writing fixtures in code to process this data. This takes some getting used to and, in my opinion, requires quite some coding. I am also used to refactoring (and renaming) quite a bit during development. But sadly, your Fitnesse test cases don't get refactored, since they live in a separate web application. I also gave myself some extra work by switching over from the slim test runner to the fit test runner. Apparently if you do this, your previously written test fixtures aren't compatible anymore. Something I found very disturbing.
After 4 days of writing Fitnesse tests, I had about 3 test cases finished, I had only a small bit of business code and a huge headache. So, I threw them out.
Instead, I started using MSpec, or Machine Specifications. It is used to write business driven tests, or employ BDD in your code. Now, although MSpec is good for doing BDD, I have noticed it's not because you are using MSpec, that you are doing BDD. You can still write tests the way you used to with NUnit for instance. MSpec is actually just a semantic layer on top of that. But still, it makes it easier to move towards a BDD kinda style of programming.
I must say I really like the way in which you write you MSpec tests, although it does have some downsides and some bits take getting used to as well. I don't think the MSpec tests are always as readable as you would wish. On the one hand, it's nice they make use of lambda expressions a lot, on the other hand, the indentation on those lambdas is just plain awful. It does make you follow an AAA style of testing. Although with MSpec they don't call is Arrange, Act, Assert, but Establish, Because and It. That's just different naming, in essence it's the same thing, right. Another downside is, you can't give these tests to your business users to write them up. It's still developer testing.
This had me looking at SpecFlow, another framework I tried out a couple of weeks later. The test cases you write are more in a business kind of style. They are also easier to write by a business representative (with some additional explanation that is). Once you read the SpecFow documentation you would think you can only test web apps with it in combination with Selenium, but actually, I found it's dead easy to test plain business code with it as well. As with Fitnesse, your SpecFlow tests require you to write some additional code to link your test cases with the actual code, but the good thing is, SpecFlow helps you a lot with this. I used it in combination with my good friend ReSharper and found that this goes very smoothly. If I haven't written any code for a SpecFlow test case, ReSharper suggests a dummy implementation for it which you can copy paste and alter as much as you'd like. That's neat. Only downside I see for now is it's inability to use it for real technical tests. It's not technically impossible, but that's not what it was designed for. Your SpecFlow tests should always be backed up by technical NUnit test cases. That's why it's recommended that you read the Cucumber backgrounder on how to write test cases. The do's and dont's are pretty important here.
I'm actually still playing around with SpecFlow for now. MSpec has already proven it's worth, as did plain NUnit tests. But I think it would actually help a lot to write even more SpecFlow test cases in close communication with the business. I think this is a framework that can deliver.
Thanks for your write-up. As the maintainer of MSpec I'm obviously biased :), but I share your findings.
ReplyDeleteMSpec is a good fit if you're not very experienced in testing and want a blueprint how to structure your tests. It also forces you to think and express the specification in terms that not only the business, but other developers as well can understand. As you said, you can follow the same context/spec pattern with NUnit, but from my experience not a lot of people do that. I've seen NUnit tests like "TestXmlSaveAndLoad" which... did a lot more things than saving and loading XML.
Regarding SpecFlow, I use that as well. But not all my clients have a Product Owner that's willing to supply specs in the form of Gherkin. Does a dev team neccessarily need to go through the effort and write the spec? They could possibly use just a tool like MSpec and work closer to the code.
That said, every tool has its place. It's good to have choice!
Interesting write-up. I use all three testing frameworks *for different reasons*. MSpec is great for unit testing. It's not just different naming of arrange/act/assert. It actually guides you to test scenarios with one assert per test, whereas with traditional unit test frameworks such as NUnit, you often end up with several asserts per test. MSpec also supports clean subclassing of tests and calls the context setups correctly.
ReplyDeleteWith Fitnesse, I think you gave up too quickly and/or didn't spend enough time learning it before writing your tests. It sounds like you didn't really understand which fixture to base your own fixtures on. Or maybe you were reinventing the wheel by creating a rich fixture when there is already a better fit in a stock fixture that already does most of what you want.
I've written hundreds of FitNesse fixtures over the years and they almost always are very trivial and can be written in a few minutes.
If you still have problems writing fixtures once you understand FitNesse well, it's likely that it's highlighting a problem with your business logic code. Well-written business logic is easy to test. This occurs when the FitNesse tests are written first and the business code is written to get the tests to pass in traditional test-first style.
The huge benefit of FitNesse is that it is very approachable and friendly to the non-programming stakeholders in a project. Not only can they read and understand the tests easily, but they can also edit them and create new ones. It provides a great document for the working application, both for programmers and business people.