Thursday, 3 September 2009

Running all tests in solution using MSTest

There aren't many times I truly have something bad to say about MS. Its not that I'm a great fun, but I admit that in most cases they do get the job done (it just takes a couple of tries though).

However, there is one thing that always tick me when working with MS products and that their ability to create product which doesn't work well with anything else besides other MS products. Normally I'm not into conspiracies and would of written it off as simple mistakes, but I'm guessing I just have to much respect for MS developers.

And the thing is that this always happen at the really annoying small things when you least expect it, but when encountered really make you go *#*#*!*!$*!*#!.

and the story goes like this:

At one of my clients I'm helping setup the build server to run all the unit tests the development team is going to write (which is why I was brought in at the first place). After some initial work in configuring CC.Net (which didn't take long at all), I've wanted to add the unit test execution part. so I went on to search for a way to execute all tests in a given solution. Shouldn't be too hard right?


After touring the web for some time it appears the Mstest has, how unconventionally, left this somewhat trivial ability out of their tool set (they didn't forgot to put a button in the IDE for doing so). its easy enough to do it if you use TFS as your build server solution, but using anything else means that one need to feed MSTest command line with all the various test dll's explicitly and MANUALLY!

so here are some other options:

1) Taken from Stack Overflow (towards the end) - a way to semi automate this using msbuild based on some naming conventions.

2) MSTest command line can use the vsmdi as their execution input, so one can simply create a list with all existing tests and use that. However now one need to maintain that all knowing list.

3) write up a small utility which based on the data inside the sln file, produce a list of all tests dll's - shouldn't be too hard but who would like to do that?

and the last thing (which i think i will choose at the end )

4) kind of a brute force, but if MSTest can only receive one dll as input, lets give him one. And I'm not actually talking about actually writing all the unit tests inside a single DLL. I'm thinking about using ILMerge to consolidate all DLLS in a given output directory (production and tests) into a giant DLL and feed it into MSTest. (I just hope that it will be able to process that dll)

Really Really ANNOYING!!!!!


Anonymous said...

I have been trying to do the same, but no luck. I have 78 test in my project and they all pass when i run them from IDE, but when i run mstest from command line it only shows 38 of them and they all fail. It also gives a message of missing 'system' assemblies which i have put in my bin folder. Please let me know if you have solution for this.

Lior Friedman said...


I don't think I can tell what is wrong there (too little information). It does however sounds to me as an unrelated problem.
but if you want to investigate it further contact me by email lfriedmal at gmail dot com and I'll try helping out.

Grzegorz Pawluch said...

For me using mstest.exe with testmetadata parameter is enough

Design by Free WordPress Themes | Bloggerized by Lasantha - Premium Blogger Themes | Walgreens Printable Coupons