Wednesday, October 28, 2009

More about Bizmonade

I got some interesting questions about Bizmonade on Michael Stephenson's blog, which I'll answer here.

Duplication between Bizmonade and BizUnit/when to use one instead of the other/when to use both

"1. Would I really want to extensively test all orchestration paths with Bizmonade and probably have an amount of duplication in my BizUnit testing":
My long term ideal would be to be able to specify tests that can run under two different contexts: simulated XLang (Bizmonade) or inside the real BizTalk. This would remove the need for duplication and also allow better trust in Bizmonade - Bizmonade can be used for fast feedback while doing development, and the result of the test can then be validated by running it inside the real BizTalk. If the test has a different result in each context, then there is a bug in Bizmonade. However I did not spend any time on this idea yet because I wanted to first focus on the simulated orchestration engine (and also because I was not sure of the best way to implement that - a BizMock-like approach, or more like BizUnit, or another solution; I would be interested in hearing feedback from the community about possible ideas)

However, there would still be some duplication because there would be different levels of tests (unit tests vs integration tests), but that's unavoidable regardless of the test tools being used.
"2. Would I be looking to use Bizmonade for edge or obscure test cases which would be a pain to test with BizUnit " and "4. Could a key place for Bizmonade be where I usually have a chain of orchestrations which could be difficult to test with BizUnit, and I could test the orchestrations individually with Bizmonade "
I can think of 3 cases where Bizmonade can test things that would be complex (or impossible) to test with a test tool that depends on the real BizTalk (such as BizUnit):
  • Error conditions that are hard to reproduce with BizUnit (for example exceptions or delivery NACKs), because you don't need to "physically" recreate the error (for example by shutting down a server or deleting a folder), you can simply tell the Orchestration Simulator you want to "inject" an exception or nack.
  • Another case is anything related to timing (delay shapes and timeouts), where Bizmonade allows "mocking" the system clock and therefore run test cases involving long delays almost instantly.
  • Also, there's the possibility to test started/called orchestrations individually, which is not possible at all using BizUnit (it can only test the "main" calling orchestration). I think that's a particularly useful case, because it allows having smaller test that focus on verifying one specific thing; this makes it easier to identify the cause of a failure when tests fail.
I would also be interested in hearing suggestions on other things that are hard to test and that could be made easier by enhancing Bizmonade.
"6. Would Bizmonade offer a better way to ensure good test coverage for orchestrations ":
A problem I've had with Bizunit tests is that they take too long to execute, as BizTalk is not optimized (yet) for low latency. Because of that, I don't have as much time to write extensive test cases (they would take too long to execute). Instead, I usually "cheat" by testing more than one thing in the same test. This however is a unit testing "antipattern", as it makes it harder to identify the cause of a test failure. It can also lead to introducing defects in tests (when modifying a test because a requirement changed, the developer can remove some important part of the test by mistake).
Therefore, because of the execution speed of tests in Bizmonade versus the real BizTalk, it's easier to ensure good test coverage while still following the idea that each test should verify a single thing.

Trust in Bizmonade test results

"3. Would I be looking to trust Bizmonade enough to reduce the amount of BizUnit testing I do "
"5. What risk does running my orchestration outside of BizTalk pose. Might I end up dealing with issues which wouldn't happen in XLANG or missing conditions that would happen in XLANG ":

Probably not in the initial releases: there can still be obscure edge cases where BizTalk's behavior is different from Bizmonade (either because I incorrectly interpreted what BizTalk was doing, or because of a bug/omitted feature in BizTalk). In future releases, I hope to achieve something that follows BizTalk's behavior closely enough to be trusted (again, having a way to execute the same tests in the real BizTalk would help - in the meantime, I'll rely on verifying my assumptions with BizUnit/manual tests and on user bug reports)

Orchestration debugging

"7. Would I lose the ability to debug orchestrations ":
No, BizTalk's own representation for orchestrations is still there (in addition to Bizmonade's representation), so you can still debug orchestrations from the BizTalk orchestration debugger.
In addition to the BizTalk debugger, an experimental feature I've been working on is to allow debugging orchestration from Visual Studio by debugging unit tests (this will allow setting breakpoints and stepping in the .odx file) - but again, it won't disable the BizTalk debugger, it would only add another option.


"8. Am I going to have to pay for Bizmonade (longer term) "

No, although the site doesn't mention anything about licensing yet, our intention is to make this a free tool, and I completely agree with the commenter who said the tool needs to be open source as well. We have two obstacles in making it open source right now, but that should be resolved in a later release:
  • The code structure needs to stabilize: the code is still being refactored heavily, so it would be hard to make it available and start accepting patches for now (complex merges)
  • Licensing issues: We need to decide on an appropriate license (good for integration in closed/non-free projects, and compatible with other components on which Bizmonade depends).

Saving time in the development cycle

Productivity was my main goal when developing Bizmonade - by allowing developers to focus on a single change at a time (and ignore configuration issues), and by decreasing the time between a change and the confirmation that the change worked as expected or not. To take this idea to the extreme, I would even say that having automated tests is a very nice "side effect", but the main benefit is accelerating development and modifications to existing code.

Michael also suggested generating code for tests by analyzing the common conditions and branches within the orchestration. I'm not convinced on the idea of test case generation (the only thing a generated test can check is that the component under test still behaves as it did when the test was generated, including any bugs that were present at that time). I prefer the reverse development methodology, where tests are written first, and then an implementation is written to make the tests pass. That said, the XLang/.odx interpreter from Bizmonade could be reused for generating other types of code, such as a test "skeleton" generator, but that would be a completely different project.

1 comment:

  1. Hi Christos, Interesting post. Bizmonade looks really cool and will have a play with it shortly and send feedback. In terms of the generating of tests, actually that will be useful when you want to retrofit an existing code base with unit tests and will be a good timesaver.