Saturday, April 4, 2009

My BizTalk vNext (post-2009) wish list

Michael Stephenson's recent thoughts about BizTalk vNext has inspired me in gathering my own BizTalk wishlist.

My wish list is categorized as follows:

  • Usability / Productivity
  • Explicitness / Clarity
  • Deployment
  • Debugability

I have given great importance to usability/productivity enhancements: BizTalk has reached a good level of architectural maturity, but it has neglected features that would allow developers to be more productive, and I hope the next version will improve on this aspect. (Note that I'm not talking about adding more wizards or drag and drop tools, but on enhancements that would improve flow)

Another important aspect is deployment. It should be as painless as possible to support several upgrade scenarios, to enable iterative development where upgrades can be done frequently and with minimal downtime.

Usability / Productivity


Title Description
Make variable/message creation more efficient
  • Allow entering a variable/message type with the keyboard, using IntelliSense. Finding a type in the type picker dialog can be annoyingly painful to use when you already know which type you want.
  • It should be possible to create a variable/message, name it and set its type without touching the mouse.

Example screenshot of IntelliSense for variable/message types and mouseless variable/message creation.

Wizards: remember previous values
  • After a wizard is run (for example the WCF Service Publishing Wizard), it should save everything that was entered in the wizard to a file (a DSL defined with Oslo?)
    WCFService
    {
    PublishOrchestration "MyProject.Orders.ReceiveOrders"
    MetadataEndpoint true
    TransportType WCF-WSHttp
    TargetNamespace http://tempuri.org/orders/receive
    Location http://${server.host}/services/ReceiveOrder/
    ...
    }
  • This file should be included in the project and checked into source control.
  • When the same wizard is rerun for a project, it should load the previously entered values (from the saved "wizard DSL" file). This would avoid re-entering the same thing 10 000 times when experimenting with a wizard (better learnability). It would also allow future updates to a project to be done without restarting from scratch every time a change is needed (for example, adding a new operation to an existing orchestration).
  • Ideally, the only thing a wizard should generate is the "wizard DSL" file containing the values entered by the user. It should NOT generate type-holder orchestrations, schemas or binding files. Instead, these artefacts should be generated at compile time from the "wizard DSLs" file and should NOT be part of the project.
    • This is probably not possible with the artefacts that are currently generated, because some of them need manual tweaking after they are generated. In an ideal solution, there would be no need to tweak the generated code (the necessary adjustments could be done in another way, for example by extending the "wizard DSLs"), so the generated artefacts would not need to be visible to developers anymore.
  • Similarly, the wizard should not deploy anything to IIS. Instead, a command line tool should be able to "import" the wizard DSL definition into an IIS web site / virtual directory.
  • Also, ideally, should get rid of the wizards and instead design corresponding DSLs that can be edited efficiently (and with good learnability) in a text editor with IntelliSense. (The goal of removing wizards is that the underlying DSL should be simple enough to manipulate directly, so that wizards would not be reallly necessary anymore)
Orchestrations: Get rid of the XML representation
  • Tools like Oslo's MGrammar would make it easier to directly manipulate XLang/s code, making it unnecessary to also have an XML representation.
  • The XLang/s representation is much more readable than the XML
  • The XLang/s representation by itself would be easier to work with merge tools. Currently, resolving conflicts/merging changes in orchestrations is very painful because the XML is not merge-tool-friendly.
  • XML representation of orchestrations is mostly a repetition of the XLang/s code. Some information is only in the XML, but it could be moved to the XLang/s by using code annotations.
  • Some tools will still need access to the XML, so there should still be a way to programatically generate, in memory, the XML representation (as it can be currently be done with XSharpP.exe). However, the XML representation should be hidden from orchestration developers, it should not be stored in source control and it should only be generated on the fly when needed by a tool (this means all tools that manipulate ODX files would need to be updated to do this conversion)
  • Another benefit of this is that it would help solving a frequent bug we encounter: while editing an orchestration, some parts of the code becomes invalid and replaced by "#error "The expression you have entered is invalid". When this happen, we have to do some modification in the orchestration designer that will force it to regenerate XLang/s code from the XML representation. If the designer worked directly with XLang/s code instead of converting to/from XML, this source of potential conversion problems would disappear.
Add a "Refactor > Extract Orchestration" feature

Example screenshot of suggested "Extract Orchestration" tool

  • Would be similar to C# refactoring tools that have a "Refactor > Extract Method" feature. It would greatly help in breaking down complex orchestrations into easier to understand "sub-orchestrations". Example use case:
    • User selects one or many shapes in orchestration and activates the "Extract Orchestration" tool.
    • User enters the new orchestration name and select the invocation strategy (Call Orchestration Shape, Publish Message to Message Box, Publish Message to Partner Orchestration)
    • User assign parameters names to the new orchestration
    • Extract Orchestration Tool generates the new orchestration with all appropriate parameters, and moves the selected shapes to the new orchestration
    • Extract Orchestration Tool replaces the shapes in the original orchestration by a "Call Orchestration Shape" or a "Send Message" shape, depending on the invocation strategy selected by the user.
    • If one of the "Publish Message" invocation strategies was selected, Extract Orchestration Tool generates appropriate schemas to pass values to the new orchestration.
  • The "Call Orchestration Shape" strategy is probably simpler to implement and would be a great start... other strategies could follow in future versions.
Pipeline designer: add pipeline components from "project reference" instead of toolbox.
  • Allow adding pipeline components without copying them to "c:\Program Files\..." and adding them to the toolbox. Many components are written to be used once (for a specific pipeline), so having them in the toolbox brings no value. Also, having pipelines in the toolbox leads to "file already in use" errors while recompiling projects (requiring a Visual Studio restart).
  • It should also be possible to reference pipelines by "project reference" instead of references to a compiled assembly. (Project references would be converted to assembly references when the pipeline is compiled)
Allow "commenting" orchestration shapes or group of shapes. This could be useful when doing major changes to an orchestration, to temporarily disable parts of an orchestration and ensure individual changes still compile.
Enhance general usability

Writing code using a text editor (with features like IntelliSense, refactoring...) is generally more efficient than using a graphical editor (such as BizTalk's orchestration designer, mapper and pipeline designer). This doesn't mean that an efficient graphical code editor is not possible, but it's still a huge unresolved challenge. Considerable user interface research must be done before reaching that point.
This research would have to consider usability principles such as "Keystroke-Level Model", "Flow", "Recognition rather than recall", "Locus of attention", "Modeless interfaces", and doing major experimental usability/efficiency studies.


Explicitness / Clarity

Title Description
Add a "Initialize Correlation Set" shape to orchestrations
  • In some cases, we need to initialize a correlation set with a value that is not in a promoted property of the received or sent messages. It could be in a non-promoted property, or it could be a value derived from the message.
  • The current solution is to send a dummy message with the values that will initialize the correlation set and have some way to redirect the dummy message to "/dev/null" (for example using a Null Adapter). This solution adds a lot of noise to orchestrations and it does not explicitly show that the goal is to initialize a correlation set.
  • A better solution would be a "Initialize Correlation Set" shape, which allows to assign the result of expressions to each property of the correlation set.
    Example of proposed Initialize Correlation Set shape
    corrEvent.EventId = configuration.deadlineReachedEventId;
    corrEvent.EventDate = deadline.ToString("yyyy-MM-dd");
Allow filter expression on non-activate receive shapes
  • This would give the same result than the "Initialize Correlation Set Shape" feature (create an instance subscription with the appropriate filters).
  • Having both options could still be useful: developers would be able to pick the option that allow the clearest representation for their context.
Never truncate any text in the orchestration editor.
  • Unclear names considered harmful:

Orchestration shape with truncated text

  • If text was not truncated in the shape's name, these names could be used to better document what the orchestration is doing.
  • It would be better to dynamically resize shapes based on their name (to allow the full name to fit), than to truncate the name.
Allow adding "documentation comments" to orchestrations
  • Another way to better document what an orchestration is doing would be to allow entering annotations.

Example of inline annotation in orchestration

  • It should be possible to edit the annotation directly in the designer (not in the property editor or a popup window)
  • These annotations would fulfill a similar purpose than the "Documentation" property, but would be visible directly in the designer (and in tools such as BizTalk Documenter). This avoids the need to click on each shape one after the other in case one of them has a useful comment.
Binding files: use "configuration by exception" for exported bindings.
  • 99% of the contents of binding files is noise: it simply repeats the default values for all properties. This make them hard to maintain. In our projects, we use another file to "configure the configuration file". This file contains only the values that can really change. We also have binding templates which reference values from the configuration-configuration-file.
  • A better solution would be to have simplified binding files that are created with a "configuration by exception" philosophy. These files would only contain values that are different from the defaults. (For this to work, the binding file export tools would need to have a way to check the default value for each property, and only export those that are different from this default.)
Binding files: Don't escape any XML
  • Many sections in binding files contain escaped XML. Some of these sections even contain "double-escaped" XML. This makes them hard to read and maintain. It can be easy to forget to change a small environment-specific value because it's buried in a long line of double-escaped XML.
  • A solution that's already available is to use the ElementTunnel tool from Scott Colestock's deployment framework.
  • A better solution would be to integrate the ElementTunnel tool in all BizTalk tools that import/export binding files, to make it transparent to developers. For example, when a binding file is exported, all XML should be automatically unescaped in the generated XML file. Similarly, when a binding file is imported, all XML should be automatically re-escaped before being processed by BizTalk. (It should also support older binding files where XML is already escaped)
Allow readable XPaths in every place where an XPath expression can be entered
  • In a BizTalk solution, most XPaths have the following form:
    /*[local-name() = 'Test' and
    namespace-uri()='http://tempuri.org']/*[local-name()='Hello' and
    namespace-uri()='http://tempuri.org']
    /*[local-name()='World' and
    namespace-uri()='http://tempuri.org']
  • This is because most of BizTalk tools don't support namespace aliases, and also because this is the way these tools generate XPaths
  • Because of this, many developers who first learn XPath when working with BizTalk jump to the conclusion that XPaths are an unredable variant of black magic.
  • It would be better if all BizTalk tools allowed XPaths in the following format (equivalent to the previous example when aliases are correctly set):
    /t:Test/t:Hello/t:World
    For this example to work, there would need to be a way to configure namespace aliases. For example, orchestrations and maps could have properties to assign aliases to namespaces URIs. When assigned, any generated XPath should use the shorter form with the user-assigned alias.

Deployment

Title Description
Create a MSI file WITHOUT first deploying to BizTalk The goal of a MSI file is to deploy, so needing to deploy in order to generate the file that will be used to deploy is a "paradox".
  • This diagram summarizes what is conceptually wrong with the current "automated" deployment process:
    BizTalk "automated" deployment process
  • This may work for an individual developer who tests on his own machine and then generates a MSI for deployment to production.
  • But it doesn't work in a team using Continuous Integration, where we need to:
    • Obtain the latest source code from source control
    • Build the solution
    • Deploy the solution, using the same process that will be used to deploy to production
    • Run automated tests on the deployed solution
    • Deploy to production using an installer that was successfully tested in the Continuous Integration environment
    Using the same deployment process that is used for production is important, because it is critical to identify problems in the deployment process before going to production. (Deployment to production should not be the first opportunity we have to test the real deployment process, to minimize downtime caused by problems in this process).
  • Because of this flawed way to generate MSI files, it is useless to us. Instead, we have developed complex scripts that deploy artefacts to BizTalk applications. These scripts are used in the Continuous Integration environment, and they are also used to deploy to production.
  • We therefore lose access to useful features of MSI files (rollback, uninstall, dependency management,...)
  • A good solution would be to have a way to directly generate a fully functional BizTalk MSI file, without needing to first deploy to BizTalk (using WiX?)
Have a way to replace a deployed schema, without undeploying assemblies that depend on it, and without changing its target namespace
  • If a schema is shared by multiple projects (as a "contract" to allow them to interact in a decoupled way), any updates to the schema mean that all dependent projects have to be undeployed.
  • For some schema updates, it is not desirable to undeploy all dependant applications. For example, if an optional element or promoted property is added, we currently need to:
    • Disable receive locations of dependent applications
    • Wait for orchestration instances of dependent applications to end (or terminate them, if acceptable)
    • Undeploy all dependant applications
    • Deploy the updated schema
    • Redeploy dependant applications and enable their receive locations
    • Manually reinject messages to resume orchestrations instances that were terminated.
    This may be appropriate when a mandatory element is added, but not for an optional element: we may have 5 projects that depend on the schema, and only 2 of them need the new property.
  • A solution would be to define a new version of the schema (by incrementing a version number in its "target namespace"). However, if an orchestration sends a message with version 2, it can't be received by another orchestration which still expects version 1. Therefore, it means that, in practice, either all orchestrations must still be updated at the same time, or that we need to setup a "version mapping" process to automatically map versions of a message to the appropriate version for its recipient.
  • Another solution is to keep the same target namespace, but increment the assembly number. In that case, BizTalk will take the schema in the assembly with the highest version number.
    • This works in pipelines and in filter based message routing solutions
    • But it fails in orchestrations: an orchestration expects the class associated to the message's .Net type. This class must have the exact version the orchestration expects. If the orchestration was compiled to accept message v1.0.0.0, and it receives an instance of message v1.1.0.0, it will throw a InvalidCastException.
  • A solution would probably require major reworking in BizTalk's internal deployment functionality, but it would greatly improve the ability to update applications in production.

Debugability

Title Description
Allow Visual Studio debugger to step through XLang/s code (when attached to BTSNTSvc.exe)

Example of orchestration being debugged in Visual Studio debugger

  • The BizTalk orchestration debugger can be used to debug orchestrations at a high level (see which shape executed and in which shape an error occurred).
  • It can sometimes be useful to debug orchestrations at a lower level (individual expressions in the .odx source)
  • This can be done by attaching the Visual Studio debugger to BTSNTSvc.exe, but only if the hack described on SymbolicDebuggingForOrchestrations was done. This hack will allow stepping through the generated C# code for the orchestration. However, the generated C# code is not designed for readability, so it can be hard to understand.
  • A better solution would be to include, in the C# code generated from orchestrations, "#line pragmas" to correlate each C# line with the original line in the .odx file (XLang/s code)
  • The solution should automatically do all the necessary steps described in the SymbolicDebuggingForOrchestrations article, instead of requiring developers to do them manually (or write their own scripts to do it). This would be done when the project is compiled in Debug (Development) configuration, but not in Release (Deployment) configuration.


Some of these may require ambitious/complex changes that may break legacy tools, so I doubt that all of them will be implemented. But it would still be nice if a few could be implemented. Also, some of these enhancements/features/critics are also relevant with WF and Dublin, where it's probably not too late to start applying them.

4 comments:

  1. You said it! Every one of these is on my team's wish list.

    ReplyDelete
  2. Hi Christos, you right about the paradox: Deploying to generate an msi to then deploy is against the real nature of CI. In fact, i have this problem at PSP, we are in the process of converting everything to Biztalk 2009 and our build server does have biztalk installed. We looked for solutions to create it without exporting it first and we noticed there were really few solutions out there.

    We are actually doing a spike on the Codeplex Deployment framework to generate MSI without Biztalk. Do you know if there are any other way actually to do it in 2010?

    I have seen also the biztalk guruz blogical but this one also requires biztalk installed.

    Continue your great work, i have added your blog in my favorites. Anxious to see your next articles.

    Patrik

    ReplyDelete
  3. Hi Patrik,

    Last time I used the Deployment framework, it also had the same problem of needing to first deploy into BizTalk to generate a MSI (it does not directly generate it). That was a few years ago, so I'm not sure if the framework has evolved to resolve that problem (by directly creating a MSI without first deploying into BizTalk)

    This means you would still have the same problems with this framework:
    - long delays to build/package application
    - complexity in undeploying / packaging / redeploying interdependent applications in the right order
    (complexity for CI builds that create and test a full package)

    But I still think using that framework instead of homemade scripts would be a good idea, because you will benefit from the solutions implemented by the framework, without having to reinvent everything (which can get very complex)

    About VS2010/BizTalk2009R2, I have not seen any new features that would make deployment simpler, so I don't think the issue will be resolved soon.

    ReplyDelete
  4. Yes, the Deployment Framework from CodePlex allows to Create an MSI using WIX without first deploying the package into Biztalk.

    ReplyDelete