ESB Pack vs. only BizTalk and ESB Toolkit
The full white paper can be downloaded here. Below is the short summary for this publication.
The ESB Feature Pack was developed for enterprise customers who are looking to modernize BizTalk Server with new monitoring and functional capabilities to support deployment on-premise or cross-cloud. It contains unique set of features that do not exist in other products related to BizTalk.
This feature pack allows customers who cannot use cloud due intellectual property concerns or regulations to achieve complete privacy for operational data. Customers can choose where to store metrics and how to process and analyze them. It includes integration with open source software for processing time series and data visualization widely used in industry today for monitoring complex cloud infrastructures and heterogenous systems on-premise.
The ESB Toolkit is included in BizTalk Server and provides many dynamic features, including routing transformation and exception management framework for implementing enterprise service bus pattern. It is logically a set of frameworks and pre-built components for designing and implementing Routing Slip pattern. The toolkit itself is a set of frameworks, which do not include telemetry or monitoring capabilities for receive locations, send ports, failed messages or itineraries. It provides optional BAM tracking for collecting basic events for messages that use BizTalk failed routing and sequence of itinerary steps.
This feature pack is an addition for BizTalk server and includes major enhancements for ESB Toolkit messaging and operations. It is logically composed as two separate products for modernizing ESB Toolkit capabilities and active monitoring. The pack does support active event collection using Event Tracing for Windows (ETW) and integrated to store events in time series and relational databases. The event tracing is not limited to failed messages and itineraries. It allows seamlessly replace monitoring feature from AppFabric for Windows Server and collect events from WCF, WF, IIS and extensible to support other event sources.
The ESB Toolkit natively fits into existing BizTalk Server architecture to execute dynamically steps defined itinerary from pipelines and orchestrations. It includes message interceptor pipeline components to execute steps defined by itinerary in pipelines and uses its runtime to trigger orchestration as a step when necessary. ESB Toolkit installs with multiple databases for storing failed messages and itineraries.
The ESB feature pack follows the architecture of the toolkit through extensibility to fully comply with BizTalk Server architecture. It includes new components and message interceptors to extend itinerary-based capabilities with new features for runtime and design time. Using message interceptors from this feature pack is optional to enable messaging scenarios not supported by ESB Toolkit.
In addition, it follows cloud native approach to integrate with modern monitoring (Prometheus and InfluxDb) that can be deployed anywhere: on a host machine, Docker or Kubernetes:
The ESB Toolkit applications can leverage BizTalk deployments with zero downtime, easy versioning and reusability provided by itinerary-based capabilities. In addition, dynamic transformation and routing can be used as a service without message submission to the message box database.
The ESB Pack extends the toolkit with granular set of features and capabilities to enable dynamic and complex messaging flows. It can be used by any application that uses BizTalk as enterprise service bus. These capabilities include enhanced itinerary flow control, branching, composition and exception handling. The monitoring features included in this pack significantly reduce time and cost developing itineraries. In addition, this feature pack includes new services, itinerary designer extensions and samples to illustrate how to use them.
Development and Testing
The itinerary debugging and traceability in ESB Toolkit is challenging. Developers and administrators use DebugView or similar tool to collect traces emitted at runtime. While one or more messaging services can be verified with unit tests, the integration testing in single or multi-node cluster is difficult due missing comprehensive telemetry that is always emitted, unlike BizTalk BAM.
The ESB pack significantly enhances development and testing by emitting and collecting multiple types of telemetry, including user defined events emitted by developer code for troubleshooting. Thus, integration tests can be easily automated to validate functional scenarios with telemetry. In addition, the effort needed for performance testing can be reduced by analyzing duration of itinerary flow as a whole and for each step during the development.
Operations and Data Visualization
The ESB Pack includes integration for sending collected metrics to event stores and data visualization.
The monitoring features included in this pack allow complete symmetry for monitoring BizTalk ESB, WCF, WF,IIS on-premise or in the cloud. This feature pack also enables collecting and storing metrics from BizTalk Server analytics, ESB Toolkit telemetry, WCF, WF and IIS in relational or time series databases. It replaces AppFabric for Windows Server monitoring capabilities to collect metrics from any applications built using .NET platform using WF, WF without any configuration changes.
For example, using Grafana software, performance engineers can easily create powerful visualizations and use alerting capabilities for multiple channels (e-mail, Slack, Telegram, etc.) In addition it supports visualization of data stored in SQL Server, for example exception management database.
The ESB Toolkit is part of BizTalk Server as set of frameworks and pre-built components for using dynamic capabilities for dynamic transformation and routing. The exception management framework from the toolkit can be utilized by any BizTalk component. It is suitable for medium to large size applications that require reusability, low cost of changes, zero downtime deployment and service discoverability. While frameworks included in the toolkit are very extensible, they remain not well known and cost of operations can be challenging.
The ESB Pack was built for developers and system reliability engineers. It extends the toolkit to reduce time and cost of delivery for applications that require using BizTalk dynamic capabilities as well as low cost of operations for medium and large deployments. This pack includes integration for storing telemetry events in time series or relational databases. Using time series databases follows IoT approach to enable integration with open source tools for anomaly detection, alerting and data visualization. The design of this pack enables customers to achieve data privacy and symmetry for operations on premise and the cloud using hosting the series database in datacenter or as SaaS offering in the cloud (AWS or Azure).