Sometime I need to retreive complex XML structure data from a database, in this case I develop a pipeline component that intercept the data content in WCF  resulset.
In this way I can use any XML syntax in my storeprocedures without problem.

We can create any type of structure in a storedprocedure using XML AUTO command and affinity instructions.
When the WCF adapter retreive data returns this structure formed below

The pipeline component filter the resulset of WCF SQL adapter and retreive the  internal XML resulset as below

Where is the advantage?, in this way I can retreive any complex data structure type from database in only one step.

Steps:

Generate your XML data structure in SQL and generate your schema using the Add Generated Item, Generated Schemas, Well-Formed XML, from Visual Studio
Create a root node and put alll schema content in, as below

Use the pipeline component is very simple, put the component in your stage as below

Set the properties in pipeline component

Namespace:
The namespace you want assign to the message

RootName:
The root name node of thee message

WcfXmlPath:

The XPath expression to use to retreivee the internal and recursive resulset of the WCF SQL adapter
Finally create a port and use the pipeline

You can download the pipeline component HERE

In SAP this file type is called BBSEG, his structure is complicate and the documentation is poor and not so detailed.

The main structure is formed from a Session header record, Header data and Subsequent data

These structures contain data from the master records and the line items. This includes, for example, the line item amounts.

for more information you can find all documentation is SAP help here
http://help.sap.com/saphelp_erp2004/helpdata/en/e9/cb8a85eb0011d184650000e8a6bfbe/frameset.htm

Master record type table

Main document structure

All of these are composed from a more of 500+ different fields

After two days of work I recreated the same structure in XSD schemas to use this in a important project

Actually I using this schemas structure in a important BizTalk solution

You can download the schema project here

http://sdrv.ms/10KzL3f

In this part of this article will start with the most important aspect the Detail Design.

The Detail Design describe in detail the high level architecture of the solution, graphically and with descriptions in detail.

Detail Design

Detailed design can be divided into three parts based on architecture.

1. Presentation Layer: In presentation layer, we will describe the messages flow layer from all actors, for example:
   the messages will be received from site1, http post . In order to receive messages from site1 we need to build site1 using technology X.

Describe the work Items for Site1:

1. Create new user profiles.
2. Modify this feature site
3. Upload file

2. Business Workflow Layer: In this layer we will describe graphically the messaging flow layer

GRAPHIC DIAGRAM

Describe all steps in the flow, for example:

A. Create New Order: Partner1 will send request either using http URL provided by Hub.

B. When New order is posted to site, ASPX page of site will drop message to MSMQT location on BizTalk Server.

If exist describe particular security aspects, for example,
This request will be encrypted using X.509 certificates.

Describe messaging typology and structure, for example,

Sample EDI Header:

After the Design Detail we will start to describe the others important aspect.

Exception Handling

Describe the methodology used in the solution covering all of the parts, coding, orchestration exception strategy, if necessary graphically too, for example:
In all use cases have Orchestration and there is no custom code involved in it.
Orchestration will have exception handler as shown in figure.

Integration

Describe the most relevant aspect about integration strategy, for example:

BAS Integration: B2B scenario will use BAS for Trading Partner Management.
Trading partner management is done using website provided BAS feature.

BAM Integration: BAM will be used to send alerts and notifications to vendors and broker.
Information required for BAM will be tracked using Tracking Profile Editor as shown in figure.

Other important aspects to cover are:

Boundary Conditions

In this section we will Identify all boundary conditions implied by this design, or to be covered by this design

Security

In this section we will Identify all boundary conditions implied by this design, or to be covered by this design

Authentication

In this section we will Identify all authentication strategy

Authorization

In this section we will Identify all authorization strategy

Impersonation

In this section we will Identify all impersonation strategy

Performance

In this section we will Identify all performance requirements, for example
According to functional specification 1.000.000 orders will be processed End-End per day.

The most important aspect is identify the possible messages peeks, for example

According to functional specification 1.000.000 orders will be processed End-End per day with a peek of 500.000 between 2:00 AM and 3:AM.

Globalization/Localization

In this section we will describe possible Globalization/Localization strict

Test Support

If functional specification contains specific items required to enable testability.
This section identifies specific work and impacts on the algorithms necessitated by test support

Manageability

List policies and settings the admin and IT professionals can set to affect this features behavior.

Supportability

List considerations for supporting the feature: things like tracing, diagnostics.

Serviceability

List any issues and considerations for updating this feature set after it’s been deployed. Do we need to stop the service to replace the component? Do we need to reboot?

Migration

In this section we will describe possible strict and requirements about migration

Multi-Machine Installation

Create and refer to an architecture diagram.

MulTi-Domain Installation

Describe the domain installation, for example:
The scenario will have two domains. One domain for front servers facing outside world.
Another domain will be used for rest for servers which are behind corpnet firewall.

Runtime Considerations

Tuning and Throttling

List considerations on tuning and throttling.

Large Messages

In this section we will list all possible large messages peeks

Multi-Message Box

If exist describe the multi-message box strategy, configuration, scopes

Tracking

Describe the tracking strategy of the solution, for example:
Tracking will be enabled for BAM. Body tracking will be enabled for this scenario.

Threading and Concurrency

Describe possible threading and concurrency issue

Fault Tolerance

Describe the fault tolerance strategy, for example:

SQL Server will be clustered in active-passive node.
Minimum scenario will have 2 BizTalk for fault tolerance.

• Introduction

Is composed from three areas

• Associated Documents
  • List of the documents associates to design specifications, normally
    • Pattern (B2B, BPM ) Functional Specification
    • Customer Scenario Specification
    • Some particular documents as
    • TPM Specification
    • BAM Alert and Notification Specification
• Supported Features

  features/functionalities supported in Scenario for example:

  • Support EDI orders over SFTP
  • Support for CS catalog, order and inventory adapters
  • Create or Update orders either using http post for given URL.
  • Check and update inventory of items.
  • Update User Profile for Brokers and Vendors.
• Definitions
• List all particular definitions used in the document, for example:
  • B2B: Business –To- Business
  • TPM: Trading Partner Management
  • CBO: Common Business Object.
  • BAM: Business Activity Monitoring.
  • BAS: Business Activity Services.

Design Overview
List and design, if need, the most important aspects of the solution design, sample:

The general design consists of an orchestration that:

Macro Requests steps

• New Order coming from Broker.
• Update Order coming from Broker.
• Update user profile from Broker.
• Update Inventory from Vendor.
• Check Inventory from Vendor.

Describe the formats type:
These requests will be received by hub in different formats such as EDI, Flat Files and XML using different transports like HTTPS, SFTP etc.

Describe the macro steps:

A. Create New Order: Broker will send request either using commerce server website or will post to http URL provided by Hub.

1. New Order using commerce site: Broker will create order using commerce website. This order will be received ………

2. New Order using HTTP Post: Some broker can send their order using HTTP URL specified by Hub…………..

3. EDI Type Orders: EDI type orders will be uploaded to SFTP site. EDI Type orders will be picked by File Receive location. ……..

B. Update Order: Broker will send request either using commerce server website or …….

1. Update Order using HTTP Post: Some broker can send their update order using HTTPURL specified by Hub. …….

2. EDI Type Order: For EDI Type change orders will be uploaded to SFTP site. . EDI Type ………….

Present the design graphically

Architecture and Topology
shows over all architecture of scenario, a sample below:

Platforms/Portability Issues

Where the solution can work, a sample:

This Scenario will only work on BizTalk Name and version, Commerce Server Name and Version.
or
In case of (A), nothing special is needed on the operating system other than bla bla bla.
In case of (B), it will require certain level of setting for MSDTC for windows [version x] (not Windows version x)

Assumptions and Dependencies

Following assumptions are made for your scenarios release:
a.    Where development of scenario will be done.
b.    Adapter for will be used for development
c.    BAM alert and notification Service will be used
d.    BAS Trading Partner will be used

Partners and Dependencies

List all partners and dependencies types, logical, application and technological, a sample:
Adapters dependencies?
There is a dependency on the SAP Adapter and the MQ Series adapter –
The SAP and the MQ Series adapters covers all requirements? .
Need mainframe connectivity for development and test?
Need Partner support for development or test?

Issues and Risks
List all Issues and Risks of the solution, sample below:

Need to get some particulars libraries or Adapter bits as early as possible. Delay in getting these bits may cause slippage in schedule.
Some issues may arise with the Inter-BTS adaptor design

The next part of this article will start with the most important aspect the Detail Design.

The Detail Design describe in detail the high level architecture of the solution, graphically and with descriptions in detail.
It’s the more complex session of the document, It must cover all the most important aspect as:

Design Challenges, Exception Handling, Integration, Boundary Conditions, Security, Authentication, Authorization,
Impersonation, Performance, Globalization/Localization, Test Support, Manageability, Supportability, Serviceability,
Migration, Multi-Machine Installation, Multi-Domain Installation, Runtime Considerations, Tuning and Throttling,
Large Messages, Multi-Message Box,Tracking, Threading and Concurrency, Fault Tolerance

next topics to the next article.

All of the most important new feature are listed in BizTalk Blog Team
The core topics are:

• Improved Performance
• Simplified Development and Management Experience
• Cloud Connectivity
• Running BizTalk Server in the cloud (IaaS)
• Support for the latest platform and standards

The most important for me and I think for a peoples that work in hard and deep with BizTalk are:

Ordered Send Port improvements, increasing performances
XslCompiledTransform, increasing performances
Host handler per adapter, good best scaling and for best dynamic scenarios
Notification Services for BAM alerting dropped!
Dependency tracking, good for monitoring aspects
SFTP adapter, so we will have a BizTalk SFTP adapter in box
Licensing changed, all info in Saravana Blog

Many BizTalk experts present and great and nice moments of discussion in deep and big passion.

This is the fourth event organized by BizTalk Crew after Amsterdam (Netherlands), Milan (Italy) and Stavanger (Norway)

The BizTalk Crew , from the right Steef-Jan Wiggers, Tord Glad Nordahl, Sandro Pereira, Saravana Kumar, Nino Crudele

And many great fun moments

Thank you to all attendees and new friends for these great three days

High Availability Test Plan of a BizTalk Solution
High Availability Test Plan of a BizTalk Solution – Methodology – requirements and expectation
High Availability Test Plan of a BizTalk Solution – A sample of a real approach

The best strategy is to manage a BizTalk solution/project as a “product”, with his milestone, we can have a table of levels with milestone

Level 0 – Basic Verification Test (also know as BVT)

The purpose of BVTs are to prove that the solution works for the most simple use cases. Passing 100% means the solution may be used for further testing.
Level 0 test cases have the following properties:

• Simple valid test case for major features of the solution
• Typically Manual (Automated if we can)
• Manual cases are run weekly
• Automated cases are run daily
• Isolated component tests with known good data
• Includes components used very frequently
• Runs on recommended platform

BVTs do not contain manual test cases

• Excludes components used infrequently
• Excludes integrated subsystem test
• Excludes atypical platform test

BVTs are used to prove that a build is good.  They are used by integration teams to prove compatibility with other products. They should also be used by Developers before check-in so they don’t break the build.  The suite for any given solution should run in under 1 hour, otherwise it adversely affects the check-in process. A balance needs to be struck between proving that the major core features of the solution are working and insuring that the BVT run quickly so they are easily used and run. For example if BVTs take 4-5hours to run will not be used for daily checkins by Development before check

Automation: you need to consider development cost vs. execution costs when determining if it makes since to automate
Also factor if following product bundling compatibility testing that make also take advantage of automated cases. 

Example: An Level 0 purchase order test case would have a single simple line item and validate that it got processed correctly.

Level 1 – Validity Testing – (also know as IDW – It Does Work)

The purpose of L1 tests are to prove that the typical valid use cases of the solution work as intended. Passing 100% means the solution may be used for external development as well as major testing cycles for solution releases. L1 test cases have the following properties:

• Valid typical test case for all features of the solution including minor features
• Basic Integrated System Tests – test with known dependencies and related features
• Typically Manual (Automated if we can)
• RC Breadth Pass
• Basic, simple developer and user scenarios
• Includes variations of platforms that are supported

IDW’s are typically made public on a restricted distribution basis (alpha or non-production beta’s) and given to select partners as well as internal customers.  It’s primary use case is to further development and get feedback from external sources.  However L1’s passing are NOT sufficient to support production usage of the solution .

Scenarios differ from features in that they are designed to mimic the end to end process a customer goes through in a use case of the solution .
Scenarios may also utilize other featuress or software tools in order to accomplish the job.

Because of this – it is expensive to execute scenarios in an automated daily manner that is typical for software products. For this reason / difference we have developed the following test case level definitions for scenario test levels.

Example: A L1 purchase order test case would have 10 line items, with discounts and changes from a prior order.  It would validate that it got processed correctly.

Level 2 – (Invalid/Error/Boundary Testing) Production Beta

L2 test cases prove that the solution works at the defined limits for each feature.  L2’s also proves that known error conditions are handled correctly.  Valid responses to invalid data or error conditions are tested. 

• Extended Integrated System Tests – test with non-standard situation mix.
• Boundary Conditions
• Invalid/Error Condition
• Typical end to end scenarios – dev through deployment to production testing
• Performance, stress and scale tests that used to meet customer/solution ship criteria

L2s passing would indicate valuable and robust build. Build behaves acceptably when used incorrectly.

Example: An L2 purchase order test case would have 100 line items (product supported max # of line items), with one of every kind of discount and changes from a prior order.  It would validate that it got processed correctly

Example: Another L2 purchase order test case would have 101 line items.  It would validate that the system responded correctly regarding the fact that this exceeds the systems known limit for line items and rejects the order.

Level 3 – Robustness Testing (Destructive testing – Extreme testing)

Solution may still ship, even if a problem was found with this type of test, depending on the scenario. Level 3 testing is very important for understanding the true quality of the solution .  Failed L3 test cases must be characterized for the customer either by KB (if not public), solution docs (readme/known issues/trouble shooting guide), or core docs for solution characterization.

• Depth/Robustness Stabilization
• Complex repro – Unlikely extreme scenarios.
• Fail and recover scenarios (denial of service – security breech..)
• Fault Tolerance
• Extreme deployment and load scenarios.
• Extreme Performance, Stress and Scalability cases used to characterize limits of the product
• Security Hacker testing
• Complex end to end scenarios – disaster recovery, over throttled runtime servers
• May include 3^rd party software that customers typically also install on the same server

Example: An L3 purchase order test case would change the configuration property for max number of line items to maxint and send in a 1GB purchase order. It would validate that the system responded correctly regarding the fact that this exceeds some limit or got processed correctly.

High Availability Test Plan of a BizTalk Solution

High Availability Test Plan of a BizTalk Solution – Methodology – requirements and expectation  

An example of a highly available setup involves setting up 2 or more ACT clients generating HTTP stress against 2 or more ‘Web Servers’ setup to use NLB. The ASP pages on the ‘Web Servers’ create objects [only proxies of which exist on the ‘Web Servers’] which reside on the middle-tier application servers running CLB. These ‘Web Servers’ and ‘Application Servers’ then utilize features provided by backend systems that are setup using MSCS, an example being clustered SQL Server or clustered Commerce Server or clustered BizTalk Server.

We can catalogue some basic scenarios and for any of these exist a correct strategy

2 Box Configuration [Separate SQL]:

1 Box BizTalk [Receive/Transmit/Scheduler/Map/Transformation]
This machine will also be used to Receive FILE/MSMQt messages.
MSMQt messages will always be received locally by BizTalk.
BizTalk will simulate the behavior of an MSMQ server.

1 Box SQL Server

This machine will host the BizTalkMsgBoxDb, BizTalkMgmtDb & the Trackingdb.

3 Box Configuration [Separate Receive/Transmit-Process + Separate SQL]:

The receive functionality will be separated from the Process/Transmit functionality spreading the load.

4 Box Configuration [Multiple Receive + 1 SQL + 1 Process/Transmit]

This is similar to the 3 machine configuration with multiple receive locations.

4 Box Configuration [Multiple Transmit + 1 SQL]

This is similar to the 3 machine configuration with multiple Process/Transmit locations.

4 Box Configuration [Separate Receive/Transmit + Multiple MsgBox]

This is similar to the 3 machine configuration with multiple SQL message boxes.
Note: This does not provide failover as both the MsgBox databases are unique.

HA deployment [Clustering/NLB]

This is a Highly Available deployment configuration where failover is provided by adding multiple machines. The Receive & Transmit machines are configured to run as NLB clusters.

Large Scale Deployment [Clustering/NLB/ISA/DMZ]

This is a Highly Available deployment configuration where failover is provided by adding multiple machines. The Receive & Transmit machines are configured to run as NLB clusters. The SQL Servers are also configured to run on MSCS clusters on RAID 5 disks providing total failover protection.

All the above configurations will run on a separate isolated domain on a private VLAN.

Data Centre Certification

Identify areas that are potential candidates for HA [E.g. NT Services] failover testing.
Testing required throughout the product cycle to ensure that the product is certifiable.
Test scripts to be written to simulate failover scenarios.
Methods of deploying multiple machines [including DC setup].
Select scenario that will be run during failover tests.
Ensure product passes all tests performed by tools
[Job Objects/Failover/Recoverability/Data Persistence/Data Integrity]

Example:

An example of a highly available setup involves setting up 2 or more ACT clients generating HTTP stress against 2 or more ‘Web Servers’ setup to use NLB. The ASP pages on the ‘Web Servers’ create objects [only proxies of which exist on the ‘Web Servers’] which reside on the middle-tier application servers running CLB. These ‘Web Servers’ and ‘Application Servers’ then utilize features provided by backend systems that are setup using MSCS, an example being clustered SQL Server or clustered Commerce Server or clustered BizTalk Server.

Following is a diagram depicting a large scale deployment with an NLB clustered front end and multiple MSCS clustered backend systems:
Following is a sample of plan for testing a BizTalk solution in a distributed environment.

Protocols:

For example we have 5 different protocols [2 way Receive/Send] will be tested on the various deployment configurations.

• FILE
• HTTP
• MSMQt
• SOAP: This will be tested using Web Services.
• SMTP: This will only be used to Transmit Messages one-way.

Testing Permutations:

Since we have 3 protocols each way, to test all possible combinations we end up with 9 possible permutations.

Rx	Tx	XLANG/S	XLANG/S
FILE	FILE	Y	N
FILE	HTTP	Y	N
FILE	MSMQ	Y	N
FILE	SOAP	Y	N
FILE	SMTP	Y	N
HTTP	FILE	Y	N
HTTP	HTTP	Y	N
HTTP	MSMQ	Y	N
HTTP	SOAP	Y	N
HTTP	SMTP	Y	N
MSMQ	FILE	Y	N
MSMQ	HTTP	Y	N
MSMQ	MSMQ	Y	N
MSMQ	SOAP	Y	N
MSMQ	SMTP	Y	N
SOAP	FILE	Y	N
SOAP	HTTP	Y	N
SOAP	MSMQ	Y	N
SOAP	SOAP	Y	N
SOAP	SMTP	Y	N

Only 4 of the protocols can be used to Receive & all 5 protocols can be used to Transmit leading to 20 permutations. This can be further complicated by running the tests with and without the XLANG/S scheduler.

Thus it can lead to 40 possible combinations.

To avoid the complexity in managing multiple scenarios, a simple scenario has been built that exercises all of the above protocols both ways. This scenario will be configurable, whereby Receive/Send locations can be Enabled/Disabled with a simple script.
This sample scenario will also permit Enabling/Disabling the code path that uses or skips the Scheduler. Although this scenario will be comprehensive, it will be easier to manage from a testing perspective. The objective of the testing in HA is not to exercise Stress/Perf, but to ensure a successful failover.

Principal Testing Defined:

Test scripts/code will be developed that simulate real world contingency situations.
E.g. Loss of network connectivity, System Crash, Service Stopped/Crashed.
Data Centre Certification requires that the product failover gracefully.
The tool will also be used to test the behavior of the product under extreme conditions.
Results of these failures will have to be measured in an automated manner.
Did the service failover in a reasonable amount of time? Response time between failover and recovery.
Did the system continue processing the last failed transaction? Did it abort it and/or lose it?
Did the system preserve data integrity? Was the transaction atomically committed or rolled back?
Did the system retry a failed/suspended transaction to completion?
Automation [scripts] will have to be developed to measure the above failure situations.
E.g. If 1000 valid files were dropped, did 1000 messages appear at the destination[FILE/HTTP/MSMQt]?

Deployment:

Since testing in this area involves multiple machines with varied configurations deployment will have to be automated such that minimal effort is required to setup one of the following configurations.
Time spent on deployment can be minimized by having dedicated machines specifically assigned for HA testing.
All machines will have the base OS + other dependencies installed and imaged.
Not all machines in the test bed will have to be re-imaged every time a build changes.
The only machines that will require re-imaging will be the BizTalk boxes.
Scripts will have to be developed to restore the SQL Server databases to a pristine state before each new run.
All scenarios used for testing will have automated scripts that deploy the scenarios on multiple-machines. Test scripts required to generate load on the deployed scenarios will also be automated. Test run results will be rolled up into the http://ebizweb site.

Performance:

The flip side of testing in a HA environment is that Performance numbers for scalability can be produced.
These types of controlled runs can be achieved once a build has been confirmed as a candidate for a performance run. If a particular build passes a test similar to a BVT, it is deployable and able to pass cursory testing like end-to-end loop, performance numbers can be extracted from this run.

Real sample Performance test tracking

Performance Parity with BizTalkGroup1

Targeted Performance Improvements Over BizTalkGroup1

New features

Capacity Planning

Stress

Adapter Matrix sample

Release Criteria Scratch

Test Suites samples

Definitions

Following are a few tips and suggestions for High Availability testing:

Understand the scope of the High Availability test

Most High Availability test scenarios involve deploying different parts of the product/s on different machines. Proper deployment guidelines need to be clearly defined and understood.

Debugger and symbols

Reproducing High Availability bugs are difficult due to the magnitude of the deployment effort. Thus when these bugs are encountered we need to ensure that all supporting software is installed or available so that the developer can diagnose the problem.
A complete stack dump will only be useful if all the matching symbols are available. These should include not just the symbols for the product but also all the symbols for the relevant OS including all the service packs and symbols for dependencies.
It is important to run all test on High Availability test beds with debuggers attached

High Availability Tools

We need to ensure that we consistently run and pass using the tools provided by Data Center. In addition to these we also need to run with customized tools that simulate real world failure situations.
These need to be built based on area of functionality and after consulting with the PM and developers.

Firewalls

In a real world scenario all serious consumers of our products will install some form of security firewall.
High availability tests should also be run with firewalls (both hardware and software) setup. This will help detect issues related to limitations existing in the product early on. E.g. raw RPC calls.

Increase Test Coverage

Write code to test new functionality. In addition to running and testing existing HA aware functionality, write scripts to increase feature area test coverage.

Use Multi-Proc servers

All High Availability tests should be run on multi-proc servers to avoid the serialization introduced by hardware. Many issues will not easily be found in a single proc server.

Dedicated platform

Although High Availability testing is not Performance testing, every effort should be made to ensure that the hardware platform does not frequently change. This will help ensure consistent results build over build.

Security

Ensure that all tests are performed using normal user credentials. Using Administrator with full access will obscure some of the results. It is imperative to test early on in the cycle with normal user rights. If normal functionality does not work without administrative rights file a bug.

Documentation

All steps taken to setup and deploy the products for high availability testing should be clearly documented and caveats should be clearly highlighted. Problems faced installing/uninstalling in a HA environment should be filed as bugs. E.g. installing on an MSCS cluster should require deployment from one node only. If setup requires manual installation on each node file a bug.

Page heap or App verifier to catch memory corruption

For memory corruption bugs, to get good call stack on where the problem is, you need to use some tools to help you on catching them right in time. Page heap/Gflags (or App verifier if you use .net server) can help.
For debugging and catching memory corruption bugs, you should always use page heap. Run with both options Normal/Full.

Occasionally use other options like:

If page heap hangs: /protect (protect internal page heap structures from corruption).
Verify failure code paths: /fault (simulate low memory conditions).
Verify pointer decrement: /backwards (reverse buffer overruns).
Some of these problems might manifest themselves only when failover occurs.

Automation

All testing on HA environment should be automated preferably through scripts. Imaging, deployment running the tests and logging the results should be setup so that they can be triggered remotely.
Time invested in this effort will save time in the long run and help transferring ownership of testing seamless.

Help Improve Functionality

Attend feature area meetings related to high availability functionality. Pay close attention to changes in functionality in these areas. E.g. a change to maintain state locally on the box could mean that the feature can no longer failover and also cannot scale out.
Call out these issues. Look for areas that can be improved, e.g. Long running tasks should be broken into smaller atomic jobs, if the system fails during one of these long tasks, the next node can and should continue after the last good checkpoint rather than start all over.

Example in a real scenario…

to be in next post…