I need to retreive any type structure data from a database with WCF SQL adapter, as you know thw WCF SQL adapter retreive a particular propetary structure and it doesn’ t permit to retreive complex structure data from a database, work with complex structure type from database with WCF SQL adapter is very difficult.
For example a complex structure could be this:

Sometime I need to retreive complex XML structure data from a database, in this case I develop a pipeline component that intercept the data content in WCF  resulset.
In this way I can use any XML syntax in my storeprocedures without problem.

We can create any type of structure in a storedprocedure using XML AUTO command and affinity instructions.
When the WCF adapter retreive data returns this structure formed below

The pipeline component filter the resulset of WCF SQL adapter and retreive the  internal XML resulset as below

Where is the advantage?, in this way I can retreive any complex data structure type from database in only one step.

Steps:

Generate your XML data structure in SQL and generate your schema using the Add Generated Item, Generated Schemas, Well-Formed XML, from Visual Studio
Create a root node and put alll schema content in, as below

Use the pipeline component is very simple, put the component in your stage as below

Set the properties in pipeline component

Namespace:
The namespace you want assign to the message

RootName:
The root name node of thee message

WcfXmlPath:

The XPath expression to use to retreivee the internal and recursive resulset of the WCF SQL adapter
Finally create a port and use the pipeline

You can download the pipeline component HERE

The data to be transferred to the SAP system is prepared in a sequential file, BBSEG is a standard ABAP Table available within SAP system, this operation is called Data Transfer Workbench.
A file may only contain data that can be processed with the same transfer program and the program creates batch input sessions from this file.

In SAP this file type is called BBSEG, his structure is complicate and the documentation is poor and not so detailed.

The main structure is formed from a Session header record, Header data and Subsequent data

These structures contain data from the master records and the line items. This includes, for example, the line item amounts.

for more information you can find all documentation is SAP help here
http://help.sap.com/saphelp_erp2004/helpdata/en/e9/cb8a85eb0011d184650000e8a6bfbe/frameset.htm

Master record type table

Main document structure

All of these are composed from a more of 500+ different fields

After two days of work I recreated the same structure in XSD schemas to use this in a important project

Actually I using this schemas structure in a important BizTalk solution

You can download the schema project here

http://sdrv.ms/10KzL3f

This is the second of a collection of articles that explain, for my experience, the best way to organize a structure of a document for all the most usual cases.
You can find the first part here

In this part of this article will start with the most important aspect the Detail Design.

The Detail Design describe in detail the high level architecture of the solution, graphically and with descriptions in detail.

Detail Design

Detailed design can be divided into three parts based on architecture.

1. Presentation Layer: In presentation layer, we will describe the messages flow layer from all actors, for example:
   the messages will be received from site1, http post . In order to receive messages from site1 we need to build site1 using technology X.

Describe the work Items for Site1:

1. Create new user profiles.
2. Modify this feature site
3. Upload file

2. Business Workflow Layer: In this layer we will describe graphically the messaging flow layer

GRAPHIC DIAGRAM

Describe all steps in the flow, for example:

A. Create New Order: Partner1 will send request either using http URL provided by Hub.

B. When New order is posted to site, ASPX page of site will drop message to MSMQT location on BizTalk Server.

If exist describe particular security aspects, for example,
This request will be encrypted using X.509 certificates.

Describe messaging typology and structure, for example,

Sample EDI Header:

After the Design Detail we will start to describe the others important aspect.

Exception Handling

Describe the methodology used in the solution covering all of the parts, coding, orchestration exception strategy, if necessary graphically too, for example:
In all use cases have Orchestration and there is no custom code involved in it.
Orchestration will have exception handler as shown in figure.

Integration

Describe the most relevant aspect about integration strategy, for example:

BAS Integration: B2B scenario will use BAS for Trading Partner Management.
Trading partner management is done using website provided BAS feature.

BAM Integration: BAM will be used to send alerts and notifications to vendors and broker.
Information required for BAM will be tracked using Tracking Profile Editor as shown in figure.

Other important aspects to cover are:

Boundary Conditions

In this section we will Identify all boundary conditions implied by this design, or to be covered by this design

Security

In this section we will Identify all boundary conditions implied by this design, or to be covered by this design

Authentication

In this section we will Identify all authentication strategy

Authorization

In this section we will Identify all authorization strategy

Impersonation

In this section we will Identify all impersonation strategy

Performance

In this section we will Identify all performance requirements, for example
According to functional specification 1.000.000 orders will be processed End-End per day.

The most important aspect is identify the possible messages peeks, for example

According to functional specification 1.000.000 orders will be processed End-End per day with a peek of 500.000 between 2:00 AM and 3:AM.

 

Globalization/Localization

In this section we will describe possible Globalization/Localization strict

Test Support

If functional specification contains specific items required to enable testability.
This section identifies specific work and impacts on the algorithms necessitated by test support

Manageability

List policies and settings the admin and IT professionals can set to affect this features behavior.

Supportability

List considerations for supporting the feature: things like tracing, diagnostics.

Serviceability

List any issues and considerations for updating this feature set after it’s been deployed. Do we need to stop the service to replace the component? Do we need to reboot?

Migration

In this section we will describe possible strict and requirements about migration

Multi-Machine Installation

Create and refer to an architecture diagram.

MulTi-Domain Installation

Describe the domain installation, for example:
The scenario will have two domains. One domain for front servers facing outside world.
Another domain will be used for rest for servers which are behind corpnet firewall.

Runtime Considerations

Tuning and Throttling

List considerations on tuning and throttling.

Large Messages

In this section we will list all possible large messages peeks

Multi-Message Box

If exist describe the multi-message box strategy, configuration, scopes

Tracking

Describe the tracking strategy of the solution, for example:
Tracking will be enabled for BAM. Body tracking will be enabled for this scenario.

Threading and Concurrency

Describe possible threading and concurrency issue

Fault Tolerance

Describe the fault tolerance strategy, for example:

SQL Server will be clustered in active-passive node.
Minimum scenario will have 2 BizTalk for fault tolerance.

This is the first of a collection of articles that explain, for my experience, the best way to organize a structure of a document for all the most usual cases.
In this article the design specification of a BizTalk Solution, the same structure can be used for all types of solutions, BizTalk and not.

• Introduction

Is composed from three areas

• Associated Documents
  • List of the documents associates to design specifications, normally
    • Pattern (B2B, BPM ) Functional Specification
    • Customer Scenario Specification
    • Some particular documents as
    • TPM Specification
    • BAM Alert and Notification Specification
• Supported Features

  features/functionalities supported in Scenario for example:

  • Support EDI orders over SFTP
  • Support for CS catalog, order and inventory adapters
  • Create or Update orders either using http post for given URL.
  • Check and update inventory of items.
  • Update User Profile for Brokers and Vendors.
• Definitions
• List all particular definitions used in the document, for example:
  • B2B: Business –To- Business
  • TPM: Trading Partner Management
  • CBO: Common Business Object.
  • BAM: Business Activity Monitoring.
  • BAS: Business Activity Services.

Design Overview
List and design, if need, the most important aspects of the solution design, sample:

The general design consists of an orchestration that:

Macro Requests steps

• New Order coming from Broker.
• Update Order coming from Broker.
• Update user profile from Broker.
• Update Inventory from Vendor.
• Check Inventory from Vendor.

Describe the formats type:
These requests will be received by hub in different formats such as EDI, Flat Files and XML using different transports like HTTPS, SFTP etc.

Describe the macro steps:

A. Create New Order: Broker will send request either using commerce server website or will post to http URL provided by Hub.

1. New Order using commerce site: Broker will create order using commerce website. This order will be received ………

2. New Order using HTTP Post: Some broker can send their order using HTTP URL specified by Hub…………..

3. EDI Type Orders: EDI type orders will be uploaded to SFTP site. EDI Type orders will be picked by File Receive location. ……..

B. Update Order: Broker will send request either using commerce server website or …….

1. Update Order using HTTP Post: Some broker can send their update order using HTTPURL specified by Hub. …….

2. EDI Type Order: For EDI Type change orders will be uploaded to SFTP site. . EDI Type ………….

Present the design graphically

Architecture and Topology
shows over all architecture of scenario, a sample below:

Platforms/Portability Issues

Where the solution can work, a sample:

This Scenario will only work on BizTalk Name and version, Commerce Server Name and Version.
or
In case of (A), nothing special is needed on the operating system other than bla bla bla.
In case of (B), it will require certain level of setting for MSDTC for windows [version x] (not Windows version x)

Assumptions and Dependencies

Following assumptions are made for your scenarios release:
a.    Where development of scenario will be done.
b.    Adapter for will be used for development
c.    BAM alert and notification Service will be used
d.    BAS Trading Partner will be used

Partners and Dependencies

List all partners and dependencies types, logical, application and technological, a sample:
Adapters dependencies?
There is a dependency on the SAP Adapter and the MQ Series adapter –
The SAP and the MQ Series adapters covers all requirements? .
Need mainframe connectivity for development and test?
Need Partner support for development or test?

Issues and Risks
List all Issues and Risks of the solution, sample below:

Need to get some particulars libraries or Adapter bits as early as possible. Delay in getting these bits may cause slippage in schedule.
Some issues may arise with the Inter-BTS adaptor design

The next part of this article will start with the most important aspect the Detail Design.

The Detail Design describe in detail the high level architecture of the solution, graphically and with descriptions in detail.
It’s the more complex session of the document, It must cover all the most important aspect as:

Design Challenges, Exception Handling, Integration, Boundary Conditions, Security, Authentication, Authorization,
Impersonation, Performance, Globalization/Localization, Test Support, Manageability, Supportability, Serviceability,
Migration, Multi-Machine Installation, Multi-Domain Installation, Runtime Considerations, Tuning and Throttling,
Large Messages, Multi-Message Box,Tracking, Threading and Concurrency, Fault Tolerance

next topics to the next article.

As many of you know Microsoft announce the new RTM version of BizTalk 2013.

All of the most important new feature are listed in BizTalk Blog Team
The core topics are:

• Improved Performance
• Simplified Development and Management Experience
• Cloud Connectivity
• Running BizTalk Server in the cloud (IaaS)
• Support for the latest platform and standards

The most important for me and I think for a peoples that work in hard and deep with BizTalk are:

Ordered Send Port improvements, increasing performances
XslCompiledTransform, increasing performances
Host handler per adapter, good best scaling and for best dynamic scenarios
Notification Services for BAM alerting dropped!
Dependency tracking, good for monitoring aspects
SFTP adapter, so we will have a BizTalk SFTP adapter in box
Licensing changed, all info in Saravana Blog

I come back from OPorto BizTalk Innovation Day, a great three days in Portugal.
Devscope in collaboration with Microsoft and Codit organized a great event in a nice place, Sandro Pereira worked hard and to him all the thanks for this.

Many BizTalk experts present and great and nice moments of discussion in deep and big passion.

This is the fourth event organized by BizTalk Crew after Amsterdam (Netherlands), Milan (Italy) and Stavanger (Norway)

The BizTalk Crew , from the right Steef-Jan Wiggers, Tord Glad Nordahl, Sandro Pereira, Saravana Kumar, Nino Crudele

And many great fun moments

Thank you to all attendees and new friends for these great three days

About this argument you can find other resources in my other three last posts,

High Availability Test Plan of a BizTalk Solution
High Availability Test Plan of a BizTalk Solution – Methodology – requirements and expectation
High Availability Test Plan of a BizTalk Solution – A sample of a real approach

The best strategy is to manage a BizTalk solution/project as a “product”, with his milestone, we can have a table of levels with milestone

Level 0 – Basic Verification Test (also know as BVT)

The purpose of BVTs are to prove that the solution works for the most simple use cases. Passing 100% means the solution may be used for further testing.
Level 0 test cases have the following properties:

• Simple valid test case for major features of the solution
• Typically Manual (Automated if we can)
• Manual cases are run weekly
• Automated cases are run daily
• Isolated component tests with known good data
• Includes components used very frequently
• Runs on recommended platform

BVTs do not contain manual test cases

• Excludes components used infrequently
• Excludes integrated subsystem test
• Excludes atypical platform test

BVTs are used to prove that a build is good.  They are used by integration teams to prove compatibility with other products. They should also be used by Developers before check-in so they don’t break the build.  The suite for any given solution should run in under 1 hour, otherwise it adversely affects the check-in process. A balance needs to be struck between proving that the major core features of the solution are working and insuring that the BVT run quickly so they are easily used and run. For example if BVTs take 4-5hours to run will not be used for daily checkins by Development before check

Automation: you need to consider development cost vs. execution costs when determining if it makes since to automate
Also factor if following product bundling compatibility testing that make also take advantage of automated cases. 

Example: An Level 0 purchase order test case would have a single simple line item and validate that it got processed correctly.

 

Level 1 – Validity Testing – (also know as IDW – It Does Work)

The purpose of L1 tests are to prove that the typical valid use cases of the solution work as intended. Passing 100% means the solution may be used for external development as well as major testing cycles for solution releases. L1 test cases have the following properties:

• Valid typical test case for all features of the solution including minor features
• Basic Integrated System Tests – test with known dependencies and related features
• Typically Manual (Automated if we can)
• RC Breadth Pass
• Basic, simple developer and user scenarios
• Includes variations of platforms that are supported

IDW’s are typically made public on a restricted distribution basis (alpha or non-production beta’s) and given to select partners as well as internal customers.  It’s primary use case is to further development and get feedback from external sources.  However L1’s passing are NOT sufficient to support production usage of the solution .

Scenarios differ from features in that they are designed to mimic the end to end process a customer goes through in a use case of the solution .
Scenarios may also utilize other featuress or software tools in order to accomplish the job.

Because of this – it is expensive to execute scenarios in an automated daily manner that is typical for software products. For this reason / difference we have developed the following test case level definitions for scenario test levels.

Example: A L1 purchase order test case would have 10 line items, with discounts and changes from a prior order.  It would validate that it got processed correctly.

Level 2 – (Invalid/Error/Boundary Testing) Production Beta

L2 test cases prove that the solution works at the defined limits for each feature.  L2’s also proves that known error conditions are handled correctly.  Valid responses to invalid data or error conditions are tested. 

• Extended Integrated System Tests – test with non-standard situation mix.
• Boundary Conditions
• Invalid/Error Condition
• Typical end to end scenarios – dev through deployment to production testing
• Performance, stress and scale tests that used to meet customer/solution ship criteria

L2s passing would indicate valuable and robust build. Build behaves acceptably when used incorrectly.

Example: An L2 purchase order test case would have 100 line items (product supported max # of line items), with one of every kind of discount and changes from a prior order.  It would validate that it got processed correctly

Example: Another L2 purchase order test case would have 101 line items.  It would validate that the system responded correctly regarding the fact that this exceeds the systems known limit for line items and rejects the order.

Level 3 – Robustness Testing (Destructive testing – Extreme testing)

Solution may still ship, even if a problem was found with this type of test, depending on the scenario. Level 3 testing is very important for understanding the true quality of the solution .  Failed L3 test cases must be characterized for the customer either by KB (if not public), solution docs (readme/known issues/trouble shooting guide), or core docs for solution characterization.

• Depth/Robustness Stabilization
• Complex repro – Unlikely extreme scenarios.
• Fail and recover scenarios (denial of service – security breech..)
• Fault Tolerance
• Extreme deployment and load scenarios.
• Extreme Performance, Stress and Scalability cases used to characterize limits of the product
• Security Hacker testing
• Complex end to end scenarios – disaster recovery, over throttled runtime servers
• May include 3^rd party software that customers typically also install on the same server

Example: An L3 purchase order test case would change the configuration property for max number of line items to maxint and send in a 1GB purchase order. It would validate that the system responded correctly regarding the fact that this exceeds some limit or got processed correctly.