Optimize the BizTalk productivity using ListDictionary

In this period, I’m supporting different development teams to implement a big integration solution using different technologies such as BizTalk Server, Web API, Azure, Java and other and involving many other actors and countries.
In a situation like that is quite normal to find many different problems in many different areas like integration, security, messaging, performances, SLA and more.
One of the most important aspects that I like to consider is the productivity, writing code we spend time and time = money.
Many times a developer needs to find a solution or he needs to decide a specific pattern to solve a problem and we need to be ready to provide the most productive solution for it.

For instance, a .Net developer can decide to use a plain function to solve a specific loop instead using a quicker lambda approach, a BizTalk developer can decide to use a pure BizTalk approach instead using a quicker and faster approach using code.

To better understand this concept, I want to provide you a classic and famous sample in the BizTalk planet.

An usual requirement in any solution is, for example, the possibility to pick up data in composite pattern and cycling for each instance inside the composite batch, in BizTalk server this is the classic situation where we are able to understand how much the developer is BizTalk oriented J

By nature, the BizTalk development approach is more closed to a RAD approach rather than a pure code approach.

To cycle in a composite message in BizTalk we can use different ways and looking in internet we can find many solution, one of the most used by BizTalk developer is creating an orchestration, create the composite schema, execute the mediation, receive the composite schema in the orchestration, in the orchestration cycle trough the messages using a custom pipeline called in the orchestration.

Quite expensive approach in term of productivity and performances.

Another way is using for example a System.Collections.Specialized.ListDictionary in the orchestration.

Create a variable in the orchestration type System.Collections.Specialized.ListDictionary

Create a static class and a method named for example GetAllMessages

Inside your method write the code to pick up you messages and stream and load into the ListDictionary

Create an expression shape and execute the method to retrieve the ListDictionary

xdListTransfers = new System.Collections.Specialized.ListDictionary();

MYNAMESPACE.Common.Services.SpgDAL.GetAllMessages(ref xdListTransfers,

ref numberOfMessages,

ref dataValid);

Use variable by ref to manage the result, the variable numberOfMessages
is used to cicle in the orchestration loop.

Because we use a ListDictionary we can easily get our item using a key in the list, as below using the numberOfMessagesDone variable.

xmlDoc = new System.Xml.XmlDocument();

transferMessage = new MYNAMESPACE.Common.Services.TransferMessage();

transferMessage = (MYNAMESPACE.Common.Services.TransferMessage)xdListTransfers[numberOfMessagesDone.ToString()];

Where numberOfMessagesDone
is the increment value in the loop

Using this method, we keep our orchestration very light and it’s very easy and quick to develop.

I have many other samples like that, this an argument which I’m really care about because able to improve our productivity and performance and first of all the costs.

 

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

 

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.

 

 

        

 

 

10 Years as Microsoft MVP !

10MVP

Today Microsoft renews my MVP award for the 10th time.

I would like to tank Microsoft for these 10 years, thank to my family and my wife for the patience and the support, thank to all my MVPs friends for the great moments together in these 10 years and a huge thank you to all my followers, supporters and the integration animals around the world for the great challenges in these 10 years, next year with the new features incoming in Azure and the great job around BizTalk Server we will raise the level🙂

10 years as MVP is a long journey, full of greats and bad moments, big challenges,  wins and losses, many conferences around the world, thousands of hours during the evenings study new patterns and writing code and a lot of time invested to share big passion to many people as possible.

Why?

Because this is a Microsoft MVP.

Proud to be a member of this awesome family.

 

Real time performances for BizTalk Server with the GrabCaster Chain Components

In the last period I worked in some new GrabCaster features like the encoding and extensibilities and about the extensibilities I implemented the possibilities to use a chain of components inside the engine.
Some people confuse the meaning of chain with the concept of pipeline but they are very different, a pipeline, as the name quote, is a pipe + line, a number of blocks executed in a sequence and a pipeline has a starting and an ending.
I like to compare the meaning using the real objects, a pipe cannot be closed otherwise the liquid stop flowing, the liquid enters in to the pipe end it come out from the end.

Interesting the idea to connect different pipelines in sequence but I see a lot more potential in the chain pattern.

A chain does not contain a real concept of starting and ending and can be closed and also executed in cycle.

Different chains can be opened and hooked together to create a longer chain and so on, I really love this concept because it gives me the opportunity to extend the capabilities in many ways and combinations.

 

A GrabCaster chain is composed by components and triggers and events can execute chains.
A chain component is a generic Interface able to ingest data, manage it and share the result with the other components in the chain.

I’m very interesting to provide more powerful options in BizTalk Server, BizTalk is a great framework and I like the idea to provide to it more interesting options and extensibility.

For that reason the first chain components I have implemented are the BizTalk pipeline component and the BizTalk transform component.
The BizTalk pipeline component executes a BizTalk pipeline DLL component and the BizTalk transform component is able to execute a BizTalk map and using a DLL like a Map storage provider, this is very powerful because it gives me the opportunity to use the BizTalk Server artefacts inside GrabCaster and use the same BizTalk features in real time performances, thousands messages a second using BizTalk Pipelines and BizTalk Transformations, I will write another post very soon about that.
The other interesting thing is the opportunity to execute GrabCaster inside BizTalk using the GrabCaster BizTalk Adapter which uses the chains as well.

To configure a chain, I use a json configuration file as below

In this pipeline I execute two chain components, the BizTalk pipeline and the BizTalk Transform.

I use a json file to configure the BizTalk pipeline component as below

Where:

AssemblyFile is the Dll component containing the BizTalk pipeline
PipelineTypeName is the pipeline type name to use
PipelinePathFile is the BizTalk pipeline file

I use a json file to configure the BizTalk transform component as below

Where

AssemblyFile is the Dll component containing the BizTalk pipeline
TransformTypeName is the transform type name to use

To assign a chain to a Trigger or event I add the chain specification to the Trigger json configuration file as below

 

I can use the chains in Triggers and Events and I can connect chains to other chains together.

In this test I imported the flat file as below

serialized as xml using the BizTalk pipeline and transform it as below

 

I will write a lot more about that in the next posts, for more information about GrabCaster you can refer here

Please contact me for any more information or if you want to collaborate inside the project.

How to use Microsoft Azure in the best way – The Microsoft Azure Dream Team

Microsoft Azure is now a consolidate technology stack able to offer a large number of technologies, in the last years I saw a lot of changes and a very important revolution inside the cloud.
I like to use any Microsoft Azure stack and I like to combine them in any way, I think this is the real power of that, being focused to use a specific technology can limit our creativity.
Entering in the Azure portal https://portal.azure.com and browsing the features is the best way to realize how many things we can use now, I think is impressive, it’s very important to understand the best strategy to combine them to obtain the best result in fast, smart and best productivity way.

I spent the last ten years using BizTalk Server, I definitely love this framework, this framework has born during the 2000 and with the 2004 version it has opened the way for the new frontiers in the world of integration.
BizTalk Server was the first one able to offer concept like reliability, mediation, adaptation, pub/sub, abstraction and more, all the other frameworks copied the concept and they have got the ideas from BizTalk Server, in the same time Microsoft started extending this idea 8 years ago creating the concept of BizTalk Services, some people still trying to correlate this concept with a specific Azure technology but in my opinion this is wrong.
The concept of BizTalk Services in Azure is a group of technologies and not a specific one, best way to understand the concept is doing a nice exercise translating the BizTalk Server stacks into the cloud.

Below the most famous picture and I love it, the story of integration, what we used for years to explain to people how to integrate technologies and to explain these concepts, the BizTalk messaging flow.

Below how I translate it using the most relevant Azure stacks.

These stacks are the most relevants and I like to call them the Azure Dream Team, below some of my preferred.

API Management is everywhere and it provide anything we need for communication, mediation, security, services management and governance, metrics and more.
Simple to use using the UI and it’s able to solve many complicate problems, for example like security and metrics, in very simple way, Microsoft has done a great job on it.

Service Bus is the door and the internal neural system in the cloud, it covers the communication with relays for example, the messaging aspect, the reliability using queues and the pub/sub using topics.

Service Fabric is the stack which covers the reliability and scalability, all the Microsoft technologies use this stack, starting from Skype for business to SQL Server or the entire Microsoft Azure.
The clever idea in this stack is the possibility to make your code reliable just using specific Reliable Collections and Reliable Queues, if you need to be reliable and scalable this is the stack to use.

Logic App is the way to cover the integration and mediation aspect, not only, Logic App is able to offer a very simple way to build flows in the cloud, this because Microsoft starts using the term Microsoft flow.
Logic App contains a lot of connectors, the concept of mediation and of BizTalk, we can extend the capabilities using the Logic App functions which are the way to provide scripting code in the blocks and the possibility to use maps and transformations as well.
Logic App is the stack we need to use to provide orchestration, mediation, transformation and strictly patterns to drive our processes in the cloud like Web API which is the way to build our application blocks and the Web Jobs for background processing.

About transformations we can also use Stream Analytics, the concept is very clever, receive streams from any other stack, use an internal scripting language based on a very well-known language like TSQL and push the stream to any other Azure stack.

EventHubs is the messaging ingestor, millions of messages a second, the interesting aspect is the possibility to use it as pub/sub engine, it is fast and reliable, for performances reasons the message size limit is 256 Kb and we can use Blobs as temporary message storage, this is for example one of the ways used by GrabCaster to exchange messages between the points.

About devices we can use Azure IoT Hub which covers any specific aspect related to IoT, one of the thing I more like is the easy way to create IoT scenarios in a minute using the internal templates, IoT is a very complex topic and this is the best way to understand how to use this technology in the best way.

There are many other stacks to use for any specific topic, I will write more in the next articles.

 

BizTalk Migration and Moving Risks Board White Paper

I like to share a couple of my white papers, I normally use these during a BizTalk migration, moving or update.
A BizTalk Server migration or moving is a good opportunity to review our environment, one of the most critical point are the risks we can find and avoid in the during.
There are a lot of issues to consider, some of these are very simple and, because simple, are the most dangerous.
The best thing to do is using a reference document to keep track about our risks, I like to call it the Risk Board.
In the Risk Board I normally put everything I think needs to be considered and covered, some of these points are critical by personal experience and, if we cover them in the first period, we will be able to do a great job.

I will keep update these whitepaper, please feel free to use them and to collaborate with your feedback and experience.

You can find the white papers here in my BizTalk documents folder on GitHub.

https://github.com/ninocrudele/BizTalk-Documents

GrabCaster Framework – Top 20 Questions and Answers

During the Integrate 2016 Sandro shown a demo using one of the GrabCaster features, the BizTalk GrabCaster Adapter and I anticipated a bit about the GrabCaster Framework, many people made me many questions like how GrabCaster works, what GrabCaster is, the story of it and so on, so I decided to group the top 20 before the TUGA IT.

company logo.fw

When you started the development?

I started the development 2 years ago and during the spare time, it was really hard but thank to my wife Grazia and my family I have been able to keep going and terminate the idea, well to be honest my wife hates GrabCaster now J.

Where the idea comes from?

I spent the last 15 years in the integration using any kind of software and I had a look in any framework and one of the aspect normally annoy me is the complexity to integrate technologies and exchange data, sometime because the framework is huge and we only need to do some specific tasks, other time because the framework is too complex inside, other time because we need to use a black box and we are forced to customize and use tricky strategies to solve a problem and last but not the least the costs.

GrabCaster is a framework able to Grab and Cast data across any network, Grab + Cast + (er) J.

What do you like more about GrabCaster?

I like the simplicity to solve problems and to integrate technologies, in my opinion GrabCaster is unique in the category of the integration frameworks, I’m not able to find something similar, I’m sure many companies and people will use the same approach in the future, it’s a new approach a people need I love the idea and I love speaking about it with other people and get opinions and new ideas.
I want to dedicate this huge job to my Dad, he was a star in his job and the most unique person I had the honour to know in my life.

What GrabCaster is?

GrabCaster is something completely new, it uses a completely new concepts to exchange data across the network and to integrate technologies, it’s clever and smart to use.
The engine is an integration framework based on a loosely couple event handling approach, a point opens a channel and any point can exchange messages with any active point.
The approach to send information is open and using any kind of pattern like Pub/Sub, Point 2 Point, Peer 2 Peer, 1 To 1, 1 To N, N To 1, N To N, Synchronous and Asynchronous ad other…

How GrabCaster connect the endpoints?

Unlike the other frameworks, GrabCaster doesn’t use a point to point approach for instance using lines, arrows or creating subscribers, a point is already connected because it exists and active and it can receive events any time from any other point.

How GrabCaster can be used?

GrabCaster is polymorphic and it supports different way to be executed and used.

GrabCaster Framework is the main engine and there are many other features.

The Framework can be executed in different way:

  • As Console Application
  • As Windows NT Service, the framework exposes all the functionality as Windows NT Service.
  • Embedded in other applications and frameworks
    • When a framework embeds GrabCaster it hosts GrabCaster inside and it get all the functionalities, for instance, in case of the BizTalk Adapter and the Logic App Adapter, GrabCaster is inside BizTalk Server and inside the Logic App connector, the BizTalk Adapter and the Logic App Adapter are just two of the GrabCaster features.
    • About the BizTalk GrabCaster Adapter, where exactly GrabCaster runs?
      • When the GrabCaster Adapter is executed BizTalk Server uses the GrabCaster functionalities inside, GrabCaster is inside BizTalk Server and BizTalk extend the functionality using the GrabCaster Framework, to use the adapter we need to install GrabCaster in the BizTalk folder, GrabCaster lives inside BizTalk.
    • How the GrabCaster Logic App Connector works?
      • The logic App Web API host the GrabCaster functionality inside, it is a GrabCaster point able to receive any data from any other GrabCaster point, the points can be everywhere, on premise in the cloud or for instance embedded in a BizTalk Server, the Logic App GrabCaster API is more an adapter than a connector.
  • Embedded in libraries.

Can I execute more than one GrabCaster instance in the same machine?

Multiple points can be executed in the same machine or environment.

How the points communicate each other?

A point can communicate with the other points using different providers as Redis database and Microsoft Azure.

The engine can change the message provider to use just switching a component as below:

  • Azure Database
  • Redis Database
    • “EventsStreamComponent”: “GrabCaster.Framework.Dcp.Redis.dll”

A single point can also execute events triggers and events locally.

How the data and the events are executed?

The point picks up the data executing a trigger and it send the data to one of the other points and execute the event remotely or locally.

What exactly can be a trigger or event?

A trigger or an event can be a .Net library, a PowerShell Script or a C# script.

So can I create a trigger and event using PowerShell as well?

Yes correct, the PowerShell script interact with the GrabCaster engine automatically, what we need to do is just create a simple variable in the script.

How much complicate is create a trigger or an event?

One of my key points in the idea was the simplicity, the operation to create a trigger, event or to extend any GrabCaster functionality is very straightforward, a junior .Net developer can create a new trigger in ten minutes and an IT administrator can use PowerShell, there are many samples and templates in the community project.

What kind of operations a trigger or an event can execute?

A trigger or an event can execute any kind of operation we define, for instance insert data in a database, executing a PowerShell script or writing a file, send data to a CRM and other more.

Can I use a different message provider or logging?

Any functionality like messaging provider or logging or configuration or other is easily extendable, we can create a new messaging provider like File System, DropBox, Amazon, NServiceBus, Google Service and use it in the engine.
We can also create different new logging providers on configuration providers.

Is GrabCaster scalable?

The engine uses a complex internal throttling system to increase the messaging performances and multiple points can be executed in the same machine.

Next features?

This job is just the tip of the iceberg and I have a lot of new features in mind like the pipeline and the transformation provider, more extensibility in the GrabCaster BizTalk adapter and in the Logic App GrabCaster Adapter and more…

How much it cost GrabCaster Framework?

GrabCaster Framework is free and open source.

Where can I find more information?

I’m going to present GrabCaster next Thursday in the TUGA IT, TUGA IT is the most important community event in Europe, I’m honoured to be speaker at this event.
In TUGA IT I will also have the opportunity to explain the engine in detail during a workshop and find other passionate developers like me for the community.

The main site is http://grabcaster.io