The IT Integration market – My point of view and my next events

I’m preparing two next events, one in Italy and one in Belgium and this article wants to explain what I’m going to present and to speak about.

In the last months, I have been focused in many activities like, improve my knowledge, study new Azure stacks, improve GrabCaster with new features.
Microsoft is doing a lot of stuff around Azure, I’m impressed by the number of new features I see now in the portal and not just Microsoft but the entire IT world is producing any kind of new framework and pattern.

The IT world is changed, most of the companies now are more and more closed to use open source and I’m very happy about that.
The Cloud is now able to proposes so many options about IaaS or Application, Services and Data and more.

In my opinion Microsoft Azure now is the leader about services and integration, I’m not saying that because I’m Microsoft MVP or because I like to be closed to Microsoft, I think that because I’m not able to see the same offering in the other platforms.

Holistic education is a philosophy of education based on the premise that each person finds identity, meaning, and purpose in life through connections to the community, to the natural world, and to humanitarian values such as compassion and peace.

I think now we need to speak about holistic Integration because each technology in now able to find his identity, meaning, and purpose in the world of technology through connections to the others.

I normally use and mix all of the stacks like Service Fabric, Azure Functions, Logic App, GrabCaster, API management, Redis Cache and more together because I’m not able to achieve the best result using one only and each of these stack is absolutely specialized for a specific area.

Azure Function is something we can define as Nano Service oriented, very light and usable, very similar and, in the same time, very different from Azure Fabric for many reasons.
With Azure Fabric we have a concept of full reliable elastic scaling approach, Azure function is something we use for specific Nano tasks, but obviously we can extend the using of Functions to drive and manage specific logic in cloud.

I normally use GrabCaster for fast hybrid integration and to manage complex remotely event handling issues in on premise environment, and GrabCaster uses all the Microsoft Azure stacks to extend the capabilities and the results are impressive.

Logic App is now the workflow engine in the cloud, used to strictly correlate processes and services in the cloud, Microsoft is extending a lot this stack and we are able now to interconnect and consume the different Azure entities in fast way.

About messaging Service Bus is an ultra-mature stack but I know that Microsoft is preparing something fantastic, I can’t say anything more on that but, believe me, you will be happy.

Another interesting area is the Enterprise Integration Pack which is a complete stack we can use for EDI, VERTE and B2B, I’m waiting some more details regarding the pricing but it is awesome for fast VERTE and B2B integration.

Last and but not the least BizTalk Server which is going to the 2016 version, he is mature, solid and absolutely reliable and I have many nice surprises about BizTalk Server to present at the events.

Holistic is something characterized by the belief that the parts of something are intimately interconnected and explicable only by reference to the whole.

I always like to say…

We need to have a look to the whole if we want to be able to understand the single one.

If you are interesting to learn more about cloud then the Microsoft Academy is the best place for.

My next events will be:

WPC 2016 in Milan, the most important event in Italy about IT

You can register here.

At BTUG.be in Microsoft Belgium with the Belgium BizTalk User Group, I’m very to be there, the Belgium BizTalk User Group has many strong integration animals and great experts and I’m sure it will be a fantastic opportunity to share knowledge and good beer.

You can register here.

Looking forward to see you in one of these events.

Optimize the BizTalk productivity using ListDictionary

In this period, I’m supporting different development teams to implement a big integration solution using different technologies such as BizTalk Server, Web API, Azure, Java and other and involving many other actors and countries.
In a situation like that is quite normal to find many different problems in many different areas like integration, security, messaging, performances, SLA and more.
One of the most important aspects that I like to consider is the productivity, writing code we spend time and time = money.
Many times a developer needs to find a solution or he needs to decide a specific pattern to solve a problem and we need to be ready to provide the most productive solution for it.

For instance, a .Net developer can decide to use a plain function to solve a specific loop instead using a quicker lambda approach, a BizTalk developer can decide to use a pure BizTalk approach instead using a quicker and faster approach using code.

To better understand this concept, I want to provide you a classic and famous sample in the BizTalk planet.

An usual requirement in any solution is, for example, the possibility to pick up data in composite pattern and cycling for each instance inside the composite batch, in BizTalk server this is the classic situation where we are able to understand how much the developer is BizTalk oriented J

By nature, the BizTalk development approach is more closed to a RAD approach rather than a pure code approach.

To cycle in a composite message in BizTalk we can use different ways and looking in internet we can find many solution, one of the most used by BizTalk developer is creating an orchestration, create the composite schema, execute the mediation, receive the composite schema in the orchestration, in the orchestration cycle trough the messages using a custom pipeline called in the orchestration.

Quite expensive approach in term of productivity and performances.

Another way is using for example a System.Collections.Specialized.ListDictionary in the orchestration.

Create a variable in the orchestration type System.Collections.Specialized.ListDictionary

Create a static class and a method named for example GetAllMessages

Inside your method write the code to pick up you messages and stream and load into the ListDictionary

Create an expression shape and execute the method to retrieve the ListDictionary

xdListTransfers = new System.Collections.Specialized.ListDictionary();

MYNAMESPACE.Common.Services.SpgDAL.GetAllMessages(ref xdListTransfers,

ref numberOfMessages,

ref dataValid);

Use variable by ref to manage the result, the variable numberOfMessages
is used to cicle in the orchestration loop.

Because we use a ListDictionary we can easily get our item using a key in the list, as below using the numberOfMessagesDone variable.

xmlDoc = new System.Xml.XmlDocument();

transferMessage = new MYNAMESPACE.Common.Services.TransferMessage();

transferMessage = (MYNAMESPACE.Common.Services.TransferMessage)xdListTransfers[numberOfMessagesDone.ToString()];

Where numberOfMessagesDone
is the increment value in the loop

Using this method, we keep our orchestration very light and it’s very easy and quick to develop.

I have many other samples like that, this an argument which I’m really care about because able to improve our productivity and performance and first of all the costs.

 

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

 

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.

 

 

        

 

 

10 Years as Microsoft MVP !

10MVP

Today Microsoft renews my MVP award for the 10th time.

I would like to tank Microsoft for these 10 years, thank to my family and my wife for the patience and the support, thank to all my MVPs friends for the great moments together in these 10 years and a huge thank you to all my followers, supporters and the integration animals around the world for the great challenges in these 10 years, next year with the new features incoming in Azure and the great job around BizTalk Server we will raise the level🙂

10 years as MVP is a long journey, full of greats and bad moments, big challenges,  wins and losses, many conferences around the world, thousands of hours during the evenings study new patterns and writing code and a lot of time invested to share big passion to many people as possible.

Why?

Because this is a Microsoft MVP.

Proud to be a member of this awesome family.

 

Real time performances for BizTalk Server with the GrabCaster Chain Components

In the last period I worked in some new GrabCaster features like the encoding and extensibilities and about the extensibilities I implemented the possibilities to use a chain of components inside the engine.
Some people confuse the meaning of chain with the concept of pipeline but they are very different, a pipeline, as the name quote, is a pipe + line, a number of blocks executed in a sequence and a pipeline has a starting and an ending.
I like to compare the meaning using the real objects, a pipe cannot be closed otherwise the liquid stop flowing, the liquid enters in to the pipe end it come out from the end.

Interesting the idea to connect different pipelines in sequence but I see a lot more potential in the chain pattern.

A chain does not contain a real concept of starting and ending and can be closed and also executed in cycle.

Different chains can be opened and hooked together to create a longer chain and so on, I really love this concept because it gives me the opportunity to extend the capabilities in many ways and combinations.

 

A GrabCaster chain is composed by components and triggers and events can execute chains.
A chain component is a generic Interface able to ingest data, manage it and share the result with the other components in the chain.

I’m very interesting to provide more powerful options in BizTalk Server, BizTalk is a great framework and I like the idea to provide to it more interesting options and extensibility.

For that reason the first chain components I have implemented are the BizTalk pipeline component and the BizTalk transform component.
The BizTalk pipeline component executes a BizTalk pipeline DLL component and the BizTalk transform component is able to execute a BizTalk map and using a DLL like a Map storage provider, this is very powerful because it gives me the opportunity to use the BizTalk Server artefacts inside GrabCaster and use the same BizTalk features in real time performances, thousands messages a second using BizTalk Pipelines and BizTalk Transformations, I will write another post very soon about that.
The other interesting thing is the opportunity to execute GrabCaster inside BizTalk using the GrabCaster BizTalk Adapter which uses the chains as well.

To configure a chain, I use a json configuration file as below

In this pipeline I execute two chain components, the BizTalk pipeline and the BizTalk Transform.

I use a json file to configure the BizTalk pipeline component as below

Where:

AssemblyFile is the Dll component containing the BizTalk pipeline
PipelineTypeName is the pipeline type name to use
PipelinePathFile is the BizTalk pipeline file

I use a json file to configure the BizTalk transform component as below

Where

AssemblyFile is the Dll component containing the BizTalk pipeline
TransformTypeName is the transform type name to use

To assign a chain to a Trigger or event I add the chain specification to the Trigger json configuration file as below

 

I can use the chains in Triggers and Events and I can connect chains to other chains together.

In this test I imported the flat file as below

serialized as xml using the BizTalk pipeline and transform it as below

 

I will write a lot more about that in the next posts, for more information about GrabCaster you can refer here

Please contact me for any more information or if you want to collaborate inside the project.

How to use Microsoft Azure in the best way – The Microsoft Azure Dream Team

Microsoft Azure is now a consolidate technology stack able to offer a large number of technologies, in the last years I saw a lot of changes and a very important revolution inside the cloud.
I like to use any Microsoft Azure stack and I like to combine them in any way, I think this is the real power of that, being focused to use a specific technology can limit our creativity.
Entering in the Azure portal https://portal.azure.com and browsing the features is the best way to realize how many things we can use now, I think is impressive, it’s very important to understand the best strategy to combine them to obtain the best result in fast, smart and best productivity way.

I spent the last ten years using BizTalk Server, I definitely love this framework, this framework has born during the 2000 and with the 2004 version it has opened the way for the new frontiers in the world of integration.
BizTalk Server was the first one able to offer concept like reliability, mediation, adaptation, pub/sub, abstraction and more, all the other frameworks copied the concept and they have got the ideas from BizTalk Server, in the same time Microsoft started extending this idea 8 years ago creating the concept of BizTalk Services, some people still trying to correlate this concept with a specific Azure technology but in my opinion this is wrong.
The concept of BizTalk Services in Azure is a group of technologies and not a specific one, best way to understand the concept is doing a nice exercise translating the BizTalk Server stacks into the cloud.

Below the most famous picture and I love it, the story of integration, what we used for years to explain to people how to integrate technologies and to explain these concepts, the BizTalk messaging flow.

Below how I translate it using the most relevant Azure stacks.

These stacks are the most relevants and I like to call them the Azure Dream Team, below some of my preferred.

API Management is everywhere and it provide anything we need for communication, mediation, security, services management and governance, metrics and more.
Simple to use using the UI and it’s able to solve many complicate problems, for example like security and metrics, in very simple way, Microsoft has done a great job on it.

Service Bus is the door and the internal neural system in the cloud, it covers the communication with relays for example, the messaging aspect, the reliability using queues and the pub/sub using topics.

Service Fabric is the stack which covers the reliability and scalability, all the Microsoft technologies use this stack, starting from Skype for business to SQL Server or the entire Microsoft Azure.
The clever idea in this stack is the possibility to make your code reliable just using specific Reliable Collections and Reliable Queues, if you need to be reliable and scalable this is the stack to use.

Logic App is the way to cover the integration and mediation aspect, not only, Logic App is able to offer a very simple way to build flows in the cloud, this because Microsoft starts using the term Microsoft flow.
Logic App contains a lot of connectors, the concept of mediation and of BizTalk, we can extend the capabilities using the Logic App functions which are the way to provide scripting code in the blocks and the possibility to use maps and transformations as well.
Logic App is the stack we need to use to provide orchestration, mediation, transformation and strictly patterns to drive our processes in the cloud like Web API which is the way to build our application blocks and the Web Jobs for background processing.

About transformations we can also use Stream Analytics, the concept is very clever, receive streams from any other stack, use an internal scripting language based on a very well-known language like TSQL and push the stream to any other Azure stack.

EventHubs is the messaging ingestor, millions of messages a second, the interesting aspect is the possibility to use it as pub/sub engine, it is fast and reliable, for performances reasons the message size limit is 256 Kb and we can use Blobs as temporary message storage, this is for example one of the ways used by GrabCaster to exchange messages between the points.

About devices we can use Azure IoT Hub which covers any specific aspect related to IoT, one of the thing I more like is the easy way to create IoT scenarios in a minute using the internal templates, IoT is a very complex topic and this is the best way to understand how to use this technology in the best way.

There are many other stacks to use for any specific topic, I will write more in the next articles.

 

BizTalk Migration and Moving Risks Board White Paper

I like to share a couple of my white papers, I normally use these during a BizTalk migration, moving or update.
A BizTalk Server migration or moving is a good opportunity to review our environment, one of the most critical point are the risks we can find and avoid in the during.
There are a lot of issues to consider, some of these are very simple and, because simple, are the most dangerous.
The best thing to do is using a reference document to keep track about our risks, I like to call it the Risk Board.
In the Risk Board I normally put everything I think needs to be considered and covered, some of these points are critical by personal experience and, if we cover them in the first period, we will be able to do a great job.

I will keep update these whitepaper, please feel free to use them and to collaborate with your feedback and experience.

You can find the white papers here in my BizTalk documents folder on GitHub.

https://github.com/ninocrudele/BizTalk-Documents