Is BizTalk server a T-Rex?

Why do people start comparing BizTalk Server to a T-Rex?

Long time ago the Microsoft marketing team created this mousepad in occasion of the BizTalk 12th birthday.

 

And my dear friend Sandro produced the fantastic sticker below

 

Time later people start comparing BizTalk Server to a T-Rex, honestly, I don’t remember the exact reasons why but I like to provide my opinion about that.

Why a T-Rex?

In the last year, I engaged many missions in UK around BizTalk Server, assessments, mentoring, development, migration, optimization and more and I heard many theories about that, let go across some of them :

Because BizTalk is old, well, in that case I’d like to say mature which is a very good point for a product, since the 2004 version the product grew up until the 2013 R2 version and in these last 10 years the community build so much material, tools and documentation that not many other products can claim.

Because BizTalk is big and monolithic, I think this is just a point of view, BizTalk can be very smart, most of the time I saw architects driving their solution in a monolithic way and, most of the times, the problem was in the lack of knowledge about BizTalk Server.

Because BizTalk is complicate, well Forrest Gump at this point would say “complicate is what complicate we do”, during my assessments and mentoring I see so many over complicated solutions which could be solved is very easy way and, the are many reasons for that, some time because we miss the knowledge, other time we don’t like to face the technology and we decide for, what I like to call, the “chicken way”.

Because now we have the cloud, well, in part I can agree with that but, believe or not, we also have the on premise and companies still use hardware, companies still integrate on premise applications and we believe or not, integrating system in productive way to send data into the cloud in efficient and reliable mode is something very complicate to do and, at the moment, BizTalk is still the number one on it.

Because BizTalk costs, the BizTalk license depends by the number of processors we need to use in order to run our solution and achieve the number of messages per second we need to consume, this is the main dilemma but, in my opinion, quite easy to solve and this is my simple development theory.
The number of messages we are able to achieve is inversely proportional to the number of wrong best practices we produce in our solution.

Many people make me this question, Nino what do you think is the future of BizTalk Server?

I don’t like to speak about future, I saw many frameworks came up and disappear after one or two years.
I like to consider the present and I think BizTalk is a solid product with tons of features and able to cover and support in great way any integration scenario.

In my opinion the main problem is how we approach to this technology.

Many times companies think about BizTalk like a single product to use in order to cover any aspect about a solution and this is deeply wrong.
I like to use many technologies together and combine them in the best way but, most important, each correct technology to solve the specific correct task.

In my opinion when we look to a technology, we need to get all the pros and cons and we must use the pros in the proper way to avoid any cons.

BizTalk can be easily extendable and we can compensate any cons in very easy way.

Below some of my personal best hints derived by my experience in the field:

If you are not comfortable or sure about BizTalk Server then call an expert, in one or two days he will be able to give you the right way, this is the most important and the best hint, I saw many people blaming BizTalk Server instead of blaming their lack of knowledge.

Use the best naming convention able to drive you in a proper way in your solution, I don’t like to follow the same one because any solution, to be well organized, needs a different structure, believe me the naming convention is all in a BizTalk Solution.

Use orchestration only when you need a strictly pattern between the processes, orchestrations are the most expensive resource in BizTalk Server, if I need to use it then I will use it for this specific reason only.

If I need to use an orchestration, then I like to simplify using code instead of using many BizTalk shapes, I like to use external libraries in my orchestrations, it’s simpler than create tons of shapes and persistent points in the orchestration.

Many times, we don’t need to use an adapter from an orchestration, which costs resources in the system, for example many times we need to retrieve data from a database or call another service and we don’t need to be reliable.

Drive your persistent points, we can drive the persistent points using atomic scopes and .Net code, I like to have the only persistent point I need to recover my process.

Anything can be extendable and reusable, when I start a new project I normally use my templates and I like to provide my templates to the customer.

I avoid the messagebox where I need real time performances, I like to use two different technics to do that, one is using Redis cache, the second is by RPC.
One of the big features provided by BizTalk Server is the possibility to reuse the artefacts separately and outside the BizTalk engine, in this way I can easily achieve real time performances in a BizTalk Server process.

Many times, we can use a Standard edition instead of an Enterprise edition, the standard edition has 2 main limitations, we can’t have multiple messageboxes and we can’t have multiple BizTalk nodes in the same group.
If the DTS (Down Time Service) is acceptable I like to use a couple of standard editions and with a proper configuration and virtual server environment I’m able to achieve a very good High Availability plan and saving costs.

I always use BAM and I implement a good notification and logging system, BizTalk Server is the middleware and, believe me, for any issue you will have in production the people will blame BizTalk, in this case, a good metrics to manage and troubleshoot in fast way any possible issue, will make you your life great.

Make the performance testing using mock services first and real external services after, in this way we are able to provide the real performances of our BizTalk solution, I saw many companies waste a lot of money trying to optimize a BizTalk process instead of the correct external service.

To conclude, when I look at BizTalk Server I don’t see a T-Rex.
BizTalk remembers me more a beautiful woman like Jessica Rabbit

 

full of qualities but, as any woman, sometime she plays up, we only need to know how to live together whit her 😉

 

 

 

CS0006 Error – Metadata file could not be found

 

In the last 2 days, I struggled against this error and it was a real nightmare and, because the solution is really wired and complicate to find I decided to keep note about that in my blog and hopefully help some other people.
One day and without any real specific reason I was not able to build my solution anymore, hundreds of CS0006 errors, so my first and usual actions were:

  1. Clean the solution, nothing…
  2. Restart VS, nothing…
  3. Clean and restart the solution, nothing…

At that point I started looking in the network and I found so many articles, tips and hints like, remove all the references and re add again, check the build settings in the configuration manager, fix precedencies but nothing and, to be honest, with 90 projects in the solution, was not a feasible option for me.

Two main assumptions drive me to the solution:

  1. Previously the solution was able to build and I didn’t touch any code
  2. I copied the solution in another machine and, with my surprise, it was able to build.

I used process explorer to check what Visual Studio does during the compilation and I noticed a quite interesting number of temporary files related to the nugget packages were created.
The length of the path generated was interesting so I decided for moving my project folder in the C:\\ ROOT and the problem has been solved.

The biggest problem I think was because I added a new nugget package to a project and VS was not able to generate specific temp files and unfortunately, I didn’t receive any exception or warning about that.

So, in case of CS0006 error the first test to do is trying to copy the solution in a shorter path and hopefully is going to fix the issue.

Holistic in the integration space – Events recap

In the last month, I’ve been speaker in two events, the WPC 2016 in Milan and the BTUG.be in Belgium, in both the events I presented the results of my studies around the holistic approach and my point of view about integration.

WPC 2016 Milan, 77 speakers for +400 attendees, makes this event the first event in Italy about IT, technology and innovation, great content and very prepared speakers.

BTUG.be is a technical event focused on Integration and organized by the BizTalk User Group in Belgium, the guys invited me 2 month ago, and I was very happy to accept because I was sure to find an audience of very strong and expert technical people.
In the BTUG.be I enjoyed all the sessions, Microsoft presented a session around integration patterns and SOLID concepts, Pieter presented a very interesting session, What’s new in BizTalk Server 2016, he has been able to present a very detailed session about the new features, Glenn, about Azure Functions, presented a very interesting point of view and of comparison between Azure Functions and the other stacks like Logic App and WebJobs.

In both the event I present my session In my session, my point of view about Integration and what means Holistic approach, I received an impressive great feedback by both the events and I’m very happy by the spontaneous feedbacks that I’m receiving via LinkedIn and email.

Looking in the feedbacks, one of the sentences which I most like is:

“finally someone that makes sense regarding integration, I know exactly what you mean with your holistic approach.
I have tried for many years to explain to people that what you’re saying, and you do it too.
So you have shown me the way”

This was exactly the scope of my session, given to people something to think about, a new view about how to use technologies and how to combine them together.
I think that, in a moment full marketing messages, thousands of technologies and options, this is one of the most important aspect of interest.

Sometime we don’t realize the potential of a technology because focused around the messages we received by the network, some other time we don’t see the technology in a different point of view.
When I approach to a new technology I’m like a child with a new toy, I normally don’t care about how the network defines or categorizes it, I just get the main marketing message and I start playing with this technology.

I like to see any technology from a different point of view, any time, this is the best approach to use to better understand the potentials.

I also use GrabCaster to explain these concepts because it contains all of my studies, GrabCaster is my personal laboratory, it’s a space which contains all of my studies and ideas, I start seeing other companies implementing patterns and concepts already implemented in GrabCaster long time ago and this is a great pleasure for me.
People are following and are looking in GrabCaster code to get new ideas and patterns like, open pattern, layering abstraction, dynamic deployment, open mediation, how to achieve real time performances in BizTalk Server and more, I still need to create more videos and tutorials and I will, family and sport activities permitting.

At the moment GrabCaster contains a lot of great features but this is probably the 10% about what I have in my mind, I will keep going implementing on it and if you like to collaborate please do it.

I will improve this session in the time and I will create more scenarios using, combining and extending what I consider the best technologies to use together at the moment.

The IT Integration market – My point of view and my next events

I’m preparing two next events, one in Italy and one in Belgium and this article wants to explain what I’m going to present and to speak about.

In the last months, I have been focused in many activities like, improve my knowledge, study new Azure stacks, improve GrabCaster with new features.
Microsoft is doing a lot of stuff around Azure, I’m impressed by the number of new features I see now in the portal and not just Microsoft but the entire IT world is producing any kind of new framework and pattern.

The IT world is changed, most of the companies now are more and more closed to use open source and I’m very happy about that.
The Cloud is now able to proposes so many options about IaaS or Application, Services and Data and more.

In my opinion Microsoft Azure now is the leader about services and integration, I’m not saying that because I’m Microsoft MVP or because I like to be closed to Microsoft, I think that because I’m not able to see the same offering in the other platforms.

Holistic education is a philosophy of education based on the premise that each person finds identity, meaning, and purpose in life through connections to the community, to the natural world, and to humanitarian values such as compassion and peace.

I think now we need to speak about holistic Integration because each technology in now able to find his identity, meaning, and purpose in the world of technology through connections to the others.

I normally use and mix all of the stacks like Service Fabric, Azure Functions, Logic App, GrabCaster, API management, Redis Cache and more together because I’m not able to achieve the best result using one only and each of these stack is absolutely specialized for a specific area.

Azure Function is something we can define as Nano Service oriented, very light and usable, very similar and, in the same time, very different from Azure Fabric for many reasons.
With Azure Fabric we have a concept of full reliable elastic scaling approach, Azure function is something we use for specific Nano tasks, but obviously we can extend the using of Functions to drive and manage specific logic in cloud.

I normally use GrabCaster for fast hybrid integration and to manage complex remotely event handling issues in on premise environment, and GrabCaster uses all the Microsoft Azure stacks to extend the capabilities and the results are impressive.

Logic App is now the workflow engine in the cloud, used to strictly correlate processes and services in the cloud, Microsoft is extending a lot this stack and we are able now to interconnect and consume the different Azure entities in fast way.

About messaging Service Bus is an ultra-mature stack but I know that Microsoft is preparing something fantastic, I can’t say anything more on that but, believe me, you will be happy.

Another interesting area is the Enterprise Integration Pack which is a complete stack we can use for EDI, VERTE and B2B, I’m waiting some more details regarding the pricing but it is awesome for fast VERTE and B2B integration.

Last and but not the least BizTalk Server which is going to the 2016 version, he is mature, solid and absolutely reliable and I have many nice surprises about BizTalk Server to present at the events.

Holistic is something characterized by the belief that the parts of something are intimately interconnected and explicable only by reference to the whole.

I always like to say…

We need to have a look to the whole if we want to be able to understand the single one.

If you are interesting to learn more about cloud then the Microsoft Academy is the best place for.

My next events will be:

WPC 2016 in Milan, the most important event in Italy about IT

You can register here.

At BTUG.be in Microsoft Belgium with the Belgium BizTalk User Group, I’m very to be there, the Belgium BizTalk User Group has many strong integration animals and great experts and I’m sure it will be a fantastic opportunity to share knowledge and good beer.

You can register here.

Looking forward to see you in one of these events.

Optimize the BizTalk productivity using ListDictionary

In this period, I’m supporting different development teams to implement a big integration solution using different technologies such as BizTalk Server, Web API, Azure, Java and other and involving many other actors and countries.
In a situation like that is quite normal to find many different problems in many different areas like integration, security, messaging, performances, SLA and more.
One of the most important aspects that I like to consider is the productivity, writing code we spend time and time = money.
Many times a developer needs to find a solution or he needs to decide a specific pattern to solve a problem and we need to be ready to provide the most productive solution for it.

For instance, a .Net developer can decide to use a plain function to solve a specific loop instead using a quicker lambda approach, a BizTalk developer can decide to use a pure BizTalk approach instead using a quicker and faster approach using code.

To better understand this concept, I want to provide you a classic and famous sample in the BizTalk planet.

An usual requirement in any solution is, for example, the possibility to pick up data in composite pattern and cycling for each instance inside the composite batch, in BizTalk server this is the classic situation where we are able to understand how much the developer is BizTalk oriented J

By nature, the BizTalk development approach is more closed to a RAD approach rather than a pure code approach.

To cycle in a composite message in BizTalk we can use different ways and looking in internet we can find many solution, one of the most used by BizTalk developer is creating an orchestration, create the composite schema, execute the mediation, receive the composite schema in the orchestration, in the orchestration cycle trough the messages using a custom pipeline called in the orchestration.

Quite expensive approach in term of productivity and performances.

Another way is using for example a System.Collections.Specialized.ListDictionary in the orchestration.

Create a variable in the orchestration type System.Collections.Specialized.ListDictionary

Create a static class and a method named for example GetAllMessages

Inside your method write the code to pick up you messages and stream and load into the ListDictionary

Create an expression shape and execute the method to retrieve the ListDictionary

xdListTransfers = new System.Collections.Specialized.ListDictionary();

MYNAMESPACE.Common.Services.SpgDAL.GetAllMessages(ref xdListTransfers,

ref numberOfMessages,

ref dataValid);

Use variable by ref to manage the result, the variable numberOfMessages
is used to cicle in the orchestration loop.

Because we use a ListDictionary we can easily get our item using a key in the list, as below using the numberOfMessagesDone variable.

xmlDoc = new System.Xml.XmlDocument();

transferMessage = new MYNAMESPACE.Common.Services.TransferMessage();

transferMessage = (MYNAMESPACE.Common.Services.TransferMessage)xdListTransfers[numberOfMessagesDone.ToString()];

Where numberOfMessagesDone
is the increment value in the loop

Using this method, we keep our orchestration very light and it’s very easy and quick to develop.

I have many other samples like that, this an argument which I’m really care about because able to improve our productivity and performance and first of all the costs.

 

Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

 

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.

 

 

        

 

 

10 Years as Microsoft MVP !

10MVP

Today Microsoft renews my MVP award for the 10th time.

I would like to tank Microsoft for these 10 years, thank to my family and my wife for the patience and the support, thank to all my MVPs friends for the great moments together in these 10 years and a huge thank you to all my followers, supporters and the integration animals around the world for the great challenges in these 10 years, next year with the new features incoming in Azure and the great job around BizTalk Server we will raise the level 🙂

10 years as MVP is a long journey, full of greats and bad moments, big challenges,  wins and losses, many conferences around the world, thousands of hours during the evenings study new patterns and writing code and a lot of time invested to share big passion to many people as possible.

Why?

Because this is a Microsoft MVP.

Proud to be a member of this awesome family.