Big Data integration in high scale scenario using Azure Service Fabric and GrabCaster

Big Data is one of the most important topic in the last year, the world of integration has changed since the companies start providing the possibility to store our information in the Cloud.
Send the data from our on premise to a database in the cloud can be achieved in different ways, I’m very keen and focused to implement feature in GrabCaster able to solve integration problems in easy way.

GrabCaster is an open source framework and it can be used in every integration project within any environment type, no matter which technologies, transport protocols or data formats are used.

The framework is enterprise ready and it implements all the Enterprise Integration Patterns (EIPs) and offer a consistent model and messaging architecture to integrate several technologies., the internal engine offers all the required features to realize a consistent solution.

The framework can be hosted in Azure Service Fabric which enables us to build and manage scalable and reliable points running at very high density on a shared pool of machines.
Azure Service Fabric servers as the foundation for building next generation cloud and on premise applications and services of unprecedented scale and reliability, for more information about Microsoft Azure Service Fabric check here

Using GrabCaster and the Microsoft Azure stacks I’m able to execute a SQL Server Bulk Insert operation across on premise and the cloud very easily, below how it works.

I’m not going in detail, you can find all the samples and templates in the GrabCaster site.

Download GrabCaster Framework and configure it to use the Azure Redis Cache, this is one of the last messaging provider implemented, what I love about this Azure stack is the pricing and the different options offered by this framework.

Using the SQL Bulk Trigger and Event I’m able to send a large amount of data across the cloud, the engine compacts the stream and it uses the Blobs to move the large amount of records.

Last the I did was moving one million record from a table in an on premise SQL database and another in the cloud.

I installed GrabCaster in Azure Fabric as below.

I configured the SQL bulk trigger in the on premise environment using the json configuration file.

I configured the SQL Bulk event and I sent the configuration to the GrabCaster point in the Azure Fabric using the internal synchronization channel.

I can activate the GrabCaster trigger in different ways, in this case I used the REST API invocation as below.

The trigger is executed and I moved one million records from an on premise SQL Server database into a SQL Server database table in the cloud.

After tested the REST API the developer implemented a simple REST call in a Web UI button.

Below the scenario implemented.

    

 

There are some important aspects which I appreciate in this approach.

  1. Thank to Microsoft Azure Fabric the solution is always-on, scalable and distributed.
  2. The simplicity into the approach and the configuration.
  3. The extensibility using the REST API call to invocate the trigger.
  4. The using of Redis Cache and Blob which have a very low pricing consume in Microsoft Azure.

 

 

        

 

 

10 Years as Microsoft MVP !

10MVP

Today Microsoft renews my MVP award for the 10th time.

I would like to tank Microsoft for these 10 years, thank to my family and my wife for the patience and the support, thank to all my MVPs friends for the great moments together in these 10 years and a huge thank you to all my followers, supporters and the integration animals around the world for the great challenges in these 10 years, next year with the new features incoming in Azure and the great job around BizTalk Server we will raise the level🙂

10 years as MVP is a long journey, full of greats and bad moments, big challenges,  wins and losses, many conferences around the world, thousands of hours during the evenings study new patterns and writing code and a lot of time invested to share big passion to many people as possible.

Why?

Because this is a Microsoft MVP.

Proud to be a member of this awesome family.

 

Real time performances for BizTalk Server with the GrabCaster Chain Components

In the last period I worked in some new GrabCaster features like the encoding and extensibilities and about the extensibilities I implemented the possibilities to use a chain of components inside the engine.
Some people confuse the meaning of chain with the concept of pipeline but they are very different, a pipeline, as the name quote, is a pipe + line, a number of blocks executed in a sequence and a pipeline has a starting and an ending.
I like to compare the meaning using the real objects, a pipe cannot be closed otherwise the liquid stop flowing, the liquid enters in to the pipe end it come out from the end.

Interesting the idea to connect different pipelines in sequence but I see a lot more potential in the chain pattern.

A chain does not contain a real concept of starting and ending and can be closed and also executed in cycle.

Different chains can be opened and hooked together to create a longer chain and so on, I really love this concept because it gives me the opportunity to extend the capabilities in many ways and combinations.

 

A GrabCaster chain is composed by components and triggers and events can execute chains.
A chain component is a generic Interface able to ingest data, manage it and share the result with the other components in the chain.

I’m very interesting to provide more powerful options in BizTalk Server, BizTalk is a great framework and I like the idea to provide to it more interesting options and extensibility.

For that reason the first chain components I have implemented are the BizTalk pipeline component and the BizTalk transform component.
The BizTalk pipeline component executes a BizTalk pipeline DLL component and the BizTalk transform component is able to execute a BizTalk map and using a DLL like a Map storage provider, this is very powerful because it gives me the opportunity to use the BizTalk Server artefacts inside GrabCaster and use the same BizTalk features in real time performances, thousands messages a second using BizTalk Pipelines and BizTalk Transformations, I will write another post very soon about that.
The other interesting thing is the opportunity to execute GrabCaster inside BizTalk using the GrabCaster BizTalk Adapter which uses the chains as well.

To configure a chain, I use a json configuration file as below

In this pipeline I execute two chain components, the BizTalk pipeline and the BizTalk Transform.

I use a json file to configure the BizTalk pipeline component as below

Where:

AssemblyFile is the Dll component containing the BizTalk pipeline
PipelineTypeName is the pipeline type name to use
PipelinePathFile is the BizTalk pipeline file

I use a json file to configure the BizTalk transform component as below

Where

AssemblyFile is the Dll component containing the BizTalk pipeline
TransformTypeName is the transform type name to use

To assign a chain to a Trigger or event I add the chain specification to the Trigger json configuration file as below

 

I can use the chains in Triggers and Events and I can connect chains to other chains together.

In this test I imported the flat file as below

serialized as xml using the BizTalk pipeline and transform it as below

 

I will write a lot more about that in the next posts, for more information about GrabCaster you can refer here

Please contact me for any more information or if you want to collaborate inside the project.

How to use Microsoft Azure in the best way – The Microsoft Azure Dream Team

Microsoft Azure is now a consolidate technology stack able to offer a large number of technologies, in the last years I saw a lot of changes and a very important revolution inside the cloud.
I like to use any Microsoft Azure stack and I like to combine them in any way, I think this is the real power of that, being focused to use a specific technology can limit our creativity.
Entering in the Azure portal https://portal.azure.com and browsing the features is the best way to realize how many things we can use now, I think is impressive, it’s very important to understand the best strategy to combine them to obtain the best result in fast, smart and best productivity way.

I spent the last ten years using BizTalk Server, I definitely love this framework, this framework has born during the 2000 and with the 2004 version it has opened the way for the new frontiers in the world of integration.
BizTalk Server was the first one able to offer concept like reliability, mediation, adaptation, pub/sub, abstraction and more, all the other frameworks copied the concept and they have got the ideas from BizTalk Server, in the same time Microsoft started extending this idea 8 years ago creating the concept of BizTalk Services, some people still trying to correlate this concept with a specific Azure technology but in my opinion this is wrong.
The concept of BizTalk Services in Azure is a group of technologies and not a specific one, best way to understand the concept is doing a nice exercise translating the BizTalk Server stacks into the cloud.

Below the most famous picture and I love it, the story of integration, what we used for years to explain to people how to integrate technologies and to explain these concepts, the BizTalk messaging flow.

Below how I translate it using the most relevant Azure stacks.

These stacks are the most relevants and I like to call them the Azure Dream Team, below some of my preferred.

API Management is everywhere and it provide anything we need for communication, mediation, security, services management and governance, metrics and more.
Simple to use using the UI and it’s able to solve many complicate problems, for example like security and metrics, in very simple way, Microsoft has done a great job on it.

Service Bus is the door and the internal neural system in the cloud, it covers the communication with relays for example, the messaging aspect, the reliability using queues and the pub/sub using topics.

Service Fabric is the stack which covers the reliability and scalability, all the Microsoft technologies use this stack, starting from Skype for business to SQL Server or the entire Microsoft Azure.
The clever idea in this stack is the possibility to make your code reliable just using specific Reliable Collections and Reliable Queues, if you need to be reliable and scalable this is the stack to use.

Logic App is the way to cover the integration and mediation aspect, not only, Logic App is able to offer a very simple way to build flows in the cloud, this because Microsoft starts using the term Microsoft flow.
Logic App contains a lot of connectors, the concept of mediation and of BizTalk, we can extend the capabilities using the Logic App functions which are the way to provide scripting code in the blocks and the possibility to use maps and transformations as well.
Logic App is the stack we need to use to provide orchestration, mediation, transformation and strictly patterns to drive our processes in the cloud like Web API which is the way to build our application blocks and the Web Jobs for background processing.

About transformations we can also use Stream Analytics, the concept is very clever, receive streams from any other stack, use an internal scripting language based on a very well-known language like TSQL and push the stream to any other Azure stack.

EventHubs is the messaging ingestor, millions of messages a second, the interesting aspect is the possibility to use it as pub/sub engine, it is fast and reliable, for performances reasons the message size limit is 256 Kb and we can use Blobs as temporary message storage, this is for example one of the ways used by GrabCaster to exchange messages between the points.

About devices we can use Azure IoT Hub which covers any specific aspect related to IoT, one of the thing I more like is the easy way to create IoT scenarios in a minute using the internal templates, IoT is a very complex topic and this is the best way to understand how to use this technology in the best way.

There are many other stacks to use for any specific topic, I will write more in the next articles.

 

BizTalk Migration and Moving Risks Board White Paper

I like to share a couple of my white papers, I normally use these during a BizTalk migration, moving or update.
A BizTalk Server migration or moving is a good opportunity to review our environment, one of the most critical point are the risks we can find and avoid in the during.
There are a lot of issues to consider, some of these are very simple and, because simple, are the most dangerous.
The best thing to do is using a reference document to keep track about our risks, I like to call it the Risk Board.
In the Risk Board I normally put everything I think needs to be considered and covered, some of these points are critical by personal experience and, if we cover them in the first period, we will be able to do a great job.

I will keep update these whitepaper, please feel free to use them and to collaborate with your feedback and experience.

You can find the white papers here in my BizTalk documents folder on GitHub.

https://github.com/ninocrudele/BizTalk-Documents

GrabCaster Framework – Top 20 Questions and Answers

During the Integrate 2016 Sandro shown a demo using one of the GrabCaster features, the BizTalk GrabCaster Adapter and I anticipated a bit about the GrabCaster Framework, many people made me many questions like how GrabCaster works, what GrabCaster is, the story of it and so on, so I decided to group the top 20 before the TUGA IT.

company logo.fw

When you started the development?

I started the development 2 years ago and during the spare time, it was really hard but thank to my wife Grazia and my family I have been able to keep going and terminate the idea, well to be honest my wife hates GrabCaster now J.

Where the idea comes from?

I spent the last 15 years in the integration using any kind of software and I had a look in any framework and one of the aspect normally annoy me is the complexity to integrate technologies and exchange data, sometime because the framework is huge and we only need to do some specific tasks, other time because the framework is too complex inside, other time because we need to use a black box and we are forced to customize and use tricky strategies to solve a problem and last but not the least the costs.

GrabCaster is a framework able to Grab and Cast data across any network, Grab + Cast + (er) J.

What do you like more about GrabCaster?

I like the simplicity to solve problems and to integrate technologies, in my opinion GrabCaster is unique in the category of the integration frameworks, I’m not able to find something similar, I’m sure many companies and people will use the same approach in the future, it’s a new approach a people need I love the idea and I love speaking about it with other people and get opinions and new ideas.
I want to dedicate this huge job to my Dad, he was a star in his job and the most unique person I had the honour to know in my life.

What GrabCaster is?

GrabCaster is something completely new, it uses a completely new concepts to exchange data across the network and to integrate technologies, it’s clever and smart to use.
The engine is an integration framework based on a loosely couple event handling approach, a point opens a channel and any point can exchange messages with any active point.
The approach to send information is open and using any kind of pattern like Pub/Sub, Point 2 Point, Peer 2 Peer, 1 To 1, 1 To N, N To 1, N To N, Synchronous and Asynchronous ad other…

How GrabCaster connect the endpoints?

Unlike the other frameworks, GrabCaster doesn’t use a point to point approach for instance using lines, arrows or creating subscribers, a point is already connected because it exists and active and it can receive events any time from any other point.

How GrabCaster can be used?

GrabCaster is polymorphic and it supports different way to be executed and used.

GrabCaster Framework is the main engine and there are many other features.

The Framework can be executed in different way:

  • As Console Application
  • As Windows NT Service, the framework exposes all the functionality as Windows NT Service.
  • Embedded in other applications and frameworks
    • When a framework embeds GrabCaster it hosts GrabCaster inside and it get all the functionalities, for instance, in case of the BizTalk Adapter and the Logic App Adapter, GrabCaster is inside BizTalk Server and inside the Logic App connector, the BizTalk Adapter and the Logic App Adapter are just two of the GrabCaster features.
    • About the BizTalk GrabCaster Adapter, where exactly GrabCaster runs?
      • When the GrabCaster Adapter is executed BizTalk Server uses the GrabCaster functionalities inside, GrabCaster is inside BizTalk Server and BizTalk extend the functionality using the GrabCaster Framework, to use the adapter we need to install GrabCaster in the BizTalk folder, GrabCaster lives inside BizTalk.
    • How the GrabCaster Logic App Connector works?
      • The logic App Web API host the GrabCaster functionality inside, it is a GrabCaster point able to receive any data from any other GrabCaster point, the points can be everywhere, on premise in the cloud or for instance embedded in a BizTalk Server, the Logic App GrabCaster API is more an adapter than a connector.
  • Embedded in libraries.

Can I execute more than one GrabCaster instance in the same machine?

Multiple points can be executed in the same machine or environment.

How the points communicate each other?

A point can communicate with the other points using different providers as Redis database and Microsoft Azure.

The engine can change the message provider to use just switching a component as below:

  • Azure Database
  • Redis Database
    • “EventsStreamComponent”: “GrabCaster.Framework.Dcp.Redis.dll”

A single point can also execute events triggers and events locally.

How the data and the events are executed?

The point picks up the data executing a trigger and it send the data to one of the other points and execute the event remotely or locally.

What exactly can be a trigger or event?

A trigger or an event can be a .Net library, a PowerShell Script or a C# script.

So can I create a trigger and event using PowerShell as well?

Yes correct, the PowerShell script interact with the GrabCaster engine automatically, what we need to do is just create a simple variable in the script.

How much complicate is create a trigger or an event?

One of my key points in the idea was the simplicity, the operation to create a trigger, event or to extend any GrabCaster functionality is very straightforward, a junior .Net developer can create a new trigger in ten minutes and an IT administrator can use PowerShell, there are many samples and templates in the community project.

What kind of operations a trigger or an event can execute?

A trigger or an event can execute any kind of operation we define, for instance insert data in a database, executing a PowerShell script or writing a file, send data to a CRM and other more.

Can I use a different message provider or logging?

Any functionality like messaging provider or logging or configuration or other is easily extendable, we can create a new messaging provider like File System, DropBox, Amazon, NServiceBus, Google Service and use it in the engine.
We can also create different new logging providers on configuration providers.

Is GrabCaster scalable?

The engine uses a complex internal throttling system to increase the messaging performances and multiple points can be executed in the same machine.

Next features?

This job is just the tip of the iceberg and I have a lot of new features in mind like the pipeline and the transformation provider, more extensibility in the GrabCaster BizTalk adapter and in the Logic App GrabCaster Adapter and more…

How much it cost GrabCaster Framework?

GrabCaster Framework is free and open source.

Where can I find more information?

I’m going to present GrabCaster next Thursday in the TUGA IT, TUGA IT is the most important community event in Europe, I’m honoured to be speaker at this event.
In TUGA IT I will also have the opportunity to explain the engine in detail during a workshop and find other passionate developers like me for the community.

The main site is http://grabcaster.io

Secure and consume your services using Azure API Management, my thoughts

In this last period I spent some time looking in Azure API Management and Azure security, the topic is quite complex and it required me a big effort to figured out how everything is able to work together, below a recap of my best thoughts and what we need to know to decide if API Management is the right technology to use for our problem.

I love Azure API Management (APIM) because is able to keep simple aspects that normally complicate are, like security services, services contracting harness,  services management, metrics and governance.

Activate APIM is straightforward, you can refer to this great Steve Danielson‘s article here

First consideration to do is the services location which could be on premise or in the cloud.
if the API is in the cloud we don’t have any particular issue to cover, different situation if the service in on premise.

About the on premise side I had a look around and discussed with other guys in order to see different opinions, below the winners.

First Option put in security you on premise API using SSA or certificate and use 443 port.

Second option using application Service Environment (ASE),  essentially it provides a virtual proxy IP environment approach so we need to replicate our “proxy service” layer in the cloud and inside ASE, a good start article here.

Second option activate the Premium Tier in the APIM and use a VPN approach, this approach is more expensive than ASE but it does not required any more effort in term of service management and governance, you services in the cloud and in the on premise side will be considered in the same way by APIM, a good article here.

About the security aspect I tried 2 options, Azure Active Directory and JWT token, both of them quite easy to set up using APIM because it is able to offer a very nice console to use.

Personally I like to use the  open standard for authorization (OAuth), APIM offer the evolution 2.0 the OAuth2.

sec

The concept in APIM is very clever, you decide your security provider and after that you only need to create a new authorization server, when you have your authorization server you can organize the security of your services assigning them to it, that awesome because you can manage a great granularity.

sec2

there are many great features to use in APIM, what I love most are policies, using policies you can control everything you need of your services starting from security like JWT, service consuming and more.

pol1

Over the simple things like control limit rate and bandwidth, but very complicate to realize using a classic code approach😉, great point about policies, I like the idea to execute C# scripting inside them, in that way I can extend the services behaviors in fast and easy way, you can also set a strong granularity control of you service caching.

My vote on this Azure stack?  Awesome Azure Stack, I really recommend to use APIM in all Azure solution.

I have my personal Microsoft Azure stacks ranking, APIM is in the top level with EventHubs and Logic Apps new version.