Filling the Cloud Data Integration Gap

The cloud as we know it is here to stay. Given how quickly it has been developed, deployed, and adopted, coupled with its pay-as-you-go pricing model, it’s no wonder why so many of today’s top organizations and new entrants are continuing to embrace all the cloud has to offer. However, as a data management professional, there has always been one thing I felt was missing from the cloud: Data Integration.

What is Data Integration?

Simply put, data integration encompasses how data moves between systems. For example, suppose you have a Instagram account that is connected to your Facebook and Twitter accounts. When you post a picture to your Instagram account, you have the option of also posting that same picture to your Facebook and/or Twitter accounts as well without separately logging into those accounts and reposting the same picture. This, my friends, is basic data integration in a nutshell. However, let’s think about if from the cloud standpoint.

Data integration in the cloud today

In the cloud, every service offering breaks down into 2 essential pieces: storage and compute. That’s it! And if you think about it, all of computing as we know it, cloud or not, revolves around these 2 key ingredients.

Now, knowing that the cloud is rooted in compute and storage, you could imagine that developers simply create programs or API’s that connect systems and move data between them. You’d be right in thinking this; it happens on a majority of data integration use cases. So, if data integration is happening today, where is the gap? In a word: Transformations.

Filling the transformations gap

Transformations, as it relates to data integration, deals with how data should be “massaged” (prepared and validated, for example) while it is being transmitted between systems. Most of today’s modern programming interfaces allow for easy connections to be made between systems, however there is a clear gap between connecting to a system and preparing the data so that the systems can interface with the data. This fundamental gap lies in skillsets (between app and data developers) as well as technologies (API’s versus data integration tools). In my opinion, this represents the data integration tools gap in the cloud today, and is something that I have often felt has not been closed in a long time…until now.

The integration Platform as a Service – iPaaS

The integration Platform as a Service (iPaas) represents a niche cloud architecture that solves for a specific demand in the marketplace that is too broad for the traditional cloud Platform as a Service (PaaS) architecture to handle.

Specifically, iPaaS platforms allow for systems to not only connect with each other but also add a layer of visibility into sophisticated transformations, validations, and any other manner of massaging data during transmission. Even better, these platforms are built for both app and data developers, thus closing the skills gap and allowing both parties to develop as they were trained.

Recently, I have had the privilege of working with one of the leaders in the iPaaS space: Dell Boomi. The Boomi organization was acquired by Dell back in November 2010, however they have retained autonomy from Dell.

Working with Boomi in recent months has lead me to realizing that many of the gaps I noted earlier are being closed and that those of us in the data management space are finding that our skills around data integration and transformations are being used in new ways in the cloud. Where this platform really excels is in a couple of areas:

  1. Seamless integration with many of today’s hot systems (Salesforce, Facebook, Google Cloud, to name a few).
  2. Cloud platform agnostic – meaning you can use Dell Boomi no matter which cloud platforms (AWS, Azure, Google Cloud, IBM) you are using today.
  3. Boomi has many built-in connectors to external systems but also comes with the ability to develop your own connectors for custom or lesser known systems.
  4. No cloud? No problem! Boomi is advanced enough to integrate with completely on-premises systems. For instance, Boomi could be used to integrate your financial reporting data in Microsoft SQL Server with your Oracle EBS financial management system.

There are many other areas where this platform excels, but at the end of the day, the platform really works to close significant people, process, and technology gaps that have been present in the cloud data space for many years.

Is iPaaS the final solution?

No way! In this great, big arena of cloud and its ever increasing velocity of services, the chances of finding one solution that does everything for your organization is very, very slim. While iPaaS platforms like Boomi offer a complete data integration solution in the cloud, there are other aspects of the entire data lifecycle we have to take into consideration – think data governance as it relates to data quality.

Nevertheless, iPaaS does represent a huge leap forward in creating a more holistic method of next generation data integration in the cloud.

Site Footer

Sliding Sidebar

About Me

About Me

I love empowering people and organizations to use data! I’m a seasoned enterprise data architect and strategist with over 10 years of experience delivering high value data and analytics solutions. I am well versed in a multitude of different industries ranging from global insurance agencies to local non-profit organizations. I have excelled in developing and implementing all aspects of enterprise architecture solutions using solid best practices and best-in-class technology.

Social Profiles

Tweets