Synchronizing Paginated Dataset by MuleSoft: An Event-Driven Approach

Introduction

Data Synchronization is a common operation in enterprise computing. It ensures consistency of data across distributed systems by a constant flow of information at scheduled intervals and sometimes near-real time. It is also important that the data synchronization patterns should be optimized for processing data as quickly as possible. MuleSoft is an excellent Integration platform that supports the implementation of most widely used Data Integration Patterns. This article explains a pattern that can be used to synchronize paginated dataset from a source to a target. This may look like a usual task, but we will discuss the problem seen with this and a solution that we propose.

Problem

Often, business use cases involve the exchange of huge volumes of data more frequently. But building an integration among such systems is really a cumbersome process. Even while dealing with paginated sources, implementing a looping construct in the flow is still unachievable using the in-house components offered by MuleSoft (though there are some components that are meant to do looping). The For-Each or the Splitter can only act upon payload of collections data type and cannot determine when to loop around the logic within itself based on specific conditions. And simulating a loop using private flows (recursion) is still unsafe and not a good solution as it tends to lock up resources (threads), which can also cause instability and demands an overhead of creating an exception handling strategy that’s complex. Also, it is usually not scalable when running on multiple workers. Additionally, having recursion implemented in Mule 4 runtimes using private flows has shown many issues pertaining to the logic involved in doing so. Using a Batch process is not ideal as well since the source is paginating the data to avoid exposing entire load and Batch, functions well only if data is ingested in high volume. So, let’s dive into a pattern which overcomes these challenges with a good decoupling between the logic and yet simulating the looping construct in a much efficient way.

from DZone.com Feed http://bit.ly/2MCxOBi