Microservices in the world of Big Data
. . . Why it is more relevant than ever before
Though the concept is simple, implementing Microservices can easily get complicated for a large scale system. It’s important to have the Big Picture in mind before an Organization moves to the world of Microservices. Let’s take a real world example and look at a legacy application (Fig. 1) that was very much a standard just a couple of years back:
Fig 1
This enterprise-ish architecture served us pretty well. After all, the responsibilities are pretty well segregated. It has a perfect Data abstraction layer, business layer and a presentation layer. It’s loosely coupled and follows the best design patterns. So what possibly is the motivation to move away from such an architecture?
Enter Big Data. By now it’s a pretty accepted fact that Big Data is not about just volume. There are other parameters like Velocity, Variety and Veracity which play an equally important role in the Big Data world. It’s about getting insights from data and feed them back to your organization to improve your overall business. Add to this, the business wants things instantly. The conventional legacy monolithic applications are not very well suited to achieve the current dynamic environment that we are in.
Let’s focus on the above architecture. As a seasoned developer and architect, you would figure the biggest problem is that this is a monolithic application. Any small change that you do, you need to pretty much deploy all your binaries. And then it needs to follow your org’s defined deployment policies. Depending on how big or small your organization is, every single deployment can take somewhere between weeks to months. In today’s world such delays are hardly acceptable. To some extent this is addressed by following Agile development, Continuous Integration and releases. But with increased complexity, addressing the above challenge is hardly possible.
It is now almost a certainty that your organization is not about one technology and couple of Big Applications. You would probably want to use something like Python, R or F# to do your analytics. Java and C# probably is your go to programming language for bulk of your Business and web interface. You must have invested in NoSQL solution by now to store different type of data in different databases. Of course Mobility is now a bigger focus and you would have Android and iOS development. And this is now 2016. Can you afford not to make everything real-time and not to make IoT enabled? All these need to co-exist and integrate to give the optimal value.
These are some of the challenges a Microservice tries to address. Each micro-service is an independent deployable component. Integration amongst Microservices is the core of the design. So let’s refactor our existing legacy application to a Microservices based architecture:
Fig 2 
The above diagram (Fig 2) might look like a massively complex bit of work. And in a way it is. The complexity arises mainly in the integration of Microservices. What if one service is not available? How do I manage and monitor potentially 10’s of Microservices? How can I maintain a predictable performance with increased load?
If addressed properly in the early stages of development, the long term benefits are too apparent. Each Microservice can be replaced by a complete new implementation in a matter of days. So rather than being stuck with the legacy outdated codebase, you may move to any next generation implementation in a matter of days. Since the Microservices are completely independent, they don’t need to be on the same technology. You want a recommendation with a GraphDB at the back; No problem. You want a search service with a rules engine integration; Easy. How about a Real Time analytics and do an A/B testing more often; Again possible.
Microservices might not be the silver bullet solution for all your problems. If done properly, it can certainly address quite a few of the challenges that today any organization faces. The Microservices technology stack is pretty matured and continues to evolve. Tech savvy companies like Google, Netflix and Uber are putting their weight behind this. If there’s a good time to implement Microservices in your organization, it’s now.