July 2020 Project: Part 1 - Event Sourcing and CQRS

Jul 10, 2020

Intro

It is a bit late for this. But I finally got this little medium where I can store my monthly projects.

So, for those of you new to this. I have decided to do a mini project on a monthly basis. One of it is to explore new technology and toys. The other is also to upscale myself technically and push myself.

So without further ado! July’s project!

One of the frameworks I have been enamoured with since my time starting as a junior developer is that of CQRS and Event Sourcing (ES).

I came into contact with ES during my time at StashAway (for more info, you can check out the last link in the section below). And one of the tech leads there have also implemented it at Endowus when he shifted over.

So we know that there are very successful businesses and real life examples of ES. I find it extremely useful when it comes to the financial industry, due to the need to have an audit trail.

Links

For those of you who are unaware of such things, here are a few good resources to start you off:

Evaluation

One of the technologies out there to implement such a thing is Lagom. Another I can mention is Axon. But I will be focusing on the former as my primary basis of comparison because that is the framework which I use the most.

Lagom is fantastic when you consider its strengths. It is basically a microservice framework that has places for you to insert ES and CQRS stuff. Less opinionated than other frameworks.

It is batteries included, so you can fire it up, and it will handle most of the background stuff for you like the messaging and event store.

It is sweet. It is easy. If you need something that works out of the box, Lagom can pretty much do it for you.

Okay that is the good stuff.

Bad stuff. It is heavy. It is really really heavy. It runs on Scala/Java. If you run it on the former, you basically have to wrangle with sbt which is not the best and brightest of all the build tools out there.

Your messaging system is Kafka. I like Kafka a lot. But for a simple messaging service. My god it is heavy as fuck. To handle all the brokers and service discovery, you have to manage Zookeeper on top of your Kafka.

And your event store is Cassandra. Now I have a bit of a problem with this kind of architecture. When you save the events to Cassandra, and then push the message out via Kafka, there is a short window where if your stack goes down, there could be inconsistency where the event is saved, but nothing gets pushed out to the message broker.

And have you tried querying Cassandra? Don’t get me wrong. Cassandra is fantastic for writes. But the tradeoff is that it is terrible at reads. The materialised view is pretty much a broken feature. So you can handle the workload of the writes, but the price is a little too high to pay when you want to go through the events.

In my role as a startup, I have always needed to juggle multiple hats. One of the most conflicting pair of hats is that of a backend lead, and a DevOps lead.

As a backend lead, I love the batteries included part of the architecture, as it means I can just focus on writing code.

But what most backend developers don’t get, is that there is the Law of Equivalent Exchange. To quote Fullmetal Alchemist:

Humankind cannot gain anything without first giving something in return. To obtain, something of equal value must be lost.

And that loss is in the DevOps part of the equation. Having to juggle the complications of such a stack is not an easy feat, and time gained on the backend side, is lost on the DevOps side of the farm.

Motivations

Those are my own personal gripes with ES as I have experienced it through the lenses of Lagom.

So during my stint at Fullerton Health, and now at Helicap, I have moved away from it, as I felt that majority of the projects I have encountered do not require the complexity that the Lagom stack requires.

For teams with a significant investment in manpower, this won’t be an issue. But not everyone has that luxury. And even then, I feel that developers tend to have a problem.

We like to use over-engineered solutions to fix a problem. Sure, a nuclear bomb can kill an ant. But why would you do that?

In my time as a developer, I realise that this philosophy of Keep it simple, stupid (KISS), and You are not gonna need it (YAGNI) has been a pretty illuminating light so far, and has forced me to really mature as a developer.

So I intend to rebuild an ES stack with these 2 principles in mind. Everything should be nice, deployable, and distributed, without having too much strain on the DevOps or the backend developer.

Yes, folks, I want to have my cake and eat it too. And why shouldn’t I? It is 2020, surely technology has moved on.

Selection of tools

I will most likely be building my project using Go for the very simple reason of: Binaries.

Yeah. I like to deploy my event store using just a binary.

To make it more consistent, the whole stack will most likely be Go, though it can probably be polyglottic if you really insist.

  • Message Queue: NATS Streaming for easy deployment
  • Database: PostgresSQL for easier queries

And that’s about it for now.

Deliverables

I will probably be writing a bunch of articles, along with a GitHub repo with my codes on it.

A version of this will probably be deployed to production in my company Helicap, depending on the use case.

And I will try to do it by sometime in August.

Yup.

That’s all folks!

projectprogramminggojuly2020event sourcingcqrsgoprogramming

yish91

Yi Sheng is a programmer by day, and still a programmer at night. But occasionally, he loves to write nonsensical posts about various things in his life.

Hello World