Building a distributed framework in Go- 1 [3AE]

So, after asking myself whether I should do this or not, I’ve finally reached an answer. And the answer is, no one cares. So, why not?

The Setup

Hear me first, I’m a big Go fan. Even though I work mostly with Python and Pyspark nowadays, I keep thinking about old and nostalgic days when I was new to Go and how cool it was to write those interfaces and packages and build those binaries. I know, that’s just the nerd in me geeking out. But that’s just how it is. So I decided, if distributed systems is something that really interests me, why not make one. And since Go is something I’m really fond of, why not in Go. And so, here we are.

Fun fact, if you YouTube search “Go song”, you’ll find this fun piece of music.

I started reading a bit about how can one make a simple distributed system in any language. Based on that, this is what I want it to do:

1. User submits a Pyspark look-a-like job which involves reading data from a source and post some processing or transformation, stores the output.
2. A driver node, which takes user job as an input and then distributes it among set of executor nodes (call it a docker container for now).
3. The executor node runs its share of the job and generates aggregated output.
4. Once done, it returns the output to the driver, which in turn returns it to the user.
5. User shuts down the workstation and go out to enjoy the weather.

This Weather = from November to February (credits to North India)

Easy peasy, right?

The Build

Based on the above narrative (references Westworld), following should be the components which exist. And already, I’m not getting a great feeling about this:

1. The job submitter, which takes user input of the job to execute.
2. The job scheduler, which schedules job to run on executor once they have the capacity to do so. (side mission)
3. The job executor(s), which takes its share of the job as an input, runs it and finally returns the output to the submitter.

If I can sum everything into a easy-to-understand diagram, this is how it looks in my head. Again, in bold letters, in my head.

And mind you, this is only the first half of the movie. If you’ve seen enough such diagrams, you can tell that the in the 2nd half, the arrows will be reversed. (see, instead of making another one like this, I put the work on the reader)

Like every other distributed architecture diagram, am I right?

Other than this, following are some additional concepts that we need to be aware of:

The reader after going through the 3rd such list in a single article.

API: In context of distributed systems, It is a standardized way for different components or systems to communicate and exchange data with each other over a network.

gRPC: A high-performance, open-source framework that is used to build remote procedure call (RPC) systems. It allows you to define a service interface using Protocol Buffers, which is a language-agnostic data serialization format.

BTS or Block Transfer Service: Responsible for facilitating data transfer between nodes during a shuffle operation.

Protocol Buffer: A language- and platform-neutral data serialization format used to transmit structured data between different services or applications.

TaskSet: For execution, A job / DAG (wth is a DAG?) is converted into a serialized form known as a TaskSet. A TaskSet is a collection of tasks that can be sent to a single executor for execution

And yes, not all of this was just a simple google search. I had help from ChatGPT for sure. Its a freaking awesome piece of technology (*cries in corner with fear of getting replaced by it in next 5 years or less*).

This was not made by AI

With all that being said, this series will focus on building each of these components. What I will attempt to do is (if I don’t have a nervous breakdown when I’m unable to crack something) to make them and run a normal PySpark look-a-like job on it and use as much Golang as possible.

The Payoff

Enough talk, lets try to build this.

All the code can be found here (nothing but initial commit at the time of writing). It won’t be possible for me to list down all the steps in building such a massive project, so I’ll try to post the intuition around each of its components in coming articles. For today, we can start this with:

Job Submitter

We can start with the following features:

1. It is hosted as a microservice, and has an API, let’s say /submit (pretty intuitive huh!). Let’s skip authentication for now (just the way I skipped the topic morphing in my CG class and then wrote 3 pages of incorrect information in the finals).
2. The user writes a flicker job (why flicker you may ask? why spark then I ask). Eventually the user triggers the execution of the job by calling some magic function like like count() or execute() or wingardium_leviosa().
3. The job submitter takes the user input and convert it into a TaskSet, which can then be communicated directly to the executors for now.

Again, all of this can certainly be something that you don’t agree with. To that I just have one thing to say, quoting Polonius:

“To each, its own”

I’ll try to keep working on the repository and we shall revisit the concepts in the next one in this series.

If the idea of the project seems interesting to you, contributors are always welcome.

Polyphia is one band I’m listening to pretty much regularly. Seems one of a kind.

Other than this, few recommendations would include: John Wick 4, Smile, Severance and The Last of Us.

Akira is one old movie which inspired quite a lot of new SciFi cinema. See this fun compilation video:

That’d be all. See you in the next one…

:)