2024: A retrospective

It’s been too long a year, I feel. Quite a lot happened in the last 365 days, and I’m here to discuss the ones that fit here.

Stuff I learnt

I learnt how different working in a big MNC and a growing startup is. And spoiler alert, its similar to living in 2 different universes. I can’t say which one makes more sense, it totally depends on what stage of life you’re at.

Its a ride for sure, doesn’t matter how long you stay in it, its a ride, day in and day out. Some interesting things I worked with in last 1 year:

1. Databricks — Jobs, Workflows, Service principals, Cluster sizing, Spark metrics, CI/CD, Deployments, Fleet clusters, Analyze, Optimize, Vaccum, Z-Order
2. AWS — S3, Glue, Athena, Lake Formation, Secrets Manager, Lambda, Sagemaker, Batch operator, SQS, SNS
3. Checking up on quality of data — Great Expectations, Deequ
4. Deploying stuff — ArgoCD, Jenkins
5. Stuff on visualizing/UIs — Superset, Streamlit, ReactJS, Gradio
6. The whole schebang of RBAC — Apache Hadoop, Apache Ranger, Hive Metastore, Okta
7. Managing data — Data mart strategies, Data lakehouses, backfilling them, performing PITR, checkpointing
8. Lakehouse tech — Delta, Iceberg, metadata manipulation, Deletion Vectors
9. Machine Learning & more— NER, Fennel AI, Basics of Transformers & LLMs, Frontier models, Ollama
10. Oncall :)

Stuff I read

Not everything that I consumed ideas from was in hard copy. And since you can read literally anything and everything, it makes more sense to mention books from completely different topics/walks of life. The top few books which taught me more than people were:

1. Principles by Ray Dalio
2. Predictably Irrational
3. Hooked
4. One Up The Wall Street
5. Designing Machine Learning Systems

Stuff I watched

Quite a lot of time was spent in watching and re-watching content of all form. Ideally YouTube should do a Recap as well. Some great shows I caught up/discovered:

The Penguin
Monster (2004)
The Sympathizer
Patriot
Lastman
Station Eleven
The Little Drummer Girl

If I had a plug to Letterboxd, I’d share that before making movie suggestions. Regardless, some cinema I’d recommend that I saw last year:

Being John Malkovich
Dune 2
Dial M for Murder
Dead Poets Society
Dumb Money
Monkey Man
Raging Bull
The Banshees of Insherin
The Sting

Stuff I listened to

Always a mixed bag of artists from all over the world, but people who made a big dent in my Wrapped this year:

fred again…
Seedhe Maut
Hans Zimmer
David Bowie
Karan Aujla
Skrillex

Stuff I wrote/talked about

LinkedIn has quite a plethora of everything I wrote this year, you can find it here.

On Medium, I wrote about the things I’ve worked with in the past and it made sense to write in long form about them:

Using poetry, and never going back to virtualenvs

Spark Listener, and how to setup yours [3AE]

Apache Arrow, making Spark even faster [3AE]

Data Quality + Orchestration — Part II [3AE]

Also, I tried to give back to the community in form of career guidance on Topmate. Got some great reviews from folks who did join.

Stuff I want to do/try doing

Finally, things I want to do or at least try doing in the coming year:

• Get back to learning the keyboard and play at least one popular tune on it
• Dive deep into LLM architectures and learn enough to teach others in whatever way I can
• Build a fun side project in travel/food exploration, which allows me to create something from scratch and see it through, but this time using tools like v0 and Cursor
• Write more for myself
• Run more regularly

In retrospect, quite a lot happened this year that made me think about the choices I make in life, the things I choose not to do, and how they impact me in the long run.

Hopefully, the coming year will bring some new learning experiences.

fin