Using poetry, and never going back to virtualenvs

In this edition of 3 act engineering, let’s go back in time back in ’07. Its the year when virtualenv was born.

The Setup

Python has had import statements which allow it to do something called dynamic linking, which is an alternative to static linking. Below is a brief of what both of them mean:

Static vs Dynamic Linking: Node, Go, Rust and Python

Credit: PrepInsta

Let’s say you want to import some code which has been written somewhere else by someone else, but is equally useful to you for your current problem at hand. One way to do this would be to copy paste all of it from its current location and paste it alongside yours, calling it static linking. Lot of modern programming languages do it till date, like, Go and Rust.

An alternate to this is dynamic linking, where the libraries are somewhere else on the same machine, but aren’t loaded when the program runs. This gives you an advantage of using the same set of code in multiple programs without ever creating its copies, unless explicitly needed.

Then sometime in ’07, virtualenv was born. The idea was simple, isolate different versions of same package under different umbrellas, and use whichever is needed at that point. Later, if you want to switch to a different version, just a handful of commands are needed and there you go.

Additionally, you can run multiple versions of the same program on the same system all at the same time, without ever facing dependency conflicts since they all practically live in different floors in the same house.

This is how virtualenv (and some other variants of it including: pyenv , venv) made everyone feel that:

Me, every time I have to make a new python project

Let’s first discuss why do you need to move from virtualenv.

>Virtualenv is pretty bad when it comes to automatically resolving dependency conflicts.

In simple words, if you have 2 packages A1 and A2 which require different versions of a package B, then you’ll be invited to a battle which will have plenty of red color logs telling you to either upgrade one, downgrade the other or quit your job.

>Separate dependency management for each of your environments

A lot of manual work. Why? have a separate requirements.txt for each of your repos. The minute you upgrade from one python version to another, the god promises you that you have to do Round 2 to figure out which library suits all the neighbors on that floor.

>No version locks. You rarely need it anyway, but when you do, you do.

Virtualenv has no such mechanism for locking dependencies to specific versions of Python. High chance of inconsistencies across environments.

>Publishing your work? Forget about it

If you want to publish your packages, you’ll need to handle that process separately, which can be more complicated without dedicated support. Virtualenv doesn’t have such a logic built-in out of the box

Alright, alright, alright

The Build

Below would be a speed run on how to setup your first poetry project:

Poetry, the first steps

Using the global python interpreter, install Poetry

python -m pip install poetry

A version check if successful will give you whatever current version you’ve installed for

Let’s create a new project now using Poetry

poetry new my_first_poetry
cd my_first_poetry
tree

This should give you something like:

A few other variants, which give you bit of customization

# use this_is_my_project_folder as project folder
poetry new my_first_poetry --name this_is_my_project_folder

# use src as project folder
poetry new my_first_poetry --src

A new important file created, pyproject.toml will have the following details:

name of the project
the current version
python dependency declaration

Creating virtual environment in Poetry

With the following command, you can see what virtual environments are currently being maintained in this project folder

poetry env list

Since we haven’t created any, as expected, it didn’t show any. But we can fix that

poetry env use python

And voila!, this time it creates a virtual environment specific to our project and also shows the current path where its installed

The installation path is among one of the properties you should worry about. Thankfully, we can always check it out using:

poetry config --list

Installing dependencies in Poetry

The python version I started out with is 3.8, so trying to install some dependencies got me this error

 poetry add pandas numpy requests

As requested by Poetry, a possible solution can be to get a new version of Python (preferably Python 3.10 and above).

Below are some easy steps to switch to a different version

# your python executive path goes here
poetry env use "C:\Python310\python.exe"

Now the interesting part

poetry env list

This gives you 2 virtual environments, and the default one is using Python 3.10

And that solves all my issues from installation, this time the same poetry add command

The last line is interesting,
“Writing lock file”

A tree command will show a new file in the folder: poetry.lock

Its a synonym to ‘going down a rabbit hole’.
It contains the following information about each of the packages we’ve installed:

Description of the package
Required python version for it
Is it optional or not?
For different OS, what exact build/wheel file to use

TLDR: To ensure stability and have the exact same setup in every possible environment, this is what gets the job done.

Cloning an existing poetry project

Imagine you clone a git repo which uses poetry as package manager. Below would be the steps to setup the project:

rm poetry.lock
poetry env remove --all # optional: remove existing python environment for this one if any
poetry install

That should be more than enough to get started.

But that’s not all, Poetry has a plethora of tools/commands for various kinds of scenarios. Some of these scenarios (google if you care about it) are:

— Group dependencies in Poetry
— Optional dependencies in Poetry
— Extra dependencies in Poetry

The Payoff

Alright, final round: Let’s migrate an existing project of yours

Before jumping right in, below is my current project, running fast-api

cd migrate_this
poetry init --no-interaction
pip freeze > requirements.txt
deactivate # get out of any virtual environment if inside one

If you’re on Windows, you can run

@(cat requirements.txt) | %{&poetry add $_}

Otherwise, for everywhere else:

poetry add ${cat requirements.txt}

Finally, let’s do bit of cleanup (optional)

rm -f venv 
rm -f __pycache__
rm requirements.txt

The project now looks like this:

And can be run as:

poetry run uvicorn main:app --reload

THIS IS THE END

I hope this post helped you know more about my favorite dependency manager as of 2024: Poetry.

A couple of good alternatives one can take a look, and maybe I might too:

1. Hatch
2. PDM

Outside work, some recent recommendations I’ve spent time enjoying:

Alien: Romulus, The Substance, Berlin & Longlegs

Patriot, Lastman, Blackbird & Monster

Fred again…, Marcin, Clancy & Beta

Till next time,

fin