Python Packaging and Virtual Environments Landscape

My User Experience Porting Off setup.py Article

It is no secret that packaging in Python is complicated. This post by Gregory runs through the challenges he had moving an existing package away from the setup.py based mechanism. Associated HN discussion.

https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-off-setup.py/

Inline Dependencies for Small Python Scripts Article

Sabs wanted the ability to specify dependencies inside a single Python script, and although there are PEPs on this topic, their implementation is a ways off. Sabs decided to solve this problem directly: introducing pip.wtf a code snippet you copy into your script that does dependency loading through a function call.

https://pip.wtf/

Improving Python Packaging: Why 14 Tools Are 12 Too Many Article

A journey to the world of Python packaging, a visit to the competition, a hopeful look at the future, and highlights from a disappointing discussion.

https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

How Virtual Environments Work Article

This article attempts to demystify virtual environments, why they exist and how they work. It even delves into why Brett is heading down this alley: running into challenges with cross platform tools and microenv.

https://snarky.ca/how-virtual-environments-work/

Steering Council Announces Optional GIL Article

https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

Do Not Use requirements.txt Article

This article discusses why using a requirements.txt file and pip is insufficient for handling dependencies on back end services. Miikka recommends Poetry instead.

https://quanttype.net/posts/2023-10-31-do-not-use-requirements.txt.html

PEP 711: Standard Format for Distributing Python Binaries Article

This PEP proposes a way of packaging pre-build interpreters “like wheels, but for python interpreters”. The intent is to re-use existing packaging standards as much as possible.

https://peps.python.org/pep-0711/

An Unbiased Evaluation of Environment and Packaging Tools Article

This detailed article covers the wide world of packaging in Python, how the different tools overlap, and how each has its own area of specialization. A great deep dive on all the choices out there that can help you pick the right tool for your project.

https://alpopkes.com/posts/python/packaging_tools/

Quirks of Python Package Versioning Article

Python packaging allows for a wide variety of version styles for your packages. This article shows you what is out there and why you might use each.

https://sethmlarson.dev/pep-440

Don’t Say “Simply” Use Pyenv, Poetry or Anaconda Article

This article talks about the issues often introduced to newer Python coders by adapting more complicated package management mechanisms, and why sticking with pip is often the better choice.

https://www.bitecode.dev/p/why-not-tell-people-to-simply-use

Use “pip install” Safely Article

The correct extra arguments to pip install can make it less risky to execute. Learn what arguments James suggests and how to use each of them.

https://www.b-list.org/weblog/2023/dec/07/pip-install-safely/

PEP 703 (Making the GIL Optional in CPython) Acceptance Article

https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075

PEP 702: Marking Deprecations Using the Type System Article

Python 3.13 will include a decorator to indicate that content has been deprecated. This PEP outlines the specification and use case.

https://peps.python.org/pep-0702/

Classifying Python Virtual Environment Workflows Article

This article discusses the various ways of creating and managing Python virtual environments, including what kinds of tools are used. It categorizes the different styles and describes the impacts of the choices you make on your workflow.

https://snarky.ca/classifying-python-virtual-environment-workflows/

briefcase: Convert Python to a Native Application Project

Tools to support converting a Python project into a standalone native application.

https://github.com/beeware/briefcase

Packaging With Native Binaries Article

The pypackaging-native site is a collection of content about key Python packaging topics and issues for projects using native code, with a focus on scientific, data science, and ML/AI projects in the PyData ecosystem.

https://pypackaging-native.github.io/

PEP 722 and PEP 723 User Study Discussion Article

User studies were conducted on on PEP 722 – Dependency specification for single-file scripts and PEP 723 – Embedding pyproject.toml in single-file scripts , this discussion summarizes the results.

https://discuss.python.org/t/pep-722-and-pep-723-user-study-discussion/34871

The Basics of Python Packaging in Early 2023 Article

This post is a good guide to the latest in Python packaging, including the minimal configuration you need to adopt the most recent packaging standards.

https://drivendata.co/blog/python-packaging-2023

PEP 704: Require Virtual Environments by Default Article

https://peps.python.org/pep-0704/

An Open Source Python Project CI Pipeline Article

This posting describes Brenton’s Python Continuous Integration (CI) pipeline setup, from using a pyproject.toml file, to the tools such as black and coverage he uses, to GitHub actions to glue it all together.

https://brntn.me/blog/open-source-python-ci/

Thoughts on the Python Packaging Ecosystem Article

There’s been lots of chatter about packaging in Python of late, this discussion centers around Pradyun Gedam’s excellent article. For more on the same topic see last issue’s How to improve Python packaging, or why fourteen tools are at least twelve too many as well.

https://news.ycombinator.com/item?id=34467952

A Not So Unfortunate Sharp Edge in Pipenv Article

Pipenv keeps hashes of installed projects to help protect from supply chain attacks. Something interesting happened though when a project didn’t change version numbers but did change hashes, learn just what the edge case was and how it complicated build repeatability.

https://0xda.de/blog/2022/12/a-not-so-unfortunate-sharp-edge-in-pipenv/

One Does Not Simply pip install Article

Find your favorite package and turn to the readme to get it installed - it seems dead simple just a ‘pip install’ away. Nothing could possibly go wrong. Right? If you’re used to it, it is easy to forget almost all the instructions are skipping a step: using a virtual environment.

https://www.ianwootten.co.uk/2023/02/17/one-does-not-simply-pip-install/

How to Evaluate the Quality of Python Packages Article

Just like you shouldn’t download any file from the Internet, you shouldn’t blindly install third-party Python packages without evaluating them first. This tutorial will give you the toolset to evaluate the quality of external Python packages before you implement them into your Python projects.

https://realpython.com/python-package-quality/

Virtual Env Structure & Surveying the Packaging Ecosystem Article

How do Python virtual environments work under the hood? How does understanding these concepts help you with managing them for your projects? This week on the show, CPython core developer Brett Cannon returns to discuss his recent articles about virtual environments and the Python packaging landscape.

https://realpython.com/podcasts/rpp/156/

Building Reproducible Python Environments With XARs Article

XAR is an archiving format that can contain a tree of files. This article details how they can be used to package Python environments for deploy-ability.

https://blog.janestreet.com/building-reproducible-python-environments-with-xars/

PEP 703 Optional GIL: Timelines and Problems Article

https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/43

Sharing Your Python App Across Platforms With BeeWare Article

Are you interested in deploying your Python project everywhere? This week on the show, Russell Keith-Magee, founder and maintainer of the BeeWare project, returns. Russell shares recent updates to Briefcase, a tool that converts a Python application into native installers on macOS, Windows, Linux, and mobile devices.

https://realpython.com/podcasts/rpp/148/

Pip 23.1 Released: Massive Improvement to Backtracking Article

https://old.reddit.com/r/Python/comments/12n5lai/pip_231_released_massive_improvement_to/

PEP 707: A Simplified Signature for __exit__ and __aexit__ Article

https://discuss.python.org/t/pep-707-a-simplified-signature-for-exit-and-aexit/24402

As if There Weren’t Enough Packaging Tools: Rye Article

https://old.reddit.com/r/Python/comments/12w2ws7/as_if_there_werent_enough_packaging_tools_already/

python-build-standalone: Redistributable Builds of Python Project

Produce redistributable builds of Python

https://github.com/indygreg/python-build-standalone

Dependency Issues Checker Project

Find unused, missing and transitive dependencies in a Python project.

https://github.com/fpgmaas/deptry

creosote: Identify Unused Dependencies Project

Identify unused dependencies and avoid a bloated virtual environment.

https://github.com/fredrikaverpil/creosote

microvenv: Minimal Re-Implementation of Python’s venv Project Started in 2023

A minimal re-implementation of Python's `venv` module

https://github.com/brettcannon/microvenv

Finds All 3rd-Party Imports in Your Python Project Project Started in 2023

A Python CLI tool (Written in Rust) that finds all third-party packages imported into your Python project

https://github.com/saadmk11/python-third-party-imports

dpv: Alternative to pyenv-virtualenv and virtualenvwrapper Project

dpv is an alternative to pyenv-virtualenv and virtualenvwrapper

https://github.com/caioariede/dpv

invenv: Python in Automatically Created Virtual Environments Project Started in 2023

A tool to automatically create and run your Python scripts in a virtual environment with installed dependencies

https://github.com/jsnjack/invenv