Exciting Developments in Python: Multithreading on the Horizon!
Written on
Chapter 1: The Promise of Multithreading in Python
If you’ve previously abandoned the idea of multithreading in Python, there’s encouraging news on the horizon. Significant changes are underway, and soon you might have the ability to disable the Global Interpreter Lock (GIL)!
Python, an intriguing name for a programming language, could have been named anything from Lizard to Labrador, but it was chosen to reflect a snake. As a result, countless individuals searching for snakes inadvertently learn about this programming language—a clever twist reminiscent of advertising for the Slytherin house.
Interestingly, the name might also serve a protective purpose: if someone were to shout, "Watch out, there’s a Python," the snake would have the upper hand, leaving many puzzled about why a language lacking semicolons and curly braces would be threatening.
However, as we await that hypothetical scenario, let’s delve into the current developments regarding Python’s multithreading capabilities. For those well-versed in Python, this news may not come as a shock, and it’s likely a somewhat imprecise statement. Nevertheless, it’s crucial to understand that the ability to disable the GIL is within reach.
For those unfamiliar, the GIL, or Global Interpreter Lock, is a concept that can be illustrated through an animation showing multiple threads executing code. A closer look reveals that, despite appearances, only one thread runs at a time.
GIL has its advantages, and the prospect of removing it has long been viewed as detrimental to the language's design. Guido van Rossum, Python's creator, expressed concerns back in 2007, warning that eliminating the GIL could complicate matters for extension modules, which rely on a "safe zone" protected by the GIL. This would require handling concurrent calls from multiple threads when global mutable data is involved.
As a result, writing Python code without the GIL would become more challenging. While there were attempts to address this issue in the past, they faced significant hurdles. For instance, during the era of Python 1.5, Greg Stein implemented a comprehensive patch set called the "free threading" patches, which aimed to replace the GIL with finer locking mechanisms. Adam Olsen attempted a similar project later on with his python-safethread initiative. Unfortunately, both experiments led to a noticeable decline in single-thread performance, often by at least 30%, due to the overhead introduced by fine-grained locking.
This performance dip reinforced the notion that introducing multithreading into Python could result in slower execution. Operating on a single thread at a time eliminates the need for additional operations required for thread communication, leading to more efficient execution.
The ongoing debate surrounding threads versus processes has been lively. Although many developers favored the idea of multithreading, Guido maintained a preference for spawning processes to manage greater loads.
Section 1.1: The Historical Context of GIL
In 2007, Juergen Brendel wrote an open letter to Guido, advocating for multithreading capabilities in Python. He argued that developers should have the freedom to make architectural choices suited to their specific problems, rather than being constrained by decisions made by others.
Guido responded, reiterating the challenges associated with GIL removal and expressing his openness to potential solutions, but acknowledging the extensive workload he faced at the time. He emphasized that Python required numerous enhancements, necessitating careful prioritization to help the language evolve into what it is today.
Though the debate between Juergen and Guido was confined to a niche group of Python developers, it sparked ongoing discussions that have persisted since 2007.
Subsection 1.1.1: How Developers Adapted Without Multithreading
Developers have explored various strategies to address Python’s limitations over the years. Despite the common perception of Python as slow, many developers found ways to work around these challenges. High-performance libraries were often written in C/C++, allowing seamless integration into Python and enabling developers to leverage the language’s simplicity while still achieving top-tier performance.
This duality—where Python serves as a user-friendly interface while heavy lifting is performed by C/C++—has long been a point of contention among Python’s critics. Some argue that relying on C++ undermines Python’s integrity, while others appreciate the flexibility it provides.
Chapter 2: A New Era for Python: The GIL Proposal
Recently, a significant breakthrough has emerged: Sam Gross introduced PEP 703, a proposal aimed at making the GIL optional, thereby facilitating multithreading in Python. His presentation is compelling, highlighting the negative impact of GIL on various applications, particularly in AI training, and advocating for broader accessibility to developers who may be hesitant to engage with lower-level languages like C++.
The Steering Council has accepted PEP 703, albeit with a cautious approach: the rollout will be gradual to minimize disruption, and there is a provision for rolling back changes if necessary.
This presentation by Sam Gross outlines the proposal and its implications for the Python community, emphasizing the potential benefits of multithreading.
As a first step, Sam has made it possible to disable the GIL using the command PYTHON_GIL=0 or -X gil=0. This change has been merged and is expected to appear in an official release of Python in the near future.
While this does not mean that multithreading will be available immediately, it marks a pivotal moment. Significant work remains to be done not only in CPython but also in extensions, libraries, and applications.
As developers begin to experiment with GIL being disabled, they can start adjusting their code in preparation for multithreading capabilities.
The second video discusses quick and effective ways to implement multithreading in Python, providing valuable insights for developers eager to explore this new frontier.
Exciting times lie ahead for the Python community, as these advancements promise to broaden the language’s applicability and empower developers worldwide.
Congratulations on reaching the end of this lengthy article! Your enthusiasm for coding is truly appreciated. If you’re passionate about Python, consider connecting with others who share your interests!