Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?
Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.
But you still do have to jump through some hoops then. But like the example in the docs show there are shareable primitives, you can share numpy arrays, etc.
Multiprocessing is pretty heavyweight, everything needs to be serialized, you can't share Python objects directly or with good performance across processes, there is (fairly new) shared memory support, but it's just a bag of bytes. It's not particularly good on the cross-platform angle, while it does work on Windows, the startup time for each process is like starting a fresh interpreter, because that's exactly what it is doing.
Multiprocessing can work if you have all your logic in Python already, and there is no notable shared state between the processes, and you don't need to coordinate much at all. So pretty much just the very simplest problems in concurrency. Everything else might technically be possible, but will likely be very slow and annoying.
I am probably the most commercially successful user of the multiprocessing module :)
It's basically fine anywhere you need a function call that you can dispatch out to like 50 or 500 workers on a queue and then do something after that returns, but any shared memory or IPC between the workers is up to you.
Python is also fine for webserving because most web servers pre-fork workers or whatever, so this doesn't come into play there either.
It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java.
Firing up multiprocessing (forking) has some costs to bringing up the interpreters so it's not something you want to start up a lot and then close down a lot, better if you can start things and leave them running. Once it's up it is pretty fast.
I guess mainly it changes the style of your program too much - it's basically just glue around forks.
> It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java.
The performance overhead of multiprocessing is pretty high, to the point that you really have to take care to ensure that whatever problem you're solving with multiprocessing is large and parallel enough to justify it, otherwise your multiprocessing solution may very well be slower than your single-process one.
That's not to say it's useless, it's a nice tool to have in the standard library, but if you're coming from one of the many mainstream programming languages with lightweight threading libraries which make it much easier to take advantage of parallelism it's easy to get frustrated at both python's standard threading and multiprocessing libraries for their respective shortcomings, at least in their CPython implementations.
On Windows multiprocessing is very poor due to lack of forking. This means for example no memory at all is shared; modules, data, external shared libraries etc are all individually loaded by each subprocess. There are also some Windows-specific oddities that make using it a further pain (can’t remember the details).
There is a project to effectively internalise multiprocessing by running separate interpreters inside one process. It’s meant to be cheaper than having separate modules too.
Sharing data between processes is _very_ expensive, so whole classes of programs are not suited for this approach.
For example I had a program that preloads large immutable dataset. and a bunch of threads use it. I couldn't do it efficiently in Python (at least not CPython).
Outside of Windows, I believe, you can do it so that the dataset is read-only shared (or copy-on-write) via fork, and then it’s as quick as local access.
Loading data before you fork works pretty well, but the overall effectiveness heavily depends on the type of data involved. If it's something like numpy arrays or similar large, indivisible objects, you're golden. If you want to preload and share something like a huge nested Python dictionary or other large collections of small objects, you immediately collide with the reference counters.
Basically, since the reference count is kept right before the object data, as soon as the child process touches it - even just to look at it! - you immediately trigger a copy on write on the nearest 4k of memory, which tends to add up fast if you're not careful. Even if you never touch 99% of them, the garbage collector is happy to do it for you.
At my previous job it was bad enough that I ended up writing a small patch to be able to set some refcounts to 0xFF...FF and treat them specially, never changing their value. (Yes, this also meant that they never got destroyed properly, and the extra checks made our codebase around 4% slower, but it was an acceptable tradeoff. No, the patch was no longer small by the time it hit production.)
As a sibling mentions, the multiprocessing support is basically support for spawning new processes. It's better than nothing, but it's not very good for the performance.
The real problem with Python multicore is that the main problem is solves is the one ctur is talking about, namely, "Oh crap, I used Python and it doesn't run fast enough for my needs... maybe I can run more processes?" Using a language that is already in the slowest class of languages and realistically 40-50x slower than other languages means that just to recover the performance you'd get from switching to a compiled language, you need ~50 perfectly parallel processes solving an embarrassingly parallel problem. If you can't do that... and that's a fairly common case, things that are a full, true 50x embarrassingly parallel aren't actually that common on real hardware because you'll get some sort of contention... then you can't even work your way up to the performance that you could have gotten by starting with Go or C# or some other reasonable language.
And that's ignoring the serialization overhead between the processes. If that starts costing you noticeable amounts the number of processes you need to recover goes up really fast, due to the way the math works.
Note the only thing about Python causing this effect is its performance; it is equally true of anything else that is the noticeably slower performance league. So, also, note that if you are using Python in one of the ways where it doesn't have this performance problem, such as NumPy with almost all your compute in native code, this analysis doesn't apply.
In the 1990s when Python was born, programming in Python was massively easier than programming in the static languages of the day. The landscape has shifted... Python is now only incrementally better than modern static languages in some dimensions, and I tend to agree with ctur, it just plain isn't better once you pass a certain size and the stereotypical problems with dynamically-typed code start hitting you harder and harder. There are now a lot of good statically-typed languages where you can start a new project, writing something just incrementally harder to write than Python, with the type inference, built-in associative types, tons more libraries, etc. all the advances of the past 25 years, and get static-typed performance. The gulf between "easy Python" and "pull your hair out static language" is not quite closed. Not quite. But it's a lot closer than it used to be; less "Grand Canyon" as it was in the 1990s and more "can I jump that creek? I mean, it's pretty close... I think I can jump it...".
The upshot is, if you're reaching for multiprocessing for performance reasons, not convenience reasons, you've very nearly already lost. There's a narrow window where it might still make a bit of sense, but you're getting perilously close to "You need to switch languages" just by reaching for it at all.
I agree with this, but when it comes to scaling, making things faster at the language level is only one option. And while, as you suggest, the Python/static_lang gulf has narrowed, I would guess the gulf between "what needs to be done in the language itself" vs. "what can be done by some external system/database/process" has also narrowed.
What I'm getting at is, let's say you have some system where you need highly performant real time processing of some data. Perhaps 10 years ago, you might have needed a complex multi-threaded java/c++ app to handle it. But perhaps now, you use, say, DynamoDB, kinesis, kafka, lambdas, or some other collection of services that you're basically gluing together with... python.
I'm not saying you're wrong. I'm just wondering, if you could truly factor in the costs in both developer time -- developing multi-threaded apps is usually pretty tricky after all -- and infrastructure costs and whatnot, where the boundary between "python is adequate" vs. "we really need go/java/c++" lies, and in which direction it's moving.
In retrospect, I guess it's sort of a silly question, since at the core, the services we'd be gluing together are no doubt written in faster languages. Perhaps a better way of framing it what % of problems can be adequately solved (cheaply) by just using language X, and how has that changed over time?
"I'm just wondering, if you could truly factor in the costs in both developer time -- developing multi-threaded apps is usually pretty tricky after all -- and infrastructure costs and whatnot, where the boundary between "python is adequate" vs. "we really need go/java/c++" lies, and in which direction it's moving."
"It depends.", of course. Despite the significant slow down of single-core performance improvements, 1 CPU is a lot of power in a lot of use cases, even if you divide it by 50. I do frequently find myself reminding some people who get a little too deeply into the cloud mindset and the believe that any non-trivial problem needs clusters of systems that you can still do an awful lot with one CPU. And I often wonder how many "clusters" are out there drinking down the watts doing work that if somebody would just spend a week or two optimizing their code to get the O(n^2.5) algorithm out of their system could be comfortably done on a mid-grade laptop... if somebody just realized you shouldn't need a "cluster" to do this task.
However, flipping your perspective around is probably more interesting... multi-threading is still not "easy", per se, but it is also way easier than it used to be. Threading hell is true and exists, but a non-trivial amount of it was down to the architecture being attempted. It's a bad idea to try to coordinate everything with piles of mutexes everywhere. If you program with more agent-like approaches to resource management, even if you don't do it 100% like Erlang forces you, multithreading gets much easier. What kind of things does it open up to for it to be much easier than it used to be?
I still use Python for certain tasks where it is suited. But I'd have a hard time going back to it for most of my programming. I've internalized the ability to say "and I need a server here with its own thread of control managing this resource, and I need to set up a worker pool there for this data processing task, and I can set up a recurring, independent process to check this other thing periodically without it interfering with anything else" whenever it is necessary to ever be able to go back to architecting systems without that capability. I'm not going back to cooperative scheduling unless basically forced. There's just so many places where you ought to have this capability available to you and it's actually easier to work multithreaded rather than do the work to try to thread a single execution context through all the things that shouldn't be that tightly coupled together.
It is generally underappreciated that having to have two bits of code share an execution context is a deep level of coupling, and languages like Python force all your code into one execution context.
Having multithreading easily available has its downsides, yes, but it also has its legitimate upsides for architecture.
Really I think this is only true for the data science ecosystem. In most other cases, modern static language ecosystems are at least as advanced as Python.
One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way.
For example MacOS cannot use fork while it was (is?) the default on Linux.
