> No Python Binary Distributions: CPython builds from python.org are completely inadequate. On some platforms you only get an .msi installer, on some you literally only get tarballs. The various Python distributions that became popular over the years are diverging greatly and cause all kinds of nonsense downstream. This is why this Project uses the indygreg standalone builds. I hope that with time someone will start distributing well maintained and reliable Python builds to replace the mess we are dealing with today.
And here is info about those particular indygreg builds.
I've never really experienced the problem they are describing. Any official Python build I've gotten from python.org has worked. Every normal old Python I've gotten from my distribution's package manager has worked. Every Python included with an official Docker container image has worked.
I'm sure their special builds will also work just fine, but their non-standardness gives me pause. There's even a list of behavior quirks. Why are we adding quirks to something that has no problems? And the fact that the rye philosophy seems so intent on using them, turns me off from it compared to the alternatives that sensibly default to using the Python you already have available on your OS.
I'm just guessing, but I imagine the scenario goes like this:
1. Work at a company that runs some ancient version of CentOS or something.
2. The system package manager doesn't have anything newer than Python 3.6.
3. Official binaries from python.org print an inscrutable error about the glibc version when you try to run them.
4. Building Python from source requires some new system dependencies, takes forever, and then produces an even less scrutable error about a missing header file.
5. Googling that error leads to a ./configure flag that works around the issue, which upstream GCC fixed in 2017.
6. Success?
If you haven't seen that error #3 before, or dealt with "manylinux" build environments, you've dodged an entire world of pain. The same goes for scripting installs on Windows, and for the part of that page that's talking about "limiting the CPU instructions that can be used" :')
Which is kinda, to some degree, on the pytorch people as well.
I would think that if you add a new way of packaging projects and managing dependecies, there will always be legacy projects that require you to manually put some files to an undisclosed location with a wheelbarrow.
I don't think any new build system is to blame if an existing projects installation process is user-hostile. It would certainly be better to make the pytorch installation more straightforward instead of carving out special niches in all build tools for it.
I have been in the #3 hell, almost exactly how you described but it was always about SSL and its missing headers. On my desktop wiki, the most important section about Python is the one that contains the incantations required to compile SSL, setting myriad of variables and using pyenv to build a newer (3.10/3.11/3.12) Python.
I've run into #3 quite often in embedded Linux projects, especially when dealing with the Jetson ecosystem where upgrading to a modern Python is a nightmare due to all the specialized hardware. Glad to see I'm not the only one who runs into this.
Python building, packaging and deployment has two extreme states: the king's highway and the hall of a thousand knives. If the portable Python suggestions do not make sense to you, then consider yourself lucky, because you have managed to stick to the highway.
I regularly download the python source code, compile it with standard prod optimizations, then install to /use/local/python${version}. This works extremely consistently since python 3.7 (released in 2018). In my experience, these commands are so stable and consistent they could be automated away. What might the author's issue or underlying protest be?
I've also compiled python from source a good amount, and it usually works... until some thing where I realize some standard lib wasn't compiled because I was missing an optional dependency. But some lib assumes that it was always included cuz the standard distro is.
I think it's easy to compile Python, but it's easy to end up just having to go re-compile it for some random component that was not compiled in the "standard" fashion.
If you have a good test suite this stuff shows up quite loudly though. At one point the core issue is more collaborators wanting to not have to compile things themselves.
(And to "automating away" as a comment... indygreg's releases _are_ this! Someone has done the work for us all)
>And to "automating away" as a comment... indygreg's releases _are_ this!
They most definitely are not. There's a world of difference between downloading a portable Python build and building one on your own machine, and C extensions can give you a world of trouble when they start referencing paths that are not on your machine, but rather were on the CI machine that built your static build. The FAQ even has a big section that boils down to "There's a bunch of paths included in these builds that will not make sense to consumers and we don't have a way to fix it."
Same. We build our own Python and have been running it for years without a single hiccup. Not sure what the big fuss is. Pyenv does the same thing.
The concern could be absolved by simply improving the docs with the most recommended compile flags. I think they are actually noted there. Also of note our build time decreased substantially with llvm.
There is also a world of windows out there. Compiling Python from scratch is not a ton of fun there. Rye wants to have a very consistent experience for everybody.
Python is a nightmare to use on those platforms as well. It's one of my favourite languages, but I have (until now I guess) started avoiding it like the plague for anything important or distributable because of the actual hell that is dealing with it in production.
Well an obvious issue is that you have to do that!
Also I think a big issue is the inconsistency between platforms. For example the official Python installed doesn't include python3.exe (frankly because the devs are idiots), but the one from the Microsoft app store does!
If you stay on one platform you wouldn't see those issues.
> No Python Binary Distributions: CPython builds from python.org are completely inadequate. On some platforms you only get an .msi installer, on some you literally only get tarballs.
I'm just guessing, but they could mean that there are no macOS/Windows binaries for security releases of older Python versions. You can't, for example, download an installer for Python 3.10.14. The last Windows installer is for Python 3.10.11 (April 5, 2023).
I tried Rye during its first days. It would (without any indication) download some custom build of Python, which was dynamically linked so won't work unless you're running a distribution similar to the build environment.
Linux distributions ARE NOT binary compatible, you can't just build Python on one distro and run in on another. You need to statically link everything to do that.
The way different Pythons from PyPI don't work can be, for example that various optional modules are (not) included. For example, for my own Python installs, I build without tkinter. I have no use for this module, and it's always a burden to find the outdated XWidget libraries necessary for this pile of garbage to build.
Seeing how this module is basically useless, a lot of Linux distros also exclude it from the Python they package. But PyPI builds try to include everything.
There are few more modules like that.
Another aspect is various defaults that help Python locate its parts or control its loading. Eg. the site module or sysconf. For various reasons various vendors may configure these differently. This may result in some packages being broken upon install.
I.e. Python programmers are notoriously bad at creating packages and understanding how they work (also, Wheel format is a dumpster fire of nonsense). So, a lot of developers don't understand the consequences of packaging anything that's not strictly Python source code (which, frankly, should never have been packaged! but hey, remember, Wheel? dumpster fire? So... where was I...) anyways, native libraries packaged with python source may end up in some random place Python doesn't look at, and consequently fails to load, or other data files end up in the wrong place because the developer packaging them after countless trial and error has made it work on their computer, with their particular Python installation, but they still don't know why it worked.
Similarly, if a package wants to provide header files so that other packages can compile against the native modules the package provides... oh-ho-ho, bet you didn't know that's possible, right?! Well, don't bother. It mostly doesn't work anyways. And there's a bunch more stuff like that.
As a "typical" user, you might have never encountered any of the issues above, but me, as a Python infra person who's summoned to extinguish fires started by talented Python programmers by using tools like the one in OP deals with this stuff pretty regularly. So, I can relate to the disappointment with any aspect of Python infrastructure. There has never been a case of me discovering something in Python infra and looking at it with admiration. At best it's passable, but in most instances it's hilariously bad.
Original author of Rye here: there are no official Python builds outside of macOS and Windows and the macOS builds cannot be installed programmatically. They also cannot be placed in any location other than the dedicated framework location which often causes issues for people who do not expect specific versions to be installed. Quite often installing the macOS builds of Python breaks other tools that expect a different version to be there.
I’m glad regular Python versions work for you, and you can register them with Rye. That’s very intentionally supported.
The goal of rye is to reach higher. It wants to see how close we can get to an experience you can get from node or Rust. That includes getting access to Python itself.
I have been programming Python since 2.2 and have helped countless of people over the years with their Python challenges. From mentoring to tutoring to just helping on IRC and other places. So many times people had a completely broken Python installation or ran into compilation errors. Even pyenv ships patches to make Python compile cleanly in some cases.
The indygreg builds have issues, no doubt about it. In an ideal world the PSF would distribute portable Python builds for all platforms.
As someone with a large project that depends on the standard readline that was a major hiccup when moving to rye. Luckily there's a gnureadline package.
From their philosophy page: https://rye.astral.sh/philosophy/
> No Python Binary Distributions: CPython builds from python.org are completely inadequate. On some platforms you only get an .msi installer, on some you literally only get tarballs. The various Python distributions that became popular over the years are diverging greatly and cause all kinds of nonsense downstream. This is why this Project uses the indygreg standalone builds. I hope that with time someone will start distributing well maintained and reliable Python builds to replace the mess we are dealing with today.
And here is info about those particular indygreg builds.
https://gregoryszorc.com/docs/python-build-standalone/main/
It is, however, possible to choose a different Python.
https://rye.astral.sh/guide/toolchains/
I've never really experienced the problem they are describing. Any official Python build I've gotten from python.org has worked. Every normal old Python I've gotten from my distribution's package manager has worked. Every Python included with an official Docker container image has worked.
I'm sure their special builds will also work just fine, but their non-standardness gives me pause. There's even a list of behavior quirks. Why are we adding quirks to something that has no problems? And the fact that the rye philosophy seems so intent on using them, turns me off from it compared to the alternatives that sensibly default to using the Python you already have available on your OS.