I would have liked to see a rationale for using virtualenv and all this python-only stuff, when rpm/apt/msi had more mature tools and would have given sysadmins a unified sanely manageable view of what's deployed and their cross-language dependencies.
I am going to guess you are coming at this entirely from the perspective of a sysadmin who doesn't do a lot with python.
virtualenv allows you to easily isolate dependencies so that multiple versions can coexist in different projects without fighting. If you sysadmin significant amounts of Python stuff then you would have already seen that benefit. It also allows you to control PYTHONPATH more elegantly than setting it in every script. This lets you write clean sensible imports rather than using fragile path tricks or wrestling with relative imports.
Your project dependencies are often not exactly the same as what the distribution gives. On one hand, distributions can be mind-bogglingly slow about releasing updates to python libraries. On the other hand, you may not want to force your project to use a new update of something because it may introduce bugs. It is pretty essential to have some lead time to port if your app is anything important. So you need control of the versions of your dependencies.
So if your project dependencies are not exactly the same as what the distribution uses, then you will need some kind of isolation mechanism in order to allow your project to work as a distribution package. You could do that, but you could also save time and just use virtualenv.
Creating platform-specific packages is nontrivial, also it is platform-specific. If I write a library for Python I only want to specify the packaging metadata/scripts once. I am certainly not interested in trying to get it accepted in repos for every distribution.
> Creating platform-specific packages is nontrivial, also it is platform-specific.
Unfortunately, Python-specific packages are Python-specific. I don't want to manage one package manager for Python, one for Ruby, one for Perl and yet another one for system libraries. And I'd prefer to have to trust only one vendor for security updates for everything.
And anything that is not Python depends on system libraries, which are supplied by apt or yum and live in an distro-specific namespace. How does setuptools declare and pull in a dependency on a system library?
I understand the need for Python, Ruby etc. to have their own packaging for cross-platform use, including on non-Unix. But system packaging tools have their uses too. There is no easy answer, and certainly no One True Answer.
I never even implied that system packages do not have their uses, or that python packages are free-standing of libc. That is just a straw man.
In fact, I agree with you for system libraries or tools which happen to use Python and really do not need virtualenv because they are really system-global.
Where I don't agree with you is with use cases you clearly don't understand, where you are developing and/or running multiple python apps in which you NEED isolation or you NEED to manage the versions of your dependencies. Your package manager is not helping with that at all.
And if you don't understand these tools and configuration management tools, then you should not be a sysadmin for projects which involve significant amounts of Python or Ruby.
A system packaging system that was better about sandboxes and running certain programs in certain contexts would be the One True Answer. NixOS (a Linux distro) at least has the requisite features as bullet-points, but I'm yet to successfully get it to install in either of the two times I've tried. (And my level of experience with Linux installations is "no longer need to consult the Gentoo manual to install Gentoo".)
> I am going to guess you are coming at this entirely from the perspective of a sysadmin who doesn't do a lot with python.
I am going to guess you are coming at this form the perspective of a small web developer who hasn't deployed large projects at various customer sites, and then had to support them. Your projects are also just python, not much anything else (C++, Erlang, data files).
For example if you deploy on a well known OS and you deliver a software package to customers and expect to upgrade and maintain them then OS packaging makes sense. Otherwise you are re-inventing the wheel and are shooting yourself in the foot.
I, for one, don't see how you can just dump files one system. Then when it comes to upgrading, just never removing the previous files and simply overwriting files again.
> Creating platform-specific packages is nontrivial, also it is platform-specific.
As much as everyone bashes distutils it does/did? have a reasonably easy way of building rpms -- setup.py bdist_rpm. It has bugs, but it works.
How do you use rpm to provide isolation of dependencies and simple control of PYTHONPATH? This lack of awareness shows that you don't understand the purpose of virtualenv. Nobody is arguing about the utility of packages, but you really don't understand the Python world at all if you think that packaging systems cover the use case of virtualenv.
I wasn't talking about virtualenv only. I was referring more to just using setuptools and pip to manage dependencies and upgrades.
However, system packaging is orthogonal to virtualenv. You can have virtualenv setup packaged inside an RPM (an isolated, self contained package).
> How do you use rpm to provide isolation of dependencies and simple control of PYTHONPATH?
Our product has about 50 or so RPMs. Only a couple of packages needed self-containment. Isolation has its downsides -- you cannot reuse code. So we have managed to avoid it as much as possible.
> but you really don't understand the Python world at all
I understand the "deliver a product" world though. Python is part of the product, not the whole product. I think it is little short-sighted to assume everything is just Python and that is the end of the world, and just run "setup.py install" and you are done. And yes, we have been through the "just run pip" or "just run setup.py install as root" before, and it creates a dependency and upgrade tracking nightmare.
Install a plugin using "setup.py install". Then update the package and remove some modules, run "setup.py install" again.
Now all of the sudden your system is messed up because it picks up old, modules that have not been cleaned by properly removing a previous package when upgrading.
My perspective is a developer on a small team who had to try and support a legacy environment in which no two machines had the same versions of the kernel, native libraries, web server, script engines, and our code, and every box was an irreplacable work of art which we basically had to /bin/dd to migrate. It took us a traumatic year to get from there "yum install our-app" producing a usable app server on a freshly-imaged box, and I don't think we would have succeeded without being able to express all our dependencies regardless of language.
As for mixing different versions of a library, that way lies madness. I regard it as a problem I want the package manager to prevent. An app that isn't on the company-wide version of a library is like radioactive waste—keep it tightly contained on separate hardware and dispose of it as soon as we possibly can. This does make upgrades a Big Deal involving a lot of regression testing, but it's better than troubleshooting code that both is and isn't what's running.
You have a system running "Package A v1.0", which requires "Package X v2.3" and "Package Y v4.5".
You need to install "Package B v2.0", which requires "Package Y v5.0".
Note that Y4.5 and Y5.0 are incompatible. RPM/APT have ways of dealing with some of this sort of thing, but python doesn't necessarily support it. Packagers would have to do things like patch both A and B to do things like "import y45 as y" and "import y50 as y". (I've seen this approach.) Of course, this breaks Packages C and D, neither of which are packaged, that want to use Y4.5 and Y5.0, respectively.
Multiply this by a dozen top-level packages and scores of dependencies with potentially sparring versions and it becomes a huge mess.
Consider that developers are keen on using the latest shiny bauble, and are thus pulling code directly from github/etc. The authors of these packages don't always advertise (because they don't know) strict requirements. They might say "Doesn't work with Y5.0", but they aren't aware that it won't work with Y4.4 and don't mention that.
Working out all of the interdependencies is why the ecosystem has packagers and distributors. These people do the thankless job of sorting this all out. But they can't keep up with the bleeding edge, and sometimes developers have nonstandard requirements, so developers still often resort to pulling from trunk (or local copies) instead of using packages.
Factor in the possibility that the system default python is ancient (2.4, 2.5) and the packages depend on newer versions (2.6, 2.7). (Sadly, there are systems that are running CentOS5 (python 2.4) with no hope of upgrade. I know of an internal production system that is still running Red Hat 9! What's that, py2.2?)
In the end the only reasonable approach is sandboxing via virtualenv.
MSI is terrible. It has older tools, and maybe that's what you mean by mature, but every time I've ever interacted with an MSI build process, I walk away amazed that this is the actual, certified, Microsoft-approved way to package applications for the most popular OS in the world.
Furthermore, it's not really that great of a dependency tracking system, although in theory you can use it for that. Certainly nowhere near APT or RPM.
And on OS X, you've got a whole hodgepodge of competing solutions. Which one of those should the Python runtime rely on to quickly resolve packages and modules at import time?
One reason is that many distros are very slow to update the versions of these packages. It's much easier to get up-to-date packages from language-specific package managers.
