No, it didn't. Yes, k8s has 'won' in large-scale deployments, but if you're working at a small shop, then just imitating what Google does with millions of servers is dumb. Do what works at your scale -- and Swarm is extremely easy to manage.
Many people ask why would someone use an orchestrator on a small cluster (dozens of hosts). Why not? Swarm is very easy to manage and maintain, using Puppet or Ansible is not less complicated at all.
The future of Docker, Inc. is of course Docker EE, and the future of Docker EE is _not_ the scheduler, it's everything around it.
> Swarm is very easy to manage and maintain, using Puppet or Ansible is not less complicated
The idea that dockerized software somehow is less dependent on configuration management seems to be a popular and completely misguided one. The two trends are completely separate, but I would argue from experience that unless you have absolutely nailed the configuration and integration of all your moving parts, don't even look at containers yet.
Containers tend to lead to more moving parts, not less. And unless you know how to configure them, and perhaps even more importantly how to test them, that will only make matters worse.
If you design your infrastructure and choose your tooling well, then containerized (not "dockerized") software is far less dependent upon configuration management; indeed, using Chef/Puppet/etc can be completely unnecessary for the containerized workload. To be clear, however, there is absolutely still a need for the now-traditional configuration management layer at the level of the hosts running your containerized workloads. What's kind of exciting about this is that the giant spaghetti madness that our configuration management repo has become—and I'm pretty sure it's not just us ;-)— at our org is going to be reduced in complexity and LOC by probably an order of magnitude as we transition to Kubernetes-based infrastructure.
> indeed, using Chef/Puppet/etc can be completely unnecessary for the containerized workload
This is more than naive. As long as your software needs any kind of configuration, there is a need for configuration management. There will be access tokens, certificates, backend configuration, partner integration of various kinds, and monitoring and backup configuration and you will want guarantees that these are consistent for your various testing and staging environments. You will want to track and bisect changes. You can either roll your own framework for this or use Ansible/Puppet.
Whether you distribute your software pieces with tar balls, linux packages or docker images or completely orthogonal to how you tie these pieces to a working whole. And the need for configuration management absolutely increases when moving towards containerized solutions, not by the change in software packaging format but by the organizational changes most go through where more people are empowered to deploy more software which can only increase integration across your environment.
I see organizations that have ignored this because they believe this magic container dust will alleviate the need of keeping a tight grip over what they run, and find themselves with this logic spread over their whole toolchain instead. That's when they need help cleaning up the mess.
I never said anything about magic container dust, nor did I say anything about having less of a grip over our operations. I was attempting to make a point about how your workloads themselves (applications/jobs) can be free of a direct need for Chef/Puppet/etc, which can dramatically simplify your configuration management layer. I never intended to claim that somehow magically our pods need no configuration bits at all, and honestly I’m not sure where you got that idea.
The statement was that containerized workloads are less dependent on configuration management. That could easily be interpreted as if configuration management gets less important when you containerize, which is an idea that seems to spread easily on its own, while I have found the complete opposite to be true. That's why number one guideline is to get a grip on your infrastructure and configuration before you move to containers. Otherwise you will end up with a mess worse than before.
No, it's not 'less dependent'. The way how it helps is that you can standardize configuration management across environments relatively easily.
Containerization does not result in less moving parts in a given environment; but it does result in less moving parts across the whole development flow (from a developer laptop to a production cluster).
Thanks to CoreOS containerLinux, you don’t really need Puppet or Ansible. You specify your Ignition config, and just drop it as a tiny partition on the servers, and drop the OS onto a separate, adjecent partition, and then have local storage even separately. Commonly you use PXE to avoid having these locally.
Then everything you do on top of that is handled by the container scheduler, and your containers.
> Commonly you use PXE to avoid having these locally.
I’ve not seen PXE used anywhere that the DC wasn’t O&O (or essentially close to it). As that’s the exception to the rule these days, isn’t your premise a bit cavalier?
I’ve used PXE a lot in my past [0] to great benefit (well, more specifically iPXE through chainloading), so I’m not detracting from it, just saying it’s applicability is limited for most folks.
[0] I wrote “Genesis” for Tumblr which handled initial machine provisioning from first power-on after racking to being ready for deploys.
> Using PXE (or, rather, iPXE) is the recommended deploy mode for container linux
Really? Is that a new thing? I don’t remember that last time I was reading the docs, but maybe I missed it (that it’s the “preferred” method rather than being just “an option”).
Seems rather odd to limit your audience like that in the age of “cloud everything” as I think it’s generally more rare that folks fully control their layer 2, but what do I know. :)
Well, you can always still use the disk install, and instead mount the ignition config as elastic block storage read-only. That's what most people do in the cloud.
But in either case, you want the config to be centrally stored, so you can modify it without having to ssh onto every machine.
Network booting combined with good out of band management controls allows fully automated provisioning of nodes in a baremetal setup, so I don’t see how this is a negative? The alternative is a live cd/usb or attaching a disk with something already preinstalled.
If you’re on prem with VMware or openstack there’s better options which you can use, but that’s not exactly baremetal. in those environments CoreOS recommends using different provisioning options more suited for those providers.
> people ask why would someone use an orchestrator on a small cluster (dozens of hosts)
I’d love to know who these mythical folks are?
1) dozens of hosts (heck, hosts >1) is exactly why you need orchestration
2) while there are huge deployments across the globe, I wouldn’t consider “dozens of hosts” small by no means. That’s actually probably above average.
3) k8s is actually easier to maintain than you allude. I see these comments about Swarm over k8s generally from folks who never even tried it (or did so years ago), is that the case here?
Disagree. Do not find using K8s any more complicated and believe it being more complicated is a myth that just gets repeated over and over to be believed it is true and not based on actual experience.
It just sounds accurate to say it is like trying to be like Google. Well if it is but less complicated then there is no down side.
I can't believe I'm saying this, but if some giant corporation is going to buy Docker, Inc. then I really, really hope it's Microsoft. An Oracle acquisition would be an absolute nightmare.
That's the thing, we use k8s in production and I just used docker-compose for testing in my device environment for the sake of simplicity. Though over time it became not so simple to maintain two parallel config as the system grown up.
This is very interesting. I've been looking for middle ground between Dokku and Kubernetes, I'll give swarm a try. Does it handle ingress at all? vhosts, TLS certificates, etc?
Docker has added ingress routing in 17.06 that works at a lower level than HTTP Routing Mesh. You should try that now.
Other than that, how have you found EE ? We have considered paying for it, but after seeing Docker Cloud at 15$ per node per month, we are considering switching to it.
I'm using https://github.com/containous/traefik with Swarm for termination, Let's Encrypt and routing with success. Configuration is handled with service labels.
Thanks for the shoutout. Would you happen to have a writeup on how to do that, or a few lines on whether it's Swarm-aware somehow? I guess it can't be too hard to figure out, but if there's something that will speed that up, that's better.
All you need to do is create a docker service out if traefik/nginx/haproxy/whatever and bind the ports of the service to type "ingress". From then on, all traffic is internal - and you deal with it normally from one docker service to another.
Ah, okay, so pretty agnostic. Sounds like this is geared towards single project deployments, though (ie it wouldn't be suited in a multitenant scenario), but that's good to know.
I've been looking for a Dokku replacement for running my side-projects on with easy scalability (by provisioning another server), but I haven't found anything that's suitable for running multiple apps together in a heroku-like way but still able to scale beyond a single server. Do you know of anything like that?
but that is what swarm is - swarm is fully production ready to scale to tons of servers and works brilliantly on a single server as well.
Not sure why you think swarm wouldnt fit the bill ?
From what I saw, it's not very easy to have a deployment with multiple applications on the swarm, is that wrong?
For example, I have ten applications, and each requires a database, a redis instance, a celery instance and two web workers. Dokku lets me deploy these independently of each other, but uses the same nginx instance and proxies them transparently.
As I understand it, Swarm has no notion of multiple projects. Each swarm is running a single deployment, where all containers are equal, is that correct?
Basically, Dokku is a self-hosted Heroku, which is what I need (I want to be able to easily create a project that I can run semi-independently of the others on the same server). My understanding is that, to do that with Swarm, I'd have to have a single termination container that would connect to every app, but apps wouldn't be any more segregated than that. Maybe I'm complicating things, though. Have you used Swarm for such a use case?
I tried the official tutorial, but couldn't get it to work, as the instructions appeared outdated and didn't work for single-host deployments, and were geared more towards Windows and Mac than Linux. Would you happen to have a good "getting started" document? All my apps are already using docker-compose.
EDIT: Also, a machine that's a manager doesn't want to join the swarm as a worker as well, that's why I'm saying that it doesn't appear good for single-server deployments:
> Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
the docker stack deploy command does automatic diffing of which services have changed. So your deploys are automatically optimized. This is generally the philosophy of Swarm vs kubernetes - everything is built in. You can argue this is less powerful, but in general it works brilliantly.
In so far as separating out the different "applications", you simply put them on a separate overlay network (encrypted if you want).
Also, if you are dead-set on making them entirely separate, every separate "application" is a separate "Stack". So you can stack deploy them separately.
If I had to do what you just told me - single nginx proxying to two different "applications" - i would do this.
1. stack 1 - application 1 + network 1
2. stack 2 - application 2 + network 1
3. stack 3 - nginx + network 1
now you can deploy any of them independently. You can make this even more sophisticated by having each stack on a different overlay network (encrypted as well). And nginx bridging between them.
Not sure why you are facing problem with the official tutorial - btw, a manager is a worker ;) I have a fairly large dev swarms on a single node.
Swarm is nice as-is but is, but really shines with support like docker-swarm-proxy. Please check out http://proxy.dockerflow.com/. It allows for automatic service discovery as well as routing/termination etc. It also runs as a stack on swarm itself. Supports custom TLS termination as well as SNI routing.
We found the same thing after the package management stuff caused issues in prod.
We rolled our own go program that just does go tpl substitution in yaml with overrides like helm. Works on charts out of the box, but instead of talking to a service, it outputs a yaml manifest ready to kubectl apply.
We've thought about open sourcing it. It took us literally a day to put together and had worked without flaw for 8 months in production.
Cool! I just don't use the package management stuff. My workflow also has local manifest template rendering, but I use helm-template[0]. My chart is just a local directory, not a package or anything. Then, to release:
* Render the chart and run it through kube-cloud-build[1] to make sure my containers are all built before I deploy: `helm template /path/to/chart | kube-cloud-build -r repo`
* Deploy with helm: `helm upgrade RELEASE_NAME /path/to/chart`
Anyways, I envision this being very useful for development, may even replace my docker-compose based test setup.