Docker Swarm over Kubernetes

A little while back someone asked a question on Reddit:

Why docker swarm is not popular as Kubernetes?

Having a few thoughts to share, i responded with my personal experience. However, i thought that it would actually make a decent, if a bit short blog post as well. I've used both Docker Swarm and Kubernetes in the past, actually utilizing both as parts of my Master's degree thesis and pitting them one against the other in some benchmarks, which revealed a few interesting things about their performance (i'll write an article about the results of benchmarking them some day).

Yet, in many cases, both in regards to my day job and also my personal side projects and my homelab, i often find myself still reaching for Docker Swarm over Kubernetes on a regular basis:

But why is that? In a word: simplicity. Docker Swarm does everything that i need it to, and in many cases it's actually better than Kubernetes. Not because Kubernetes can't run deployments of thousands of containers across hundreds of nodes, but because i don't need a tool that does that. What i need is a more lightweight tool for running a few hundred containers across a few dozen nodes. For example, here's one of my Swarm clusters, which i run in a hybrid cloud for personal projects:

And i'd like to argue that such configurations will also match what most organizations will run and what they will actually need, instead of just chasing hype. Allow me to illustrate some of the ways in which Docker Swarm is better than Kubernetes in 2021.

Docker Swarm should be more popular

In many ways, Docker Swarm is superior to Kubernetes:

• all of the versions of the Compose specification are far simpler and easier to use when compared to Kubernetes manifests; there even isn't a single page that gives examples for making Kubernetes manifests like the link for Compose (the API docs are the closest thing AFAIK) and tools like Kustomize and Helm exist just to deal with this overcomplication
• the usefulness of parity with Docker Compose also cannot be overstated - you can usually get your stacks up and running with just some simple YAML changes
• Docker Swarm comes out of the box with any installation of Docker and also can be set up both on-prem as well as in the cloud; contrast this to tools like Minikube, MicroK8s and Kubespray among many others
• furthermore, Docker Swarm also has very good networking support out of the box, and therefore you're not forced to deal with integrating loosely coupled and abstracted networking solutions, of which there is a whole list here and each of which has their own idiosyncrasies
• Docker Swarm also wins in regards to resource usage, hands down - the master nodes will usually use a few hundred MB of RAM for smaller clusters (~10 nodes), whereas Kubernetes has large overhead in that regard and will need multiple GB of RAM free to work
• not only that, but Kuberentes doesn't really support overcommit out of the box and will simply refuse to run pods on nodes that don't meet some hard limits; want to run things on smaller VMs? tough luck. want to overcommit resources slightly? nope, not without messing around with the configuration, you won't
• the tooling around Docker Swarm is also pretty good and minimalist, for example, you can use Portainer (which also supports Kubernetes), much like you would use Rancher for easy administration of clusters
• speaking of tooling, there is a very pleasant integration with Ansible which allows easily controlling stacks and in addition to that, Portainer also provides webhooks to redeploy stacks in a really easy way

I'd say that for about 80%-90% of the environments out there, Docker Swarm or even Hashicorp Nomad are far better choices in regards to the operational complexity and the amount of resources that would be necessary to maintain the infrastructure. Yet, in reality, that's not what's going on.

So why is Kubernetes more popular than Docker Swarm?

Hype and monopoly, plain and simple.

Google has more money to spend on marketing. Google has more money to spend on ensuring that most of the platforms out there are integrated with Kubernetes. Google has more money to spend to throw millions of man hours at problems until they dissipate under the collective effort of thousands of engineers. And eventually, this cycle becomes self-perpetuating: to get a job, you need to know Kubernetes. You won't be fired for choosing Kubernetes in your projects either, because that's what the industry uses and it'll look good on your CV. It doesn't matter, that Kubernetes is resource hungry and is overkill for most projects and deployments out there. It doesn't matter, that cloud vendors will vendor lock you with Kubernetes in many different ways, and that running a cluster can be so difficult that you'll become dependent on their services. It doesn't matter, that your company will need to spend anywhere between 2x and 5x the resources you'd otherwise spend with Docker Swarm deployments.

Because Kubernetes is what everyone uses, because Kubernetes is what everyone writes tutorials for, because Kubernetes is what everyone talks about. Docker Swarm is largely alive only thanks to similarity of its manifest format with Docker Compose. If that was not the case, getting things running on it would be as hard as using Hashicorp Nomad - you'd find yourself in an almost total vacuum of examples, tutorials etc., and couldn't just take a docker-compose.yml file from the repository and run on that.

There are valid reasons as well, such as CRDs and almost endless abstraction and extensibility possibilities that Kubernetes has and Swarm doesn't due to architectural differences, but i'd argue that for most small deployments out there, such level of abstractions works AGAINST the developers, since all that the orchestrator should do is basically handle something similar to systemd services across many nodes.

So what to do about it?

Keep using Docker Swarm for as long as you can, for as long as it's suited for what you need to do. Its death is inevitable, yet isn't quite here either. Sooner or later, you WILL have to migrate to Kubernetes, but put that off for as long as you can while learning it in private (or pilot projects) and hope that more tools to simplify its utilization will prop up as time passes.

If you're forced to use Kubernetes and don't use the cloud managed (and vendor locked) variety, then consider something like K3s or even RKE with either Rancher or Portainer. You'll make your own life more tolerable. And don't get caught up in running software that's too complicated either: Caddy or even just Nginx with Certbot (or the Kubernetes variety of it) will be fully sufficient for ingress with SSL/TLS in most cases. NFS will be good enough for sharing files across multiple pods as well and so on...

The biggest problem i see is tools or pieces of software being developed ONLY for Kubernetes in the future - thus infecting the landscape and FORCING people to use Kubernetes to do the things they need. In my opinion, this is wrong - the actual software SHOULD be separate from how it's run. To that end, i also recommend that everyone familiarize themselves with The 12 Factor Apps which will be useful regardless of the technology that you end up using.

Will Docker Swarm die soon?

Lastly, i also believe that this definitely should be addressed. In 2019, Mirantis acquired Docker Enterprise and had this to say in regards to Swarm:

What About Docker Swarm? The primary orchestrator going forward is Kubernetes. Mirantis is committed to providing an excellent experience to all Docker Enterprise platform customers and currently expects to support Swarm for at least two years, depending on customer input into the roadmap. Mirantis is also evaluating options for making the transition to Kubernetes easier for Swarm users.

Of course, not everyone cares that much about Docker Enterprise and if you look at the moby repository then you'll notice that there is still ongoing work on Swarm functionality, as well as in Docker's own API repository, even if having separate orgs/repos like that is slightly confusing.

That said, Swarm is definitely in maintenance mode, which is actually how some people like it, given that this means pretty good stability for the current features, instead of breaking changes, which are a slightly larger concern in the case of Kubernetes. However, this also means that new features will get implemented only occasionally, if ever - the last big one that i recall was Swarm jobs. Essentially, i'd say that it's still just a question of time until Swarm dies, but for now it's in the sweet spot of stability and usability if all that you need is a pretty simple and lightweight orchestrator.

And in my mind, that's what most deployments out there need, at least before hitting a limit and then easily migrating over to K3s with something like Kompose. That way you'd also be sure that you're not relying on the actual orchestrator too much, but just use it as a tool, instead of building your entire infrastructure around it.