I would echo the general sentiment in this article for my experiences with bare metal clusters.
I started with MicroK8s, and while it's a functional solution for some use-cases, I was genuinely disappointed in it's overall behavior (esp with regards to small node count clusters, in the 3 to 10 node range).
The biggest hit was Dqlite - Overall I had a tremendous number of problems that originated explicitly with Dqlite. Everything from unexpected high cpu usage, failure to form consensus with even node counts (esp after a network split), manual configuration files that needed to be deleted or renamed to get specific hosts back into the cluster, and generally poor performance for a long term setup (2 year old cluster stalled to basically a standstill spinning on Dqlite).
I have not used Dqlite in other projects, so it's possible this was a Microk8s problem, but based on my experience with Microk8s... I won't touch either of these projects again.
I switched away to K3s about 3 years ago now and have had essentially no problems. Considerably fewer random headaches, no unexpected performance degradation, very stable, incredibly pleasant to work with.
---
I have also migrated about half of my workloads to Longhorn backed PVs at this point (coming from a large shared NAS exposed as NFS) and while I've had a couple more headaches here than with K3s directly - this has been surprisingly smooth sailing as well, while giving me much more flexibility in how I manage my block devices (for context, I'm small - so just under a petabyte of storage, of which ~60% is in use).
If you want to run a cluster on hardware you own rather than rent - K3s and Longhorn are amazing tools to do so, and I really have to give Rancher/SUSE a hand here. It's nice tooling.
K3s is truly great - been using it for years for just about everything not warranting a full cluster. MicroK8s feels like it does too much in non-standard ways and when it breaks you're suddenly dealing with Snap related issues and it's a total immersion break.
Lately there's also RKE2 (https://docs.rke2.io/) that I've been growing fondness for and it's only marginally more tricky to setup, with the bonus effect of having a more 'standard' cluster distribution and more knobs to twist.
Not that I'd be shy of running K3s in production, but it seems easier to follow 'standard Kubernetes way' for things without having to diff with some of K3s's default configuration choices - which, again, aren't bad at all for folks who do not need all of the different options.
For edge workloads and smaller clusters / less familiar operators that want to run Kubernetes platforms themselves without depending on a managed provider, K3s is pretty impossible to beat.
If Docker Swarm is on the table (it's practically abandonware), I'd like to throw in Nomad by HashiCorp. It's also fairly lightweight, very flexible (can run various types of containers but also exec any executable), decent ecosystem (supports CNI and CSI). It can scale from single node control plane + workloads to tens of thousands of nodes, and the workers can also be geographically spread out (different regions/zones/dcs).
A few years back I wrote about it, and most of the core principles of the article are still valid:
I had a distaste for Nomad before the rug pull[1], but hopefully no serious person would voluntarily choose Nomad after that stunt. Doubly so that going with the Nomad ecosystem mandates self-hosting as one cannot $(aws elastic-nomad create-cluster). I was similarly surprised that HashiCorp didn't even offer "you pay for compute but we will administer your control plane" since both the "community" and the "enterprise" links from the front page both say "self-managed"
I will be super curious to see if IBM Cloud actually does ship $(ibm-cloud-cli create-nomad)
Agree to disagree on if a license change (to one that impacts nobody but potential resellers) constitutes a rug pull.
BSL and similar are the second best thing (after open source licenses), and still drastically better than proprietary/closed source. If it allows someone to pay the salaries of the people developing it, I'm fine with that. And not only because my own salary is one of them - I had the same view for MongoDB, Elastic, Sourcegraph, etc. In the era where massive behemoths can just ship your software as a service for free, companies need to protect themselves as much as they can. Do you ever wonder why there are very few profitable open source companies? Most people would know Red Hat and that's it.
Oddly, I'm moving all my stuff to k8s because of how convenient and easy everything is to admin with tooling like k9s. But I've only ever worked with toy clusters with nodes in the single digits, and I can see what a nightmare debugging something like networking or storage at scale can be.
Also, helm gave me a loathing for yaml I never knew I had in me. I avoid some of helm's more vulgar gyrations by using helmfile, but I'm also giving yoke a look.
> Also, helm gave me a loathing for yaml I never knew I had in me
If it's the whitespace that jams you up, I wanted to point out that since YAML is a superset of JSON, there's nothing stopping you from using text/template to cook JSON manifests. There is, of course, a toJson function available in helm <https://helm.sh/docs/chart_template_guide/function_list/#typ...>
But I'm conceptually with you that the jokers who decided to use a text templating language for a structured output were gravely misguided and now we all suffer
I'd be curious to know what prompt was used to generate this article. I don't mean any offence, most content on the web is generated by LLMs these days one way or another anyway, I'm just curious about the exact prompt used in this case.
Write a comprehensive technical blog post targeting small and medium-sized businesses that use Hetzner Cloud and are evaluating lightweight Kubernetes alternatives. Compare and analyze the following solutions: k3s, MicroK8s, Docker Swarm, and Minikube.
Structure:
Start with a title and abstract introducing the comparative scope.
Provide sections such as:
Architectural Requirements,
Deployment & Lifecycle, Management,
Cost Analysis,
Security & Compliance,
Developer Experience,
Strategic Recommendations and
end with a clear conclusion and deployment advice.
Style:
Technical but accessible to DevOps engineers or small startup CTOs.
Use citation-style references for data (e.g., [1], [2], etc.).
This article is centered around Hetzner as a cloud provider but I'm not sure I'd trust Hetzner for anything actually important. It's been known to boot regular, paying customers off the platform without appeal, without prior announcement and without giving proper time to back up data and move the service elsewhere.
Other than the `hetzner-k3s` installer, nothing in the article is specific to Hetzner. And even that script is just some terraform presets.
Spoiler for people who actually care about TFA: it's a runaway win for k3s. I've never even heard of minikube being run in production. The setup stuff at the end is a complete handwave though, like "Implement Traefik ingress with Let’s Encrypt via servers.ingress.enabled=true". No serious setup works well with traefik's default LE integration, and even the k3s docs recommend you install cert-manager. Which isn't too awful to set up, but it's still several hours of fiddling if you haven't done it before (pro tip: install it with kubectl, do not use helm. helm and CRDs are still not friends.)
I would echo the general sentiment in this article for my experiences with bare metal clusters.
I started with MicroK8s, and while it's a functional solution for some use-cases, I was genuinely disappointed in it's overall behavior (esp with regards to small node count clusters, in the 3 to 10 node range).
The biggest hit was Dqlite - Overall I had a tremendous number of problems that originated explicitly with Dqlite. Everything from unexpected high cpu usage, failure to form consensus with even node counts (esp after a network split), manual configuration files that needed to be deleted or renamed to get specific hosts back into the cluster, and generally poor performance for a long term setup (2 year old cluster stalled to basically a standstill spinning on Dqlite).
I have not used Dqlite in other projects, so it's possible this was a Microk8s problem, but based on my experience with Microk8s... I won't touch either of these projects again.
I switched away to K3s about 3 years ago now and have had essentially no problems. Considerably fewer random headaches, no unexpected performance degradation, very stable, incredibly pleasant to work with.
---
I have also migrated about half of my workloads to Longhorn backed PVs at this point (coming from a large shared NAS exposed as NFS) and while I've had a couple more headaches here than with K3s directly - this has been surprisingly smooth sailing as well, while giving me much more flexibility in how I manage my block devices (for context, I'm small - so just under a petabyte of storage, of which ~60% is in use).
If you want to run a cluster on hardware you own rather than rent - K3s and Longhorn are amazing tools to do so, and I really have to give Rancher/SUSE a hand here. It's nice tooling.
K3s is truly great - been using it for years for just about everything not warranting a full cluster. MicroK8s feels like it does too much in non-standard ways and when it breaks you're suddenly dealing with Snap related issues and it's a total immersion break.
Lately there's also RKE2 (https://docs.rke2.io/) that I've been growing fondness for and it's only marginally more tricky to setup, with the bonus effect of having a more 'standard' cluster distribution and more knobs to twist.
Not that I'd be shy of running K3s in production, but it seems easier to follow 'standard Kubernetes way' for things without having to diff with some of K3s's default configuration choices - which, again, aren't bad at all for folks who do not need all of the different options.
For edge workloads and smaller clusters / less familiar operators that want to run Kubernetes platforms themselves without depending on a managed provider, K3s is pretty impossible to beat.
If Docker Swarm is on the table (it's practically abandonware), I'd like to throw in Nomad by HashiCorp. It's also fairly lightweight, very flexible (can run various types of containers but also exec any executable), decent ecosystem (supports CNI and CSI). It can scale from single node control plane + workloads to tens of thousands of nodes, and the workers can also be geographically spread out (different regions/zones/dcs).
A few years back I wrote about it, and most of the core principles of the article are still valid:
https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
Disclaimer: I work at HashiCorp, but I've had that opinion since before joining and in fact it's among the reasons I joined
I had a distaste for Nomad before the rug pull[1], but hopefully no serious person would voluntarily choose Nomad after that stunt. Doubly so that going with the Nomad ecosystem mandates self-hosting as one cannot $(aws elastic-nomad create-cluster). I was similarly surprised that HashiCorp didn't even offer "you pay for compute but we will administer your control plane" since both the "community" and the "enterprise" links from the front page both say "self-managed"
I will be super curious to see if IBM Cloud actually does ship $(ibm-cloud-cli create-nomad)
1: I also didn't realize they rug pulled Consul, too; that's just cruel https://github.com/hashicorp/consul/blob/v1.20.5/LICENSE
Agree to disagree on if a license change (to one that impacts nobody but potential resellers) constitutes a rug pull.
BSL and similar are the second best thing (after open source licenses), and still drastically better than proprietary/closed source. If it allows someone to pay the salaries of the people developing it, I'm fine with that. And not only because my own salary is one of them - I had the same view for MongoDB, Elastic, Sourcegraph, etc. In the era where massive behemoths can just ship your software as a service for free, companies need to protect themselves as much as they can. Do you ever wonder why there are very few profitable open source companies? Most people would know Red Hat and that's it.
Oddly, I'm moving all my stuff to k8s because of how convenient and easy everything is to admin with tooling like k9s. But I've only ever worked with toy clusters with nodes in the single digits, and I can see what a nightmare debugging something like networking or storage at scale can be.
Also, helm gave me a loathing for yaml I never knew I had in me. I avoid some of helm's more vulgar gyrations by using helmfile, but I'm also giving yoke a look.
> Also, helm gave me a loathing for yaml I never knew I had in me
If it's the whitespace that jams you up, I wanted to point out that since YAML is a superset of JSON, there's nothing stopping you from using text/template to cook JSON manifests. There is, of course, a toJson function available in helm <https://helm.sh/docs/chart_template_guide/function_list/#typ...>
But I'm conceptually with you that the jokers who decided to use a text templating language for a structured output were gravely misguided and now we all suffer
There is also JSonnet and the appropriate tooling with tanka.dev
I'd be curious to know what prompt was used to generate this article. I don't mean any offence, most content on the web is generated by LLMs these days one way or another anyway, I'm just curious about the exact prompt used in this case.
I think something around those lines:
Write a comprehensive technical blog post targeting small and medium-sized businesses that use Hetzner Cloud and are evaluating lightweight Kubernetes alternatives. Compare and analyze the following solutions: k3s, MicroK8s, Docker Swarm, and Minikube.
Structure:
Start with a title and abstract introducing the comparative scope.
Provide sections such as:
Architectural Requirements, Deployment & Lifecycle, Management, Cost Analysis, Security & Compliance, Developer Experience, Strategic Recommendations and end with a clear conclusion and deployment advice.
Style:
Technical but accessible to DevOps engineers or small startup CTOs.
Use citation-style references for data (e.g., [1], [2], etc.).
Incorporate code examples (Terraform, Helm, CLI snippets).
Use tables for cost comparisons or benchmarks where appropriate.
Mention open-source tools specific to Hetzner (e.g., hetzner-k3s, hcloud Terraform provider).
Tone:
Professional, analytical, and informative. No marketing fluff.
This article is centered around Hetzner as a cloud provider but I'm not sure I'd trust Hetzner for anything actually important. It's been known to boot regular, paying customers off the platform without appeal, without prior announcement and without giving proper time to back up data and move the service elsewhere.
Other than the `hetzner-k3s` installer, nothing in the article is specific to Hetzner. And even that script is just some terraform presets.
Spoiler for people who actually care about TFA: it's a runaway win for k3s. I've never even heard of minikube being run in production. The setup stuff at the end is a complete handwave though, like "Implement Traefik ingress with Let’s Encrypt via servers.ingress.enabled=true". No serious setup works well with traefik's default LE integration, and even the k3s docs recommend you install cert-manager. Which isn't too awful to set up, but it's still several hours of fiddling if you haven't done it before (pro tip: install it with kubectl, do not use helm. helm and CRDs are still not friends.)