Been playing around with setting up a Talos Linux-based kubernetes cluster recently, although I’ve found distributed storage very difficult to do cheaply.
I need enterprise SSDs with power-loss protection (PLP) for Ceph or it won’t be stable… and then having a HDD network share for bulk storage…
@snuggle I have 10GbE on a separate network already, but IOPS with HDDs on Ceph was so poor it kept locking up.
I wonder if Longhorn could do what I want with having distributed HDD storage, but it seems to only be block-based storage.
It’s a shame TrueNAS Scale doesn’t yet scale.
@Snuggle rook-ceph does do EC so it's pretty resilient, so you probably don't need PLP unless you are doing something super intensive
@snuggle My ideal is my 3 nodes to have distributed HDD storage and it sounds like this could be done with GlusterFS before it was discontinued.
Now my options are Ceph, which seems PLP SSD-only, and Longhorn which likely has similar hardware requirements.
