• 0 Posts
  • 19 Comments
Joined 1 year ago
cake
Cake day: June 20th, 2023

help-circle



  • Myself over NFS can have serious latency issues. Some software can’t correctly file lock over NFS too which will cause write latency or just full blown errors.

    iSCSI drops however can be really really bad and cause full filesystem corruption. Also backing up iSCSI volumes can be tricky. Software will likely work better and feel happy however and underlying issues may be masked until they’re unfixable. (I had an iSCSI volume attached to vmware silently corrupt for months before it failed and lost the data even though all scrubs/checksums were good until the very least moment).

    You can make your situations with with either technology, both are just as correct. Would get a touch more throughput on iSCSI simply down to the write confirmation being drive based and not filesystem locks / os based.

    YMMV









  • So my production setup is 2x10Gb bonded NICs for networking and 2x10Gb bonded NICs for Ceph/Cluster stuff. I suspect that when ceph is being heavily used you may see bottlenecks however once you have host based failure then in theory your data should be closer to the correct host and not have an issue. But it’s on a basic level like have 3 copies of data, one on each host so it doesn’t save you any storage, just reduces the risks during failure.

    Thinking about it, you may actually see better results with ZFS and replicate jobs. As there’s fewer overheads and the ZFS sending is incremental. You’d obviously just loose X minutes of data instead of ceph being X seconds.


  • Ceph works best if you have identical osd, quantity, type and capacity across the cluster, also works best on a 3+ node cluster.

    I ran a mixed sata SSD/HDD 256gb/4tb cluster and it was always a bit pants. Now I have 7x1tb SSD per node (4nodes) and it works fantastic now.

    Proxmox uses replica 3/2 failure at host level but you may find that EC works better for your mixed infra as you noticed you can’t meed the 3 host failure and so setting to osd failure level means data may be kept on a single host so would need to traverse the network to the other machine.

    You may also need more than a single 10Gb nic too as you might start hitting bandwidth issues.





  • IPv6 doesn’t support NAT… Or am I woefully out of date.

    But your home router will just firewall like it does already but you don’t have NAT as a simple fall back for “security”. It does make running internal services much easier as you no long need to port forward. So you can run two webservers on port 80 and they be bother allowed inbound without doing horrible load balance or NAT translation.