this post was submitted on 15 Aug 2023
20 points (100.0% liked)

Selfhosted

39937 readers
373 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

I currently run a Lenovo minipc and an rpi4 to host all of my homelab services, and I just purchased a new minipc with an 11th Gen i5 in order to better handle Plex and transcoding. I would also like to add Immich to my homelab. Here is a breakdown of my current state:

Lenovo Mini PC running Proxmox: Home Assistant OS VM Ubuntu VM running:

  • Frigate
  • Pi-hole
  • Plex
  • Arr Stack
  • Transmission
  • Has two ext HDs in a mergerfs configuration to host all media and onsite homelab backups.

RPi4:

  • Wireguard VPN
  • Pi-hole

I am going to install Proxmox on the new minipc, and setup both in a cluster. I would like to move Plex, Arrs, Transmission, and Immich to the new mini PC. Because of that I am thinking the external drives should be connected directly to the new minipc to maximize performance. I would also like to have the ability to share the external drives to any nodes in the cluster.

Do you have any recommendations on how to best setup these drives? Should I setup a TrueNAS VM on the new minipc and share to all others via NFS? Any other tips for my setup?

top 12 comments
sorted by: hot top controversial new old
[–] DrWeevilJammer@lemmy.ml 6 points 1 year ago (1 children)

You may have trouble keeping a quorum with a proxmox cluster of 2. You should really have 3, or set up a q-device. (Source)

Each node in a cluster has a "vote" in a healthy cluster. If one of your devices goes (or is taken) down, you'll lose quorum, and the GUI and some other stuff. You need 3 votes for things to work reliably. I have an old RPi set up as a q-device and it works fine.

[–] monty@lemmy.one 1 points 1 year ago (1 children)

This only applies to high availability clusters, correct?

[–] DrWeevilJammer@lemmy.ml 2 points 1 year ago (1 children)

Unfortunately not, it applies to all clusters. A quorum requires at least 2 votes, which means 2 nodes. But if one node goes down, you only have 1 vote, and the cluster will go into read-only mode, which means you'll lose the GUI and the ability to manage your nodes:

When your cluster is non-quorate (so at least half of all nodes are dead), the remaining nodes will change the PVE management into read-only mode. Because of that you will no longer be able to change and manage your VMs and containers and will also not be able to log into the GUI. This is done to avoid cluster split-brain problems in which they run into inconsistent states. (Source)

But if you have a device that will supply a vote in the event that one of the two votes is unavailable, the cluster will continue to function with a single node, which will allow you to use the normal Proxmox tools and interfaces to diagnose the problem, while also keeping any VMs on the single remaining node up and running (available).

Here is a Proxmox employee explaining it a bit more clearly than the official documentation:

... a cluster needs to be always quorate to work properly, not just for HA. High availability just means that the cluster will try to keep your HA-enabled VMs and containers always available, i.e. if a cluster node fails the HA-manager will launch HA-managed guests on another cluster node.

While a 2 node cluster should work with 2 active nodes, if one of your nodes goes down your cluster will automatically be non-quorate and will no longer work as expected. To have quorum in your cluster, you need a setup of at least 3 nodes, though you do not need 3 full Proxmox installations ... you can setup something like a Raspberry Pi as a QDevice for external vote support.

(Source)

[–] monty@lemmy.one 2 points 1 year ago

Wow thanks for that detailed explanation! Very helpful and I'll be setting up an rpi as a device.

[–] MangoPenguin@lemmy.blahaj.zone 3 points 1 year ago (1 children)

No need for the TrueNAS VM, Proxmox supports RAID, ZFS, and all that natively.

To share the storage with other nodes you can set up an NFS share and add that in Proxmox and into VMs to use as storage.

[–] monty@lemmy.one 1 points 1 year ago (2 children)

I did some searching trying to figure out how I can share the NFS directly from Proxmox and came up short. Any tutorials or other links you know of?

[–] VelociCatTurd@lemmy.world 2 points 1 year ago

Well it’s a Debian based Distro, so you could install NFS. What I do is I have OMV running in a vm and use that to serve up my smb and NFS shares

[–] MangoPenguin@lemmy.blahaj.zone 2 points 1 year ago (1 children)

It's just a Debian system so any general linux guide for NFS shares will work.

[–] monty@lemmy.one 1 points 1 year ago (1 children)

Got it so I'm just overthinking this. I setup an NFS share through the host via the command line and then I'll be able to share that to anything on the network.

[–] MangoPenguin@lemmy.blahaj.zone 1 points 1 year ago (1 children)

Yep! That's how I do my media share that's using ZFS on a Proxmox host

[–] monty@lemmy.one 1 points 1 year ago (1 children)

So I'm having some permission issues with this. If I create a file within one container, it is read only when accessed from another. Any suggestions on how to fix this?

Easiest is just chmod -R 777 the files which is what I do. I'm sure there's a proper way to do it, but it's just media files.