Doing it right this time.

Until now I’ve been using a RockPro64 in the (excellent for starters) Pine64 NAS case for my NAS. It runs Armbian, openzfs, and I manage my ZFS as needed on the CLI. I share my datasets over NFS to my Proxmox guest VMs for access. This has served me reasonably well for the past few years. But we have a baby coming soon, and along with that a flurry of new photos and videos for my family to store on my NAS. And although I probably have enough buffer to survive the initial few months, I certainly wish to do better than my current 2xHDD mirrored pool giving me ~4TB of storage.

The Plan

Luckily I already possess the hard-drives I need for my desired pool. In total, I have 2 HDDs from my current NAS, and 2 spares - all the same 4TB capacity.

I still want to keep my current NAS running as a backup store, but I only need 1 drive for this purpose (for now). That leaves me with 3 HDDs we can use for our new raidz1 pool which would give me 2 drives for storage and 1 for parity.

Now for the platform. I have been a very happy user of Proxmox (on my compute node) and it already comes ready with ZFS support so I’ll stick with that. I did consider TrueNAS (both standalone, and virtualized within Proxmox with disk-passthrough), but when it comes to storage, I want as few surprises as possible so I stuck with what I know. I’m already used to ZFS on the CLI and Proxmox is based on Debian so if/when something goes wrong, I don’t want to be messing around with an unfamiliar UI.

Although I don’t want to manage ZFS via a GUI, I’d like a way to manage access to my datasets via a GUI. Cockpit seems like a good light-weight choice. It runs as an LXC container within Proxmox using bind-mounts for the dataset paths. I can then select which paths I allow access to over my network using NFS, and SMB protocols on a per-user basis.

Shopping

Needs:

• >= 3 hot-swappable HDD bays
• >= 16 GB RAM (for ZFS)
• runs Proxmox

Wants:

• IPMI
• 1U
• 10Gb networking

On eBay, I ended up checking out with:

which gave me everything I needed and wanted!

Build

First I free one of my HDDs from my RockPro64 NAS for use in my new pool. This means I am running on a single HDD for the remainder of the migration.

With the free HDD, and my two spares, I get everything assembled, and set up a new Proxmox node with a ZFS pool from the UI using the 3 disks.

Sync

To migrate all of my data from the old pool to the new pool I use syncoid. It uses ZFS snapshots to accomplish this. It’s awesome.

This takes many hours depending on the size of your pool and your connection, but since I saw almost full utilization of my network link I was convinced it’s the best I can do right now.

I’ve seen in a few Reddit posts suggesting ways this can be optimized using some combination of zfs send with mbuffer rather than ssh which syncoid uses, but I’m happy with the convenience of syncoid.

The best part is that everything can still be running while this is going on! This is because only the first sync takes a long time since it needs to copy everything over. All subsequent syncs only transfer the delta since the last sync which should be pretty quick if your data does not change all that much.

That being said, since I am now running on a single drive, and hitting it hard during sync, I do not want to stay in this state for any longer than I need to due to potentially leading my single disk to failure.

Once the sync is complete, I do a second sync (for the delta), and promptly shut down all of my services to prepare for the cut-over.

Cockpit

Before I can re-enable my services. I need to expose my datasets on Cockpit.

I create a Debian LXC, and assign the generated MAC a static IP on my network. Other than that, I just make sure my container has the NFS feature enabled:

Setting up user permissions is a pain, but once done I expose the relevant NFS as well as Samba paths for my services - replicating my old NAS’s shared paths.

Voila

In my homelab nameserver, I update the A record for my NAS to point to the new IP of my Cockpit LXC container, and start my services back up.

I confirm everything is still good by turning off the old NAS, and yep still good.

Luckily for me all good on the first try!

Finishing touches

I set up a cron on my old NAS to sync data nightly from the new pool. This will only work while the actual data size on my new pool is less than 4TB, so I’ll need to get a new drive with higher capacity to keep the backups going in the future.

Luckily I should have about a year until I exceed 4TB on the new pool so I’ll be keeping an eye on deals to snag an 8TB backup drive along with a few spares for my new NAS.

Resources

• https://www.apalrd.net/posts/2023/ultimate_nas/
• https://blog.kye.dev/proxmox-cockpit