So let me set the scene.
On my rack I have Nasira, which is a FreeNAS server with about 30TB effective storage. It’s connected to a 10GbE connection, built on an AMD FX-8350 and 32GB ECC RAM.
I also have a few loose machines. One is a GPU compute box for BOINC, sporting three NVIDIA graphics cards. Another is another GPU compute box sporting two AMD graphics cards for Folding@Home. And the thought came to mind that I didn’t want to put an SSD in either. But I don’t have any spare SSDs at the moment and didn’t feel like buying any.
But I do have a few spare USB drives lying around. So I wondered if I could get a quasi-diskless server up and running without setting up a PXE server, with all the time and complication that comes with that.
And since you’re reading this, you can pretty readily assume that the answer is… Yes, right? Well, read on and find out.
iSCSI
Briefly, I’ll discuss iSCSI here. It’s basically remote storage across a network similar to a file share. The exception is where a NAS is visible to your computer as a filesystem, an iSCSI share is treated as if it’s a physical drive. Just through a network connection instead of a SATA port or USB plug.
It has numerous advantages to a regular network file share, including being usable as direct storage for installing an operating system.
Since setting up an iSCSI share varies based on the NAS software, I won’t go any further here. All I’ll say is to make sure to set up your iSCSI share so it points to a file instead of one of the physical drives attached to the NAS.
Making it work
So what’s the catch? Why the USB drive? Well something needs to hold the /boot partition and the bootloader. (And /boot/efi if you’re setting up using UEFI.) I used a 16GB USB drive, but I see no reason why this can’t work with smaller capacity drives, likely even as small as 1GB if not smaller.
But is there a catch? Well there is one big one: the Linux distro you select must be able to link to an iSCSI target during installation. (Unfortunately that means Ubuntu and Debian are both out.) How to set that up varies based on distribution. I’ve done this only with Fedora and CentOS, so I can only speak directly to those and will use only those for demonstration herein. (Note: With CentOS, make sure to enable the network connection before you try to select an iSCSI target or it’ll say your login credentials are bad.)
(Note: In the screenshots below, I’m using VirtualBox for demonstration, so where you see “VBOX HARDDISK”, that’s the USB drive.)
The main thing here is you cannot use any “automatic” partitioning. You instead need to use a custom partition layout. The /boot mount (and /boot/efi, if applicable) needs to reside on the USB drive. Everything else (swap and / at minimum) resides on the iSCSI target.
Note as well that, at least with CentOS (I don’t recall seeing this with Fedora), when you set your /boot mount (and /boot/efi, if applicable), it may try to assign that to both the iSCSI target and USB drive. So if you see “and 1 other” under “Device(s)”, click Modify and click just the USB drive, then Select to save it.
And the USB drive needs to be set as the boot device. Obviously.
Conclusions
So if you want to set up servers to boot from iSCSI, this is a way to do it without setting up PXE and all the complications that come with it. Just enable iSCSI in your NAS, set up the targets as you need for each server, install Linux as described above, and you’re golden.
Sure it’s not a true disk-less setup, but it’s enough for my needs. It’s quick to setup. And a lot easier to setup as well compared to creating a PXE server and getting that working. (Plus that server becomes an additional point of failure in your network along with another target for ongoing maintenance.) And if you don’t have spare hardware you can use to make a PXE server, it’ll likely be cheaper to just buy a few USB drives if you don’t already have a few lying around.
It’d be great if I could just enable iSCSI in the UEFI for each server, but that’s probably not coming to consumer motherboards any time soon, if ever.
You must be logged in to post a comment.