Dell R520 - The expansion continues...

[Kai Robinson]

For those that are unaware, the TD Game server is run on a Dell R520 - as shown here: https://tinkerdifferent.com/threads/tinkerdifferent-game-server-now-testing.2909/

In modern terms, it's ancient, was destined for the e-waste until i rescued it. I mean, who'd want a 10-core, 20 thread machine, it's SOOO old, right?

Since then, thanks to @Bolle it's been upgraded to 192GB of RAM and i've replaced the battery in the H710 PERC RAID card and added a few more trays ready for 6x SAS drives.

This is of course mega-overkill for the likes of an XP64 VM, i've allocated a whole TWO cores and 8GB of RAM to it (mad, right?). Obviously there's also the Rust VM for the TD PvE server, which is maintained by Xenocide from the TD Discord, which uses a few more cores, but again, 8 cores and 64GB RAM allocated to it - which leaves plenty more RAM and 10 more cores...

I also run my PleX server off another VM in it - with another 8-cores allocated and 16GB RAM.

Anyway, the point is, general resources are not an issue on this, but what I wanted to work on, mostly for learning, was to upgrade the disk subsystem and the networking to make it an all-in one solution to replace my very, very old (10 year old) Netgear READYNAS 104 (single core ARM w/512MB ugh!) with 4 x 6TB Disks (thanks to Erebus, also from the TD Discord).

Even though I work at an MSP and have been in IT for years - my experience with server hardware has some holes in it - specifically with regards to anything higher than Gigabit ethernet. I've not had experience with HBA's and NIC's using SFP/SFP+ so it's a little bit of a minefield.

So, with the additional 6 bays, I wanted to setup a RAID 50 - 6x6TB disks total split into two RAID5 arrays, mirrored, for some redundancy, running off the H710 PERC.

I wanted to have the storage pool directly available to both the PleX VM and create a new VM for the purpose of being a fileshare on the network - an iSCSI target basically.

I want to have a FAAAAST connection to my main PC, was going to get an Intel X520 10Gbit SFP+ NIC for both the R520 and my desktop PC side of things (they're about £19 here) and a single 3m Dell DAC cable to connect the two directly without relying on a 10Gbit switch. But, my knowledge here is patchy - i've read up, and this LOOKS like the right way to do it - rather than using a generic SFP+ transceiver and an LC-LC multimode fibre cable. It's all theory - so...would this work and does this sound feasible?

 

[Kai Robinson]

I have binned off the Dell specific DAC cable for one from FS - the NIC selection has also changed as i've been reading up that the Intel X520's are a bit picky with what transceivers and DAC cables they want to use.

Instead, it's looking like i'm going to pull the trigger on some single port Mellanox ConnectX-3's.

The VM has already been spun up and is ready for a test run this week over standard 1Gb ethernet.

Then all that remains is to purchase a boatload of disks!

 

[bakkus]

My 2 cents:

I wouldn't recommend running RAID50 in a 6-disk setup.
As we touched upon on Discord, RAID50 or rather 5+0 isn't for redundancy - it's for performance: https://www.techtarget.com/searchstorage/definition/RAID-50-RAID-50

Splitting the load across 2x RAID5 sets for more speed, mainly for writes.
However, if you're doing 2x 3-disk RAID5 sets, each of those arrays are running at the _absolute bare minimum_ amount of disks for RAID5.
In other words, you're pretty much guaranteed to lose large amounts of data you lose a disk. Risk times 2, since each 3disk set is its own separate RAID5.

For a 6-disk array I'd look into RAID10 (1+0), or just RAID5 across all disks - won't be SPEEED, but it'll be reliable and you'll still win performance since you're distributing the load across more spindles.

And yes, you can connect a DAC back-to-back.

 

[Kai Robinson]

My 2 cents:

I wouldn't recommend running RAID50 in a 6-disk setup.
As we touched upon on Discord, RAID50 or rather 5+0 isn't for redundancy - it's for performance: https://www.techtarget.com/searchstorage/definition/RAID-50-RAID-50

Splitting the load across 2x RAID5 sets for more speed, mainly for writes.
However, if you're doing 2x 3-disk RAID5 sets, each of those arrays are running at the _absolute bare minimum_ amount of disks for RAID5.
In other words, you're pretty much guaranteed to lose large amounts of data you lose a disk. Risk times 2, since each 3disk set is its own separate RAID5.

For a 6-disk array I'd look into RAID10 (1+0), or just RAID5 across all disks - won't be SPEEED, but it'll be reliable and you'll still win performance since you're distributing the load across more spindles.

And yes, you can connect a DAC back-to-back.

Yeah i might just YOLO it, run a RAID5 setup and then have a separate on-site backup to a NAS as well as Crashplan for really important data.

 

[ajacocks]

That’s what I do on my small NAS. I have a 4-drive RAID5 and external HDDs for backup. I use mergerfs to join my backup drives, since my RAID is far larger than any one available SATA/SAS HDD currently sold.

- Alex

 

[Kai Robinson]

Update time!

I ended up indeed getting some Mellanox ConnectX-3's from eBay - not bad for about £19 each. One in the R520, one in my desktop. A single 5M DAC from FS links the two together. I created a test iSCSI share that ran solely over the Mellanox on the R520 (2TB) using the primary RAID0 boot volume for the VHDX storage. Speeds were NUTS! Exceeded 300MB/sec, peaking around 340MB/sec, for the 56GB .mkv i sent over to it (Aliens Re-Mastered Directors Cut in 4K). I think that'll be fast enough.

Now the only thing I was lacking was the disks themselves. As @bakkus has mentioned, RAID50 was a bad idea - so here, i'm going for a standard RAID5. Doing it like this also allows me to expand it by a further 2 disks when I need to. IE - in 2 months time, when i've likely filled it

Therefore - after a boatload of overtime, I pulled the trigger on these:

That's 4 x HGST HUS726060AL4210's - 6TB, 12GBps SAS drives. Total cost? Well i offered £30 per drive, to be cheeky...they accepted!

System Supply Industries in the UK has you covered for 2nd hand enterprise gear and it is CHEAAAAAP! Why bother with a Synology when you can get an R520 or R520 chassis with CPU and RAM for £100 and 18TB of disk space for £120 on top?

I've setup a separate NAS VM for it, with iSCSI enabled - the idea is, I'll browse for files on my desktop, download them locally to SSD, punt them over the iSCSI super quick, and the VM will share everything else out locally via the internal Hyper-V switch to the PleX VM and via SMB to the rest of the network.

Ideally, i'd like a switch with at least 12 PoE ports, another 2 x 10Gb SFP+ ports - but i'm not finding anything I like, at least nothing that's not absolutely honking massive (48 port ex-enterprise gear, which really IS overkill!

I have my eye on one of these, but they're about 10 years old now: https://www.ebay.co.uk/itm/126516738905
And then there's this wee-beastie: https://www.ebay.co.uk/itm/296403866432

Argh, decisions, decisions...

 

[This Does Not Compute]

Ideally, i'd like a switch with at least 12 PoE ports, another 2 x 10Gb SFP+ ports - but i'm not finding anything I like, at least nothing that's not absolutely honking massive (48 port ex-enterprise gear, which really IS overkill!

Juniper EX2300-C-12P would be a good choice if you want an enterprise-quality switch in a compact, fanless footprint -- but it's best managed via CLI, so you'd have an opportunity to learn JunOS.

If you want just a "dumb" unmanaged switch, I'd recommend keeping an eye on ServeTheHome's networking section, as they frequently review no-name switches that are showing very interesting specifications at crazy low prices.

 

[Kai Robinson]

Juniper EX2300-C-12P would be a good choice if you want an enterprise-quality switch in a compact, fanless footprint -- but it's best managed via CLI, so you'd have an opportunity to learn JunOS.

If you want just a "dumb" unmanaged switch, I'd recommend keeping an eye on ServeTheHome's networking section, as they frequently review no-name switches that are showing very interesting specifications at crazy low prices.

I came across a Juniper EX2200 although saw some comments that the CPU was slow, i'll have a look at the range to see what's available...also being that i work for an MSP it's entirely possible thatI might end up finding a suitable switch bound for e-waste.

The NAS VM was binned off - getting it to work right just did my head in, so i've decided to run the file sharing on the root Hyper-V instead, makes the volume management easier.

Transferring files over now is just...

Mounted the file share as a mapped network drive in write-through mode to disable any SMB caching for more consistent transfer speeds - 550MB/sec for the whole duration of a file transfer, instead of 900MB/sec for a short 5 second burst, then a sharp drop and wobbly 150 to 300MB/sec.

Not bad for £120 outlay!

I'll get another two more 6TB SAS drives on payday, to get this thing up to 30TB total capacity.

The original ReadyNAS 104 that has been doing daily service for a decade was finally decommissioned from it's primary role and i've reset it to factory settings and setup two iSCSI LUN's on it instead - one 8TB, one 6TB with the 8TB now the primary repository for Veeam, so I have my desktop PC and all the VM's being backed up as well. Finally i'm getting to where i'm happy! That being said, projects like this never end...and I did spot a 12-bay Rackmount QNAP NAS for £129 that I might have to get my grubby mitts on

After the drive stack for the server and the NAS are sorted, I think the next step will be a cheap nVidia TESLA card for PleX transcoding, take the load off the CPU.

Overkill? Yes.
Is that bad? No.

 

[Chace]

Good Afternoon. Thank you for posting about your journey in building your NAS and gaming server using the R520. I just bought one a few weeks back from a county sale in my area with no plans or clue of what I was going to do with it. Your posts have given me a lot to understand and the basics to attempt as I am am fairly new to servers in general. From your experiences with this machine, is there any must do's that you would recommend or things to avoid when building mine into a gaming server/plex streaming machine?

 

[Kai Robinson]

Good Afternoon. Thank you for posting about your journey in building your NAS and gaming server using the R520. I just bought one a few weeks back from a county sale in my area with no plans or clue of what I was going to do with it. Your posts have given me a lot to understand and the basics to attempt as I am am fairly new to servers in general. From your experiences with this machine, is there any must do's that you would recommend or things to avoid when building mine into a gaming server/plex streaming machine?

Hi - Apologies for the delay getting back to you!

So - the must do is to replace the battery in the PERC H710i Minimono RAID controller - this will allow you to change the cache settings for the controller, and will remove the potential spicy pillow situation from forming. Also, the machine will moan at you with a dead battery.

Secondly - upgrade the power supplies, they're dirt cheap and a pair of 750Watt units can be yours for less than £40. Maxing out the CPU's is easy and as mentioned previously in this thread - cheap. A pair of 10c/20t's will do almost everything you want. I'd recommend at least 64GB RAM for messing around with VM's (no one wants a VM with 8GB or less of RAM!)

If you're going to setup plex - do so in a VM. That makes the VM and the PleX database more portable if you ever need to migrate to other hardware.

Funnily enough, I grabbed a Dell T430 for....free, through the works e-waste bin, so i've just got a pair of E5-2650 V4's for....£9! (12c/24t each) and 128GB ECC DDR4 for £72 and now i'm adding a Quadro P620 card for hardware transcode of the PleX streams - it can handle two concurrent 4K video streams and I just paid £30 for the card. (It's Pascal architecture, sits between a GT1030 and a GTX1050 in terms of performance, but has full NVENC engine).

The R520 can get a bit noisy, and although i've had 0 problems with the CPU doing the transcoding, i'd advise getting a compatible GPU to offload that to.

Let me know what else you'd like to know.

 

[Kai Robinson]

So, I have grabbed a T430 from the e-waste bin with the view of replacing the R520...unfortunately, it looks like at least for the moment, the T430 is dead, with a 'CPU1 M23 VLT0204' and 'VLT0304' error message, regardless of the CPU in the socket.

Oh well, i'll get around to spending some more time with it eventually, so that'll be sidelined...for now.

However, one of the things that was purchased for the T430 project will be used, or at least attempted for use, in the existing R520:

An nVidia Quadro P620!

Think of it as a wierd in-between model, sitting in the middle of the GT1030 and the GTX1050 - a GT1040 if you will. Based on the same Pascal Architecture, the GP107 in this has a full NVENC block, and this is precisely the intended use - not for rendering or graphics, but for video transcoding.

The P620 can handle, for encoding:

...and for decoding:

It lacks AV-1 support but honestly, I have zero AV-1 content, so de-nada!

What's really cool is that it's a single slot card - uses 47watts max, can handle 8 concurrent encodes although it's more limited by the amount of VRAM on the card.

Judging from the *extremely cool* calculator here: https://www.elpamsoft.com/?p=Plex-Hardware-Transcoding

this means the card can transcode TWO simultaneous 4K streams on PleX. And for 1080p? It's not even going to break a sweat, given that the most load i've had at once is three concurrent streams of mostly 1080p content.

The fun part though? Getting passthrough working so the PleX VM can use it!

 

[Kai Robinson]

Things have taken a bit of a swerve - I never did get around to installing the P620 in the R520.

One thing that's happened recently, is thanks to my work, from the legendary E-Waste bins, I picked up a Synology Rackstation RS3617xs+ - a 12 bay NAS with 10GbE (albeit RJ45). On top of that, EIGHT 10TB Seagate IRONWOLF SATA NAS disks were also rescued (I love decommissioned DC stuff!)

So what did I end up doing?

MAKING A MEGA STORAGE ARRAY!

I added the additional four 6TB WD RED SATA disks to the bottom row of the Rackstation - to create a separate array solely for Veeam Backups.

D: is the 'old' R520 RAID5 array, E: and F: are iSCSI volumes on the Rackstation - I think I have in total, roughly 100TB of storage, at least for now.

So the plan? Well, DIMM A6 in the R520 is throwing an error and power is becoming a real issue, in terms of the cost constantly rising here - so....medium term plan is to decommission it. It's done good work, but it's time to downsize a bit, now I have the Rackstation. Since the only two VM's i'm currently serving are the TD Game Server (2 Cores, 8GB RAM, Windows XP) and my PleX Server for family and friends, I don't really need 192GB RAM, 20 Cores and 40 threads of Ivy Bridge era Xeon.

Ideally, i'd like to replace this with a Lenovo ThinkCentre Tiny M920Q. The 8th/9th Gen CPU's it supports can do HARDWARE transcode in plex without the need for me to add a Quadro P620, and they, unlike other USFF 1L PC's, also include the ability to add a PCIe card (albeit low profile, single slot). What'll go in the slot? Another Mellanox ConnectX-3, albeit a dual SFP+ card instead of the single ones I have now.

As 10Gb Switches are still mad priced, it'll let me move the Mellanox ConnectX-3 from the decommissioned Dell R520 to the Synology, and use the M920Q as a switch of sorts, allowing my desktop to shove data to the Synology super fast. 1Gbps Ethernet just isn't fast enough when you're dealing with multiple Terabytes of files at a time - seriously, migrating stuff from the R520's SAS array to the Synology took FOREVER.

Now looking into it, the M920Q chassis is capable of OoB management through it's 1Gbps Ethernet jack as well. It has support for TWO M.2 nVME drives so one boot volume and one for VM storage. RAM wise, 32GB should be enough - 8GB is probably overkill for the XP game server and PleX is fine with probably 12GB RAM. Now, the big thing here, is i'll be ditching Windows Server 2019 for....Proxmox.

I've not dealt with it at all before, but, I need to learn and expand my skillset. I've already read up that moving the VHDX over is pretty simple, as is configuring it and passing through the Intel iGPU for Quicksync.

All in all, i'm hoping to get a server solution that sits on my desk and doesn't sound so noisy when PleX software transcodes are running!