10Gb network upgrade

Goodbye, MikroTik

Posted on by Kenneth Ballard / 0 Comment
Build Log:

Four (4) years ago, I bought the MikroTik CRS317 after seeing it retailing for… around 300 USD. And it’s a great switch, so long as you use it as a switch. Later I also acquired a MikroTik CSS610. And I recently replaced both for switches from the TP-Link lineup.

A couple details pushed me to do this, but it mostly stems from the upgrade to Google Fiber’s 5Gb Internet service. Other choices made here were also about consolidating. Prior to this changeover, I had three switches in my network rack:

  • MikroTik CRS317
  • MikroTik CSS610
  • BV-Tech POE-SW801

And I consolidated to two switches, both from TP-Link and manageable via the Omada Controller software:

TP-Link TL-SG3210XHP-M2

This switch replaced two in my network rack: the MikroTik CSS610 and BV-Tech POE-SW801. The latter is an 8-port 10/100 POE switch with a 100Mbps uplink. I bought it to support the security cameras I have, but never used all the ports on it.

The TP-Link TL-SG3210XHP-M2 has eight (8) 2.5GbE ports, all of which are Active POE+ enabled. This allows me to consolidate my security cameras and the TP-Link EAP670 wireless access point. Previously I had the EAP670 connected to the CRS317 via a TP-Link 10GbE RJ45 module and powered using its included DC power supply.

And being Active POE+ allows me to consolidate the couple Gigabit connections from the CSS610, putting me in a position to upgrade those connections to 2.5GbE. Since 2.5GbE runs across Cat5E without issue, it’s a drop-in upgrade.

One has already been upgraded as of this writing, that being the connection to my work laptop. My wife’s laptop is the other connection that will be upgraded.

I’ve considered swapping out the Gigabit switch on my living room rack for a 2.5GbE switch, but what’s connected to it isn’t really making me all that enthused about doing it. For the curious, all that’s connected is the uninterruptible power supply, mail server, and IP-KVM. And the only one that’s even slightly bandwidth intense is the latter, but it doesn’t saturate a GbE connection.

So this switch, though, means the wireless access point is powered from the TP-Link switch, removing a connection on the CRS317. My security cameras are as well, and if I expand my security camera setup more, I’ll add one of TP-Link’s Gigabit POE+ switches to the rack.

I’ve already swapped the stock fans with Noctua NF-A4x20 FLX fans as well. Omada is reporting a fan fault with it, but that’s merely due to the RPM of the Noctua fans being far lower compared to the stock fans. But that also means it’s practically inaudible from my office.

Note: I’m aware of a lot of negative reviews on this switch that indicate an… oddly short lifespan. And I’ll definitely be posting an update if this switch dies sooner rather than later. (Given the fan swap, I doubt TP-Link will honor the warranty.)

TP-Link TL-SX3008F

The CRS317 has 16 SFP+ ports. This is overkill for my home network. At the time I bought it, though, my only other option for reasonably-priced 10GbE was the MikroTik CRS305, which has only 4 SFP+ ports plus a Gigabit uplink port.

MikroTik wouldn’t introduce the 8-port CRS309 till later in 2019. And there really was no point in changing over to it at the time. After adding the aforementioned 2.5GbE switch to my rack, removing the connection for the EAP670, only six (6) ports were being used:

  1. Mira
  2. Amethyst
  3. Nasira
  4. Virtualization server
  5. Uplink from TL-SG3210XHP-M2
  6. Uplink to router

But most of the ports sitting dormant isn’t my reason for changing this out. Performance is the main reason here. The issue is either the 98DX8216 controller, or it is MikroTik’s SwitchOS.

Either way, my 5Gb Internet connection revealed the MikroTik’s limitations when it comes to its switching functions. After putting the TP-Link switch into service, I was finally able to get 5Gb from my desktop with other clients also connected to the switch.

And other random speed tests I’ve done since taking that screenshot have produced similar results. Even one taken in the middle of a Saturday afternoon.

So the MikroTik’s switching capability is a massive bottleneck. I could easily do 10Gb to my NAS, such as when I’m syncing my camera card dump folder, but I’m typically the only one accessing it at any given time. But the speed test screenshot above shows that it chokes off when multiple clients are trying to tunnel through a single port – such as the one linking up to the router.

Now don’t get me wrong. The MikroTik CRS317 is a decent switch. And it was an inexpensive way to get 10GbE in a quiet package – especially if you change out the fans.

Its initial MSRP was about 400 USD, but you could easily find it for less. But MikroTik bumped that to 500 USD, with a lot of sellers making that the shelf price, making it difficult to recommend this switch when there are better options available at the same price point. Just as an immediate example, TP-Link has their own fanless 16-port 10GbE SFP+ switch for about 500 USD (as of this writing) – the TL-SX3016F – that, if similar to the TL-SX3008F, is likely to also perform much better.

There may be tweaks you can make in RouterOS – configuring it in “bridge mode” – to allow it to perform better. But the TP-Link switch is performing better out-of-the-box.

MikroTik started the trend of bringing 10GbE to the home lab in an inexpensive package that was also very quiet. Both with the CRS305 and CRS317. But competition at MikroTik’s price point revealed its weaknesses. To stay competitive, MikroTik should consider releasing a new switch to replace the CRS317 that performs much better.

Life/Logic/Politics

“Flesh Cult of Carnism”

Posted on by Kenneth Ballard / 0 Comment

“Flesh Cult of Carnism”? Tell me you’ve lost sight of reality without telling me you’ve lost sight of reality.

In all seriousness, manufacturing phrases like that means you really need to take a few miles worth of steps back and re-evaluate your psychological standing. This shows you’re so deep in your ideology, the Mariana’s Trench is like a crack in the pavement. Being vegan is one thing. But manufacturing phrases like this and posting propaganda like this to the Internet is more about re-justifying to yourself the choice to be vegan and shows a massive choice-supportive bias that is well beyond the point of delusion.

Get help.

OPNsense router

It’s final (for now) form

Posted on by Kenneth Ballard / 0 Comment
Build Log:

New specs:

The chassis and power supply were the previous of each in Mira before I upgraded to the beQuiet Dark Base 900 and EVGA 1000 G6. The former was to get more space for HDDs. The latter was because I upgraded to the RTX 3070 and needed additional PCI-E connectors but couldn’t find my original cable kit. The power supply is overkill for this use case, but at least it’s being put to use again.

I chose the Machinist mainboard after learning about it through the Craft Computing YouTube channel. Specifically the below video, which also made my decision on the CPU.

The mainboard supports everything you’d expect from an X99 mainboard, including “Above 4G decoding”. It does NOT support bifurcation, at least that I could find, but you might be able to mod the BIOS to include that – at your own risk, of course. So I’m not going to be buying another one of these for my NAS unless I need more than 64GB RAM for some reason.

But I will eventually buy one for upgrading my virtualization machine later this year since it supports up to 256GB RAM (8x32GB) and the E5-2699v4 (22 cores/44 threads), which will make for one hell of a home lab. I might even consolidate the NAS and virtualization together to one box. Not using TrueNAS SCALE’s virtualization, but putting TrueNAS in a VM, which would require going back to Proxmox since VirtualBox does not support PCI passthrough with version 7.

As mentioned previously, using ECC RAM isn’t required here, but it’ll help merely because of how much bandwidth will be going through this. Plus the price (at the time I bought it) was only 10 USD per stick on eBay brand new. So… why NOT use it?

So for now, this is the form this router will take. It’ll be interesting to see how long this will last, and I anticipate hopefully not needing to do any hardware changes on this unless there is some incompatibility with OPNsense. Which shouldn’t happen unless the FreeBSD developers go off their rocker and start removing support for older hardware from their operating system.

* * * * *

Update (2023-03-31):

Consider this kind of a lateral move. I pulled the MR9S board for the Machinist MR9A Pro (buy it on eBay). It’s a slightly smaller motherboard with only four (4) DDR4 slots instead of 8, but still able to operate in quad-channel. The MR9S will be going into my virtualization server, so keep an eye out for that.

OPNsense router

The road to 5Gb

Posted on by Kenneth Ballard / 0 Comment
Build Log:

In the mention of my custom router, I talked about how Google Fiber would soon be introducing 5Gb and 8Gb service. Recently I was upgraded to the 5Gb service and… let’s just say it isn’t what I expected.

I had a feeling this would be the outcome as well.

The service overall felt snappier compared to the 2Gb service. But maintaining line speeds above 2Gb or 3Gb was proving difficult, even during off-peak hours. Running a speed test from the router using the SpeedTest CLI demonstrated this. Upload speeds would have no issue breaking 4Gb or even 5Gb, but download speeds would typically max out at 3Gb.

So what gives? In short, it’s the router itself. I just don’t think the APU can keep up with the demand. It has no issue keeping up with 2Gb service or less. But beyond 2Gb it becomes inconsistent.

But I’m not about to switch back to using Google’s router. For one that would require adding back in the 10GbE RJ45 SFP+ module, which runs hot, and the active cooling to go with it. Or using a media converter.

So instead, I need to upgrade my custom router. The big question is what platform to jump to: 990FX or X99? Now reading that question, you’re probably already shouting “How is that even up for debate?”

Current specs

Before going too far, here’s what I’m starting with.

CPU:AMD A8-7600 APU with Noctua NH-D9L
Mainboard:Gigabyte GA-F2A88X-D3HP
RAM:16GB DDR3-1600
PSU:EVGA 650 G2
Storage:Inland Professional 128GB 2.5″ SATA SSD
WAN NIC:10Gtek X540-10G-1T-X8 10GbE RJ45
LAN NIC:Mellanox ConnectX-2 10GbE SFP+
Chassis:Silverstone GD09
Operating system:OPNsense (with latest updates as of this writing)

Which path forward?

In a previous article about doing a platform upgrade on Nasira, I mentioned I have a 990FXA-UD3 mainboard from Gigabyte. Talking specifically about how it does its PCI-E lane assignments before revealing, ultimately, that I went with a spare X99 board for Nasira due to memory prices. And that gave the benefit of PCI-E 3.0 as well, which was important for the NVMe drive I was using as an SLOG.

For a router, PCI-Express 3.0 isn’t nearly as important so long as you pay attention to lane assignments. Though for a gigabit router, even that doesn’t matter much. Both cards were on at least 4-lanes and running at their full speeds – 5.0GT/s for both.

So if lanes aren’t the problem, that leaves the memory or processor. And there isn’t much benefit to bumping from DDR3-1600 to DDR3-1866 for this use case. The memory just isn’t going to make that much of a difference here since the memory already provides more than enough bandwidth to handle this use case.

So that leaves the processor.

990FX with FX-8320E

Compared to even an FX-8320E, the AMD A8-7600 APU is underpowered. The onboard GPU is the only benefit in this use case. The FX-8320E doesn’t provide much of a bump on clocks, starting out at 3.2GHz but boosting to 4GHz. Performance metrics put the FX-8320E as the better CPU by a significant margin. The FX-8350 would be better still, but not by much over the FX-8320E.

So while it’s the better CPU and platform on paper compared to the APU and the A88X chipset, is it enough to serve as a 5Gb router?

Well I didn’t try that. I decided to jump to the other spare X99 board instead.

Or, rather, the X99 with i7-5820k

So again you were probably asking why I was even considering the 990FX to begin with? And it’s simply because I had one lying around not being used. Specifically the Sabertooth 990FX from Nasira still assembled with its 32GB DDR3-1600 ECC, FX-8350, and 92mm Noctua cooler. And I actually have a few 990FX boards not being used.

But I also had the Sabertooth X99 board that was in Mira still mostly assembled. It hadn’t been used in a while and just never torn down, so it was relatively easy to migrate for this.

So why the leap to the X99 over the 990FX? In short, it’s the specifications for the official pfSense and OPNsense appliances.

The Netgate 1537 and 1541 on the pfSense front are built using the Xeon-D D-1537 and D-1541, respectively, which are 8-core/16-thread processors, and DDR4 RAM. Both are rated for over 18Gb throughput.

And OPNsense’s appliances use either quad-core or better AMD Ryzen Embedded or Epyc processors. The DEC740 uses a 4-core/8-thread Ryzen with only 4GB DDR4, while the slightly better DEC750 doubles the RAM. Both are rated for 10Gb throughput.

But their DEC695 has a 4-core/4-thread AMD G-series processor and DDR3 RAM, and is rated for only 3.3Gb of throughput. Hmm… that sounds very familiar…

Quad-channel memory is where the X99 platform wins out, compared to dual-channel support for the aforementioned Ryzen and Xeon CPUs. But to get started, I ran with dual-channel since two sticks of DDR4-3200 is all I had available at the moment. If everything worked out, that would be replaced with 4x4GB for quad-channel RAM and a Xeon E5-2667 v4, which should yield overkill performance.

Tell someone this is your router, and they likely won’t believe you.

Here’s the temporary specs:

CPU:Intel i7-5820k with NZXT Kraken M22
Mainboard:ASUS Sabertooth X99
Memory:16GB (2x8GB) DDR4-3200 running at XMP

Side note: I was able to move the SSD onto the new platform without having to reinstall OPNsense. It booted without issue. I still backed up the configuration before starting just. in. case.

So was this able to more consistently sustain 5Gb? Oh yeah!

One rather odd thing I noticed with the speed test, both on the old and new router setups: when trying to speed test against Google Fiber’s server, it capped out at 2Gb. But in talking to the Misaka Network server, shown in the screenshot, it now consistently gets 5Gb at the router.

Note: The command-line tool allows you to specify a server to test against. So going forward with my speed testing from the router, I’ll need to remember that.

With the AMD APU, it wasn’t getting close. And the FX-8320E or FX-8350 on the 990FX probably would’ve done better, but clearly it was best that I jumped right to the X99 board.

So what does this mean going forward?

Road forward

So with the outstanding test results, this will be getting a few hardware changes.

The CPU and memory are the major ones, and the mainboard will also get changed out. Something about either the processor or mainboard isn’t working right, and none of the memory slots to the left of the CPU are working – as the image above shows. This tells me it’s the mainboard, the CPU socket specifically (e.g. bent pins), but it could be the CPU as well.

Either way it means I can’t run quad-channel memory. And while the above speed test shows that quad-channel memory isn’t needed, I’d still rather have it, honestly.

But I have an X99 mainboard and Xeon processor on the way which will become the new router. Quad-channel memory is the more important detail here since Xeons do not support XMP. That does mean saving money on the memory, though, since DDR4-2400 is less expensive.

The Xeon on the way is the aforementioned E5-2667 v4. That’s a 40-lane CPU with 8-cores and 16-threads. Definitely overkill and I’m not going to see any performance improvement compared to the i7-5820k. As mentioned, it does not support XMP, so the fastest RAM I’ll be able to run is DDR4-2400. But in quad-channel.

The Xeon does also allow me to use ECC RAM, and the mainboard that is on the way supports it. While the router chugs along perfectly fine with non-ECC RAM, ECC is just going to be better given the much higher bandwidth this router needs to support.

Scams

Blackmail variant – 2023-02-25

Posted on by Kenneth Ballard / 0 Comment

Same scam, different cover.

Also the vulnerability they’re referencing can’t be used in the fashion they are stating. It’s a security vulnerability in Cisco’s browser-based management interface on their switches and routers.

Dear user of kennethballard.com!

I am a spyware software developer.
Your account has been hacked by me couple months ago.

The hacking was carried out using a hardware vulnerability through which you went online (Cisco router, vulnerability CVE-2023-20026).

I went around the security system in the router, installed an exploit there.
When you went online, my exploit downloaded my malicious code (rootkit) to your device.
This is driver software, I constantly updated it, so your antivirus is silent all time.

Since then I have been following you (I can connect to your device via the VNC protocol).
That is, I can see absolutely everything that you do, view and download your files and any data to yourself.
I also have access to the camera on your device, and I periodically take photos and videos with you.

At the moment, I have harvested a solid dirt... on you...
I saved all your email and chats from your messangers. I also saved the entire history of the sites you visit.

I note that it is useless to change the passwords. My malware update passwords from your accounts every times.

I know what you like hard funs (adult sites).
Oh, yes .. I'm know your secret life, which you are hiding from everyone.
Oh my God, what are your like... I saw THIS ... Oh, you dirty naughty person ... :)

I took photos and videos of your most passionate funs with adult content, and synchronized them in real time with the image of your camera.
Believe it turned out very high quality!

So, to the business!
I'm sure you don't want to show these files and visiting history to all your contacts.

Transfer $900 to my Bitcoin cryptocurrency wallet: 1A9A9LAo MRUu9xBMY WgutcKVWi ciuWkw6Z
Just copy and paste the wallet number when transferring.

An important notice: I have specified my Bitcoin wallet with spaces, hence once you carry out a transfer, 
please make sure that you key-in my bitcoin address without spaces to be sure that your funds successfully reach my wallet!

If you do not know how to do this - ask Google.

My system automatically recognizes the translation.
As soon as the specified amount is received, all your data will be destroyed from my server, and the rootkit will be automatically removed from your system.
Do not worry, I really will delete everything, since I am 'working' with many people who have fallen into your position.
You will only have to inform your provider about the vulnerabilities in the router so that other hackers will not use it.

Since opening this letter you have 48 hours.
If funds not will be received, after the specified time has elapsed, the disk of your device will be formatted,
and from my server will automatically send email and sms to all your contacts with compromising material.

P.S. Do not try to contact me (this is impossible, sender's address was randomly generated).

I advise you to remain prudent and not engage in nonsense (all files on my server).

Good luck!

Other wallets associated with this scam:

Nasira

Throwing a short pass

Posted on by Kenneth Ballard / 0 Comment
Build Log:

In the previous iteration, I mentioned my intent to add more NVMe drives to Nasira. Right now there is only one that is being used as an SLOG, which I’m debating on removing. But the desire to add more is so I can create a metadata vdev.

Unfortunately doing that with the Sabertooth 990FX mainboard currently in Nasira is going to be more trouble than it’s worth. So to find something easier to work with, I considered ordering in a Gigabyte GA-990FXA-UD5 through eBay. But I realized I had a GA-990FXA-UD3 lying around unused. So I did some research into whether that would suit my needs.

And it looks like it will.

What’s the issue?

First, let’s discuss what’s going on here.

With the AMD FX processors, the chipset controlled the PCI-E lanes, not the CPU. This was a significant difference between AMD and Intel at the time. Though the CPU now controls the PCI-E lanes and lane counts with Ryzen.

And the 990FX chipset has 42 PCI-E lanes. This surpasses the lane count available on any Intel desktop processor at the time. The Intel i7-5960X had 40 lanes. Only Intel’s Xeon surpassed it, and only if you used more than one of them.

How they were divvied up between slots was up to the motherboard manufacturers, but generally every 990FX board gave you two (2) x16 slots so you could use Crossfire or SLI. What you could run and at what speed it ran depended heavily on the mainboard, since the mainboard determined lane assignments to slots. I’ve previously discussed how the Sabertooth 990FX assigns PCI-E lanes, showing the counter-intuitive chart from the user manual, so now let’s look at the Gigabyte lineup.

Gigabyte released three 990FX board models (with several revisions thereto) as part of their “Ultra Durable” lineup: the GA-990FXA-UD3, -UD5, and -UD7. And each has different lane assignments. The -UD7 is easily the most flexible, guaranteeing four (4) full-length slots at x8 or better. The UD5 guaranteed three (3) slots at x8 or better.

The -UD3 is a little different. That board also has 6 PCI-E slots: 2 x16, 2 x4, and 2 x1. And unlike the -UD5 and -UD7, the -UD3 does not share lanes between any of the slots or onboard features. Each slot has its own dedicated lanes. What you see is what you get. Or, at least, that is what the specifications heavily imply.

Why does this matter?

Obviously lane counts matter when you’re talking about high-bandwidth devices. You shouldn’t just randomly insert cards into slots without paying attention to how many lanes it’ll receive.

While any PCI-E device can operate on as little as just one lane – something anyone familiar with crypto-mining can attest – you definitely want to give bandwidth-critical devices all the lanes they require. SAS cards. 10GbE NICs. NVMe SSDs. You know, the hardware in Nasira.

So when the NVMe SSD I installed as an SLOG reported up that it had only a x1 link, I needed to swap slots to get it running at a full x4. The Sabertooth 990FX divvies up its PCI-E lanes in a very counter-intuitive way, leading me to believe the NVMe drive would have its needed 4 lanes in the furthest-out slot where I wanted to run it. And it turned out that wasn’t the case.

Had I swapped out the board sooner for the -UD3 I have on hand (it wasn’t available when I initially built Nasira), I wouldn’t have run into that issue.

That this was all on a 990FX mainboard is immaterial. Indeed the issue is more acute on many Intel mainboards unless you’re running one of the Extreme-edition processors or a Xeon due to PCI-E lane count limitations.

And many mainboards have a mix of PCI-E versions, so you need to pay attention to that as well to avoid, for example, a PCI-E 3.0 card being choked off by PCI-E 2.0 speeds. This is why many older 10GbE NICs are PCI-E 2.0×8 cards. PCI-E 2.0×4 has just enough bandwidth for two (2) 10GbE ports, but 1.0×8 really has enough bandwidth for only one (1). While PCI-E 1.0×8 should, on paper, allow for dual 10GbE ports, in practice you won’t see that saturated on such PCI-E 1.0 mainboards.

And 3.0 x4 10GbE NICs, such as the Mellanox ConnectX-3 MCX311A, will run fine in a 2.0 x4 slot – such as the slots in my virtualization server and the X470 mainboard in Mira. And I think it’s only a matter of time before we see PCI-E 4.0×1 10GbE NICs, though they’ll more likely be PCI-E 4.0×2 or x4 cards to allow them to be used in 3.0 slots.

Thermals is the other consideration. You typically want breathing room around your cards for heat to dissipate and fans to work. SAS cards can run hot, so much so that I wanted to add a fan to the one in Nasira after realizing how to add one to the 10GbE NICs in my OPNsense router. And even for 10mm fans, I need at least one slot space available to give room for the fan and airflow.

So with all of that in mind, I swapped out the Sabertooth 990FX board for the ASUS X99-PRO/USB 3.1.

Wait, hang on a sec…

So after initially jettisoning the idea of a platform upgrade, why am I doing a platform upgrade? In short… memory prices right now. I was able to grab 64GB of DDR4-3200 RAM from Micro Center for about 200 USD (plus tax) – about 48 USD for each 2x8GB kit. Double the memory, plus quad-channel.

And PCI-E 3.0. That was the detail that pushed me to upgrade after looking at the PCI-E lane assignments with the 5820k, which is a 28-lane CPU. Fewer lanes compared to the 990FX, but still enough for the planned NVMe upgrade. (4 lanes to the 10GbE NIC, 8 to the SAS card, 16 to the NVMe carrier card.) While upgrading to the 5960X is an option to get more PCI-E lanes – they’re going for around 50USD on eBay as of when I write this – it isn’t something I anticipate needing unless I upgrade the SAS card.

It’s also kind of poetic that it’s my wife’s X99 mainboard and i7-5820k that will be the platform upgrade for Nasira. Since acquiring that board and processor freed up her Sabertooth 990FX and FX-8350 to build Nasira in the first place.

Performance

So how does the new platform perform compared to the old? Well this probably speaks for itself:

That is a multi-threaded robocopy of picture files from a WD SN750 1TB to one of the Samba shares on Nasira. That’s the first time I’ve ever seen near. full. 10GbE. saturation. That transfer rate is 1,025,054,911 bytes per second, which is about 997.6 megabytes per second. I never saw anything near that with the Sabertooth 990FX. Sure I got somewhat better performance after adding the SLOG, but it’s clear the platform was holding it back.

More and faster memory. Faster processor. PCI-E 3.0.

But… ECC….!!!

Hopefully by now the religious zealotry and doomsday catastrophizing around not using ECC with ZFS has died down. Or does it persist because everyone is copying and pasting the same posts from 2013? It seems a lot of people got a particular idea in their heads and just ran with it merely because it made them sound superior.

The move to the 5820k does mean moving to non-ECC RAM. And no, there isn’t nearly the risk to my pool that people think… I went with ECC initially merely because the price at the time wasn’t significantly more expensive than non-ECC, and the mainboard/processor combination I was using supported it.

And when I wrote the initial article introducing Nasira, I said to use ECC if you can. Here, though, I cannot. The X99 board in question doesn’t support ECC, and neither does the processor. And getting both plus the ECC DDR4 is not cheap. It’d require an X99 mainboard that supports it, plus a Xeon processor. Probably two Xeons depending on PCI-E lane counts and assignments. And as of when I write this, the memory alone would be over 50 USD per 8GB stick, whereas, again, the memory I acquired was under 50 USD per pair of 8GB sticks.

But, again, by now the risk of using non-ECC with ZFS has likely been demonstrated to have been well and truly overblown. Even Matt Ahrens, one of the initial devs behind the ZFS filesystem, said plainly there is nothing about ZFS that requires ECC RAM. So I’m not worried.

And if your response to this is along the lines of, “Don’t come crying when your pool is corrupted!”, kindly fuck off.

Because let’s be honest here for a moment, shall we? It’s been 7 years since I built Nasira. In that time, there have probably been thousands of others who’ve taken up a home NAS project using FreeNAS/TrueNAS and ZFS. With a lot of those likely also using non-ECC simply to avoid the expense needed to get a platform that supports ECC RAM along with the memory itself. A lot of them likely followed a similar story to how I first built out Nasira: platform upgrade that freed up a mainboad/processor, so decided to put it to use. Meaning desktop or gaming mainboard, desktop processor or APU, and non-ECC DDR3 or DDR4.

Now presuming a small percentage of those systems suffered pool corruption or failures, how many of those could be legitimately attributed to being purely because of non-ECC RAM with no other cause?

In all likelihood – and let’s, again, be completely honest here – it’s NEXT. TO. NONE. OF. THEM.

And with Nasira, if anything is going to cause data corruption, it’s likely to be the drive cables, power cables, or the 10+ year-old power supply frying something when it gives up the ghost. Which is why I’m looking to replace it later this year for the same reason as the other pair of 4TB hard drives: age.

Again, use quality parts. Use a UPS. Back up the critical stuff, preferably offsite.

Now that’s not to say there is no downside to not using ECC, as there is one: you’ll get quite a lot of checksum errors during scrubs.

Current specs and upgrade path

So with the upgrade, here are the current specifications.

CPU: Intel i7-5820k with Noctua NH-D9DX i4 3U cooler
RAM: 64GB (8x8GB) G-Skill Ripjaws V DDR4-3200 (running at XMP)
Mainboard: ASUS X99-PRO/USB 3.1
Power: Corsair CX750M green label
Boot drive: ADATA ISSS314 32GB
SLOG: HP EX900 Pro 256GB
HBA: LSI 9201-16i with Noctua NF-A4x10 FLX attached
NIC: Mellanox ConnectX-3 MCX311A-XCAT with 10GBASE-SR module

The vdevs are six (6) mirrored pairs totaling about 54TB.

Soon I will be adding a metadata vdev, which will be two NVMe mirrored drives on, likely, a Sonnet Fusion M.2 4×4 carrier card. The SLOG will be moved to this card as well. That card doesn’t require PCI-E bifurcation, unlike other NVMe expansion cards like the ASUS Hyper M.2 x16 and similar cards, since it uses a PLX chip. But that’s why the Sonnet Fusion card is also more expensive. (X99 mainboards almost always require a modded BIOS to support bifurcation.)

There’s also the SuperMicro AOC-SHG3-4M2P carrier card. But that is x8, compared to x16 for the Sonnet Fusion. And the manual says it may require bifurcation whereas, again, the Sonnet Fusion explicitly does not.

There are off-brand cards as well. And 10Gtek sells NVMe carrier cards as well that do or do not need bifurcation. Most of what you’ll find is x8, though. 10Gtek has a x16 card, but I can’t find it for sale anywhere. And I may opt for a x8 card over the Sonnet Fusion since overall performance is unlikely to completely saturate the x8 interface under typical use cases. And PCI-E 3.0×8 is far, far more bandwidth than can be saturated with even 10GbE.

So stay tuned for updates.

Pool corruption!

So in the course of this upgrade, I suffered pool corruption. Talk about bad timing on it as well since it happened pretty much as I was trying to get the new mainboard online with my ZFS pool attached to it. So was it the non-ECC RAM? Have I been wrong this entire time and will now repent to the overlords who proclaim that one must never use non-ECC RAM with ZFS?

Yeah, no.

Initially I thought it was a drive going bad. TrueNAS reported one of the Seagate 10TB drives experienced a hardware malfunction – not just an “unrecoverable read error” or something like that. A lot of read errors and a lot more write errors being reported in the TrueNAS UI. And various error messages were showing on the console screen as well with the drive marked as “FAULTED”.

Thankfully Micro Center had a couple 10TB drives on hand, so I was able to pick up a replacement. Only to find out the drive wasn’t the issue as the new drive showed the exact same errors. The problem? The drive cable harness. If only I’d thought to try that first.

Something about how I was pulling things apart and putting them back together damaged the cable. And that it affected only one of the drives on the harness was the confusing bit. I’m sure most seeing what I observed would’ve thought the same, that the drive was going instead of the cable harness.

Unfortunately the back and forth of trying to figure that out resulted in data corruption errors on the pool, but thankfully to files that I could rebuild or re-download from external sources or restore from a backup. An automatic second resilver on the drive, which started immediately after the first finished, saved me from needing to do that and corrected the data corruption issue. At the cost of another 16 hour wait to copy about 8TB of data, about the typical 2 hours per TB I’ve seen from 7200RPM drives. (5400RPM drives tend to go at 2.5 hours per TB.)

So lesson learned: if TrueNAS starts reporting all kinds of weird drive errors out of the blue, replace the drive cable harness first and see if that solves the problem.

On the plus side, I have a spare 10TB drive that I thought was dead. But it came at a cost I wouldn’t have had to spend if I was a bit more diligent in my troubleshooting. Again, lesson learned.

Since the resilver finished, the pool has been working just fine. Better, actually, than when it was attached to the AMD FX, though the cooling fan on the SAS card is probably helping there, too.

Scams

Blackmail variant – 2023-02-08

Posted on by Kenneth Ballard / 0 Comment

It’s actually kind of concerning that a lot of my content recently has been posting these scam emails. But that’s also a testament to how pervasive they’ve become.

Hey! 

As you can see, this is not a formal email, and unfortunately, it does not mean anything good for you. 
BUT do not despair, it is not critical. I am going to explain to you everything right now. 

I have access to your electronic devices, which are the part of the local network you regularly use. 
I have been tracking your activity for the last few months. 

How did that happen?
You visited some hacked adult websites with Exploit, and your device was exposed to my malicious software (I bought it in Darknet from specialists in this field).
This is a very complex software, operating as Trojan Horse. It updates regularly, and your antivirus can not detect it. 
The program has a keylogger; it can turn your camera and microphone on and off, send files and provide access to your local network. 

It took me some time to get access to the information from other devices, and as of now, 
I have all your contacts with conversations, info about your locations, what you like, your favourite websites, etc. 

Just recently, I came up with an awesome idea to create the video where you cum in one part of the screen, while the video was simultaneously playing on another screen. That was fun! 

Rest assured that I can easily send this video to all your contacts with a couple clicks, and I assume that you would like to prevent this scenario. 

With that in mind, here is my proposal: 
Transfer the amount equivalent to 1650 USD to my Bitcoin wallet, and I will forget about the entire thing. I will also delete all data and videos permanently. 

In my opinion, this is a somewhat modest price for my work. 
If you don't know how to use Bitcoins, search it in Bing or Google 'how can I purchase Bitcoins' or other stuff like that. 

My Bitcoin wallet: 18GaQVHRKASCZu9B9oH1gnHENfSHL352G8

You have 55 hours to reply and you should also bear the following in mind: 
It makes no sense to reply me - the address has been generated automatically.
It makes no sense to complain either, since the letter along with my Bitcoin wallet cannot be tracked. 
Everything has been orchestrated precisely. 

If I ever detect that you mentioned anything about this letter to anyone - the video will be immediately shared, and your contacts will be the first to receive it.
Following that, the video will be posted on the web!

Remember! Time tracking will start as soon as you open this email, I am monitoring this! 

Good luck and take it easy! It was just bad luck, next time please be careful.
Scams

Refund scam variant – 2023-02-02

Posted on by Kenneth Ballard / 0 Comment

Scams like this aren’t uncommon. Basically the idea here is to get you to call the number on the “invoice”, where they will then give you all kinds of fake information about “hackers” taking over your computer and bullshit like that. Like a hacker would hack into your system so they can purchase antivirus service for you using your own account.

But when you call in, they’ll have you install software so the scammers can actually take over your computer so they can scam you out of your money.

So yeah, when you see this email, just delete it. It isn’t from PayPal or Lifelock. In fact, the “PayPal” in the invoice is spelled “PayPaI”. The font hides the fact the last letter is a capital-I instead of a lower-L.

 

Tax Invoice

(818) 649-9458

 

 

 

Date: Feb 02, 2023

INVOICE#: PP01948838388373

 

Thank you for doing business with PayPaI.

 

 

 

 

 

 

 

 

 

 

DESCRIPTION

QTY

UNIT PRICE

TOTAL

 

Lifelock Antivirus

5pc

$199.00

$995.00

 

 

 

 

SUBTOTAL

$995.00

 

*Note to customer

We found some unusual login of your account from different locations. If you did not make this transaction, please reach out to the billing team 1(818) 649 9458, to cancel and claim a refund.

 

TAX RATE

0.00%

 

 

TOTAL TAX

0.00

 

 

SHIPPING/HANDLING

0.00

 

 

 

Balance Due

$995.00 –

 

 

 

 

 

 *Payment terms (due on receipt)

 

Scams

Blackmail variant – 2023-02-01

Posted on by Kenneth Ballard / 0 Comment

Hi Brian Black, thanks for the content! Yeah I know you don’t actually have access to my email account, and I know you don’t have a video. And hopefully with this article going live broadcasting your scam to the world, you’ll end up with zero BTC as well.

Hey, Kenneth

Your device was infected with my private malware, your browser wasn't updated / patched, in such case it's enough to just visit some website where my iframe is placed to get automatically infected, if you want to find out more, Google: Drive-by exploit.

My malware gave me full access to all your accounts (see the name above), full control over your device and it also was possible to spy on you over your webcam.

I collected all your private data and I RECORDED YOU (through your webcam) SATISFYING YOURSELF!

After that I removed my malware to not leave any traces and this email was sent from some hacked server.

I can publish the video of you and all your private data on the whole web, social networks, over email and send everything to all your contacts.

But you can stop me and only I can help you out in this situation.

The only way to stop me, is to pay exactly 750$ in bitcoin (BTC).

It's a very good offer, compared to all that horrible shit that will happen if I publish everything.

You can easily buy bitcoin here: www.paxful.com , www.coingate.com , www.coinbase.com , or check for bitcoin ATM near you, or Google for other exchanger.

You can send the bitcoin directly to my wallet, or create your own wallet first here: www.login.blockchain.com/en/#/signup?product=wallet , then receive and send to mine.

My bitcoin wallet is: bc1qtt5akr5undmey83wcnyns2jxdy7kqp7p5a7j54

Copy and paste my wallet, it's (cAsE-sEnSEtiVE).

I give you 4 days time to pay.

As I got access to this email account, I will know if this email has already been read.

AFTER RECEIVING THE PAYMENT, THE SPAM WILL STOP, I WILL REMOVE EVERYTHING AND YOU CAN LIFE YOUR LIVE IN PEACE LIKE BEFORE.

Next time make sure that your device got the newsest security updates.

ClientMailID: 0450021
Scams

Blackmail variant – 2023-01-31

Posted on by Kenneth Ballard / 0 Comment

And another one…

Remember, it’s stupid easy to spoof the from address in an email. And in my case, as I’ve said several times, I don’t have a webcam, so there’s no way for them to have created any kind of video. And looking at the source of the email, there is no separate Reply-To address.

Hi there!
Unfortunately, I need to start our conversation with bad news for you.
Around few months back I managed to get full access to all devices of yours, 
which are used by you on a daily basis to browse internet.

Afterwards, I could initiate monitoring and tracking of all your activities on the internet.

I am proud to share the sequence of how it happened: 
In the past I bought from hackers the access to various email accounts (today, that is rather a simple thing to do online).
Clearly, it was not hard at all for me to log in to your email account ([REACTED]).

A week after that, I had already managed to effortlessly install Trojan virus to Operating Systems of all devices that are currently in your use, and as result gained access to your email.
To be honest, that was not really difficult at all (because you were eagerly opening the links from your inbox emails). 
I know, I am a genius. ^^ 

With help of that software, I can gain access to all controllers in your devices (such as video camera, keyboard and microphone).
As result, I downloaded to my remote cloud servers all your personal data, photos and other information including web browsing history.
Likewise, I have complete access to all your social networks, messengers, chat history, emails, as well as contacts list.
My intelligent virus unceasingly refreshes its signatures (due to its driver-based nature), and hereby stays unnoticed by your antivirus software.

Herbey, I believe that now you finally start realizing how I could easily remain unnoticed all this while until this very letter...
While collecting information related to you, I had also unveiled that you are a true fan of porn sites.
You truly enjoy browsing through adult sites and watching horny vids, while playing your dirty solo games.

Bingo! I also recorded several filthy scenes with you in the main focus and montaged some dirty videos, which demonstrate your passionate masturbation and cum sessions.

In case you still don't believe me, all I need is just one-two mouse clicks to make all your unmasking videos become available to your friends, colleagues, and even relatives.
Well, if you still doubt me, I can easily make recorded videos of your orgasms become a public.
I truly believe that you surely would avoid that from happening, 
taking in consideration the type of the XXX videos you love watching, (you are clearly aware of what I mean) it will result in a huge disaster for you.

Well, there is still a way to settle this tricky situation in a peaceful manner:
You will need to transfer $960 USD to my account (refer to Bitcoin equivalent based on the exchange rate at the moment transfer), 
so once funds transfer is complete, I will straight away proceed with deleting all that dirty content from servers once and for all.

Afterwards, you can consider that we never met before. You have my honest word, that all the harmful software will also be deactivated and deleted from all your devices currently in use. 
Worry not, I keep my promises.
That is truly a win-win solution that comes at a relatively reduced cost, mostly knowing how much effort I spent on monitoring your profile and traffic for a considerably long time.
In event that you have no idea about means of buying and transferring bitcoins – don't hesitate to use any search engine for your assistance (e.g., Google, Yahoo, Bing, etc.).

My bitcoin wallet is as follows: 19yaJM8qhsyXnwoQP7zQbMkqJStoMYxPmE

I have allocated 48 hours for you to do that, and the timer started right after you opened this very email (2 days to be exact).

Don't even think of doing anything of the following:
*Abstain from attempting to reply me (this email was created by me inside your inbox page and the return address was generated accordingly).
*Abstain from attempting to get in touch with police or any other security services. Moreover, don't even think of sharing this to you friends. 
 Once I discover this (apparently, that is absolutely easy for me, taking in consideration that I have complete control over all systems you use) - your kinky video will straight away be made public. 
*Don't even think of attempting to find me – that is completely useless. Don't forget that all cryptocurrency transactions remain completely anonymous.
*Don't attempt reinstalling the OS on all your devices or getting rid of them. That won't lead you to success either, because I have already saved all videos at my remote servers as a backup.

Things you should not be concerned about:
*That your funds transfer won't reach my wallet.
- Worry not, I can see everything, hence after you finish the transfer, I will get a notification right away 
 (trojan virus of mine uses a remote-control feature, which functions similarly to TeamViewer).
*That I will still distribute your videos although you make the funds transfer.
- My word, I have no intention or interest in continuing making your life troublesome. 
 Anyway, If I truly wanted that, it would happen long time ago without me notifying you! 

Everything can be settled in a peaceful and just way!
And lastly... make sure you don't get caught afterwards in such type of incidents anymore!
My fair advice – ensure you change all your passwords on a regular basis.