- Putting older hardware to work
- Colony West Project
- Adding the GT 620
- Follow-up on ASRock BTC Pro and other options
- More proof of concept
- Adding a GTX 680 and water cooling
- Finalizing the graphics host
- ASRock BTC Pro Kit
- No cabinet, yet…
- Finally a cabinet!… frame
- Rack water cooling
- Rack water cooling, cont.
- Follow-up on Colony West
- Revisiting bottlenecking, or why most don’t use the term correctly
- Revisiting the colony
- Running Folding@Home headless on Fedora Server
- Rack 2U GPU compute node
- OpenVPN on Docker
- Proxmox VE vs VMware vSphere
- Migrating a Plex Media Server to another system
- Volunteer distributed computing is important
EK got back to me pretty quickly on my inquiry. The screws in question that I’d need are M3x10 DIN 7991. They even offered to mail me a set if I couldn’t find them locally. Grainger says they carry M3x10 DIN 7991, but the screws they sell are actually longer than 10mm, so they wouldn’t work for this. Thankfully McMaster-Carr also carries them, so I was able to have them shipped in. Unfortunately I had to order 100 of them when I only need 16. At least the excess dowel pins I ordered can be put to use. Not sure what I’m going to do with these.
On cleaning the blocks, after a lot of research, and posting a thread to the Linus Tech Tips forum, I opted toward Mother’s California Gold Metal Polish. The difference is night and day, and the blocks look almost brand new.
Getting into the channel over the VRMs and in the fins over the GPU was interesting. And to get around the edge of the channels, I used a blue shop paper towel as it was easier to push into the corner than the microfiber towels I bought with the polish. For the fins over the GPU, just used a toothbrush.
But like I said, like night and day.
After seeing the EK blocks, though, I decided to also take apart and polish the Koolance blocks. Definitely glad I did, too, even though it meant having to re-flush the blocks. The liquid shown in the picture is distilled water, not old coolant.
Then came assembling everything. The GTX 660s went without a hitch. The GTX 680s, however, had a couple minor hiccups. On the Zotac card, there was a small foam block that was used to keep the massive cooler suspended over the card. It interfered with the Koolance block, but once I removed it, it wasn’t a problem.
For the EVGA card, I needed to pick up some M3 nuts from Home Depot along with finding spare PVC washers or a substitute for them. The reason is the Koolance VID-NX680 doesn’t come with a separate screw and nut for the back panel, while the EK-FC680 does. That extra screw wasn’t needed for the Zotac card since it didn’t use a reference cooler, but since the EVGA GTX 680 does use a reference blower, it needs the extra screw for the back panel.
A quick glance at the instruction manual for EK’s block showed what I needed to buy. I already had ¼” M3 screws — plenty of them — but I just needed a nut for securing it.
Unfortunately I would come to that realization well beyond Home Depot’s closing time. Oh well. As Gosalyn Mallard says in Darkwing Duck, don’t let a minor setback stop you. Unfortunately there was a more major setback.
For the graphics cards I intend to run them in a split parallel configuration. The GTX 660s will be in parallel, and the GTX 680s will be in parallel with a single line coming from the outlet on the GTX 660s going to the GTX 680s. And I want the GTX 680s to be right up against each other, and same for the GTX 660s.
I have the SLI fittings for the GTX 680s. I’m using Swiftech fittings for that. For the GTX 660s, I have an EK SLI bridge, but it’s a dual serial 3-slot bridge. So I decided to try to buy one from Micro Center, but it was an SLI Terminal I purchased, not the SLI Bridge. I should’ve paid a little more attention as now that means I have something I need to order in and more delays on this project.
But again, don’t let a setback stop you where you can. This isn’t a show stopper. There is still other progress to make in the mean time, and a receipt to find. Before closing out the night I attached fittings to the bulkheads: the male quick disconnects to the outside and compression fittings to the inside.
* * * * *
Sunday started off with a trip to Home Depot to obtain some more small parts. I needed #6-32 screws, ½” length, some 00 rubber washers, and M3 nuts (mentioned previously). The 00 washers and screws were for mounting the pump to the radiator — the 00 washers act as vibration isolation and work quite well.
At home, I turned my attention first to the EVGA GTX 680 to get the bracket screwed to the card, then turned to doing test fits. Unfortunately having the GTX 660s all the way at the back corner of the case would not be feasible. The mainboard standoffs stick up too far and aren’t removable from what I can tell. So I had to have them one slot over.
I turned my attention then to trying to get some of the cable management out of the way. The USB 3.0 panel mount cables, however, I should never have purchased. At least not the ones I did buy. So I placed an order for the 12″ cables as I think those will be much, much easier to cable manage. I ordered the 3′ cables as I didn’t think the 12″ cables would be long enough, and there wasn’t anything in between.
So basically until orders arrive later in the week, there isn’t much farther I can go on this, at least with the chassis. The radiators on the other hand…
* * * * *
The pump and reservoir were the easy part. Getting everything back onto the rack was a little interesting. But once I had everything on the rack, I started to tube up where I could. Which wasn’t much initially. I used EK 90-degree fittings plus Swiftech 3/8″x1/2″ fittings to run tubing between the radiators. And I ran a tube from the top radiator back to the top reservoir. That was about all I could accomplish initially.
The order arrived a few days later with the EK SLI bridge and the extra fittings. With these, I finished tubing up the graphics cards, mostly. After getting the SLI bridge mounted to the GTX 660s, I saw how close the fittings for the GTX 660s and GTX 680s would be.
I measured this gap at 2-3/16″, with about a 2½” offset. But I didn’t want to order more fittings, so I looked at what I still had. Leftover from moving β Ori.’s pump and reservoir external, I had a Koolance 4-way fitting and a male-to-male rotary fitting. This served as a make-shift 90-degree fitting and I just looped tubing around to reach the SLI Bridge. Sometimes you need to improvise. With that tubed up, I turned to the leak test.
I leak tested the graphics chassis in isolation simply because it was going to be much easier. Initially there was a pretty significant leak: I forgot to tighten down one of the stop fittings on a Koolance block. Thankfully that was the only leak.
Back to the rack and the radiators, or rather the pump and reservoir mounted to them. Between the pump and reservoir I opted to use 1/2″x3/4″ tubing just to avoid having to buy still more fittings to finish this. I already have such fittings with some tubing left over, so it was trivial, or nearly so.
The Swiftech 1/2″x3/4″ fitting wouldn’t sit flush and I didn’t want to go out and buy another extension fitting, so I used a T-fitting plus a male-to-male fitting to get things run. The tubing then curves down to meet another fitting on the pump. The outlet on the pump feeds to the quick disconnects. The outlet on the graphics chassis also goes through quick disconnects to get to the lower radiator. The top-most fitting on the top radiator goes back to the reservoir.
This actually made bleeding the system a little interesting. The reservoir is entirely bottom-fed, meaning the water blocking the inlet to the reservoir was going to also make it difficult to get air out of the system. It took a few tries to actually get fluid moving. But once I did, it didn’t take much to bleed it. With that bleeding, I turned my attention to getting the graphics host and my wife’s gaming server back onto the rack and getting the cabling managed.
And that was basically it. Unfortunately I’m having some stability issues, but I think I’ll just need to rebuild the operating system to get that figured out. But with Fedora 23 due out in a few days as of when I write this, I’ll just put up with the stability concerns until then.
I’ve got a couple other minor enhancements to make to this after doing some experimentation, primarily involved around getting the tachometer for the pump plugged into the graphics host and synchronizing the power supplies between the host and the GPU chassis.
But before I go for now, let’s talk about temperatures. To show that these are temperatures under a full load from the Berkeley client and not idle temperatures, I made sure to show both in the same image. These are two GTX 680s with Koolance VID-NX680 blocks and two GTX 660s with EK-FC680 blocks and two XS-PC EX360 radiators with Cougar Vortex HDB fans.
Right-click on the image and select “View image” to see it full-size. Given these temperatures I think I’ll see what they look like with the fans turned down. Stay tuned…