154 lines
16 KiB
Markdown
154 lines
16 KiB
Markdown
+++
|
|
date = "2021-04-03T00:00:00-04:00"
|
|
tags = ["diy","homelab","buildlog"]
|
|
title = "A Custom Monitored PDU"
|
|
description = "Building a custom power monitoring PDU for fun and profit"
|
|
type = "post"
|
|
weight = 1
|
|
draft = true
|
|
+++
|
|
|
|
As a veteran homelabber, one thing that always comes up is power usage. Electricity is, unfortunately, not free, even if I do get half of it from "too cheap to meter" nuclear power here in Ontario. And servers can use a lot of it. Some systems provide on-demand power monitoring via their IPMI/BMC interfaces, but not all do. And figuring out how much power the other, non-intelligent, systems use can be a hassle.
|
|
|
|
The most obvious solution is what is normally called a "per-port monitored PDU". Typically used by colocation providers and large enterprises, these power distribution units (PDUs) allow the administrator to see the actual power usage out of each individual port at a given time, both for billing and monitoring purposes. They're the perfect solution to the problem of not knowing how much power you are using.
|
|
|
|
But these PDUs are not cheap. New, the cheapest ones I've been able to find run over $1400 USD, and they're gigantic 5-foot monsters designed for full-sized 42U+ racks. Add in dual circuits/UPSes, and the cost doubles. There has to be a better way.
|
|
|
|
Well, there is. With a decent amount of electrical know-how, some programming, 3D printing, and a lot of patience, I've been able to build myself several custom PDUs. Read on to know how!
|
|
|
|
**DISCLAIMER/WARNING:** While I am not a professional licensed electrician, I've spent a large portion of my life working with A/C electricity, pretty much from the time I could walk and hold a screwdriver, including a year as an Electrical associate at The Home Depot, as well as numerous home projects and two previous PDUs. I know what I'm doing. **Working with mains electricity is very dangerous, especially if you do not know what you're doing.** This post is provided as a curiosity for most, and a build log/guide only for those who are well-versed in working with this sort of thing. **Do not try this at home. I will not provide advice or guidance on any aspect of any similar project(s) outside of the scope of this document. Contact an electrician if in doubt.**
|
|
|
|
## PDU 1.0 and 2.0 - Hall Effect sensors
|
|
|
|
My first two forrays into the custom PDU project were simple devices that used the ACS714 [Hall effect](https://en.wikipedia.org/wiki/Hall_effect) current sensors alone. These units were built out of plastic wall boxes with the sensors held in series with the hot lines connecting to each plug.
|
|
|
|
The first iteration worked well, but was quite small, with only 16 outlets (4 boxes), which I quickly outgrew.
|
|
|
|
|
|
|
|
The second iteration was a fair bit larger, with 28 outlets (7 boxes), which was more than enough for my rack even now.
|
|
|
|
|
|
|
|
This design had a lot of downsides however:
|
|
|
|
1. In terms of monitoring, only getting the current was problematic. Current, measured in amperes, is only one part of the energy equation, and voltage and power factor are other important components which the ACS714 sensor does not provide. I was able to hack together a solution in my monitoring software by using the output voltage readings from my UPSes, but this wasn't satisfactory to me, especially given the slow speed of readings and the inaccuracy relative to a live device reading.
|
|
|
|
2. The sensors were, in my experience, quite unreliable. They were nearly impossible to calibrate and would sometimes report wildly inaccurate values due to interference. This was especially pronounced with low loads, since I needed to use 20A sensors for safety which have a correspondingly low threshold. Under 0.1A (about 12W) they were completely useless, and under 0.5A (about 60W) they were often +/- 15-20% out from a Kill-A-Watt's readings. Only at very high current values (>1.0A) were they accurate, and then only to about 1 decimal place, a fairly rough value.
|
|
|
|
3. The physical design of the PDU was cumbersome. Each box had to be wired in a very tight space with very tight tolerances on wire length, leading to many a scraped and cut finger. This was fine at the start, but connect 8 of these boxes together and the unit became cumbersome to work with. Maintenance was also a hassle for this reason. If a sensor died, which thankfully has not happened, replacing it would be a massive chore. And due to the through runs of the power busses, made out of normal 14-2 Romex wire, the boxes were permanently attached to each other, making disassembly tricky at best.
|
|
|
|
In setting out to design version 3 of the PDU, I wanted to solve all 3 issues, making something more robust and easier to service and maintain, as well as more accurate.
|
|
|
|
## PDU 3.0: Physical design
|
|
|
|
Solving issue 3 turned out to be fairly easy - the solution was a 3D Printer, specifically my new Ender 3 v2. Instead of using pre-made 3-gang plastic wall boxes, I could design individual "modules" one at a time, print their components, and then assemble them together using some sort of quick-release between them.
|
|
|
|
I began with a plan to create a CAD design of a full box, but ultimately this ended up going nowhere, not least due to my lack of experience (and patience!) with 3D modeling software. Instead, I was able to find two smaller components with which I could build out larger boxes: a 2-gang wallplate, and a 120mm "honeycomb" fan grill. These two components could easily be combined with a bit of superglue and electrical tape to form a 2-outlet cubic box which would hold all the wiring, the sensors, and the plugs, while providing ample room to work, as well as an open visual apperance to allow easy inspection of the internal components.
|
|
|
|
|
|
|
|
To connect the electrical portion of the modules together, I avoided the 1.0 and 2.0 method of using marette connectors to join multiple leads, and instead went with a busbar concept. I was able to find two designs of busbar: a dual-bus, 4-post version which would be used for the hot and neutral leads, and a raised 6-post version for the ground leads. This would greatly simplify the assembly by allowing me to use Y-connectors and securely screw down the leads, keeping everything very neat.
|
|
|
|
|
|
|
|
These busbars were then mounted on the bus sidepeace pictured above, to give a secure base to work off of. This piece alone was enough to assemble the core electrical components easily with plenty of working room.
|
|
|
|
|
|
|
|
Connecting the leads was then a trivial exercise of cutting exactly-length pieces of 14-gauge wire, stripping the right amount off each end, and bending them into position. Each sensor - we'll cover these in the next section - required 4 leads: two hot, in and out, and two neutral, in and out (though bridged internally), so all 4 leads met up in a level location towards the back of the module.
|
|
|
|
|
|
|
|
The final step was a method to connect the modules to each other. Rather than fixed, through wire, I settled on a relatively-recent innovation, which is very common in Europe but virtually unknown in North America: clamp-down connectors. These are absolutely fantastic for this purpose, able to handle a full 32A through them while being absolutely trivial to connect and disconnect. And it turned out that the clearances between modules were absolutely perfect for these. Thus, I could easily connect and disconnect modules during assembly or maintenance.
|
|
|
|
|
|
|
|
And *voilà*, a finished module, ready to accept the four sensor modules. I could then attach the other side plate and the back when I was ready to connect the modules and microcontroller.
|
|
|
|
|
|
|
|
The input section proved to be equally simple to assemble. I was able to find a set of combination IEC-13 input, fuse, and switch units from Amazon fairly easily. While technically rated only for 10A, I feel comfortable with this since each circuit should only ever be expected to run just under 10A load, and this is a derated value. I simply replaced the stock fuses with a 15A fast-blow variety for safety, and assembled a final segment to hold them. Input can now be handled by a simple IEC cord, along with switching, power control, and an indicator light, protected by a fuse should anything ever short out.
|
|
|
|
|
|
|
|
While I ultimately did need to permanently attach the modules physically to keep the movement from fatiguing the wires, breaking this connection in the future would simply be a matter of cutting some cable ties and breaking some glue bonds to replace the module. I could also extend the unit arbitrarily, with another module or two should I eventually need more than 32 ports.
|
|
|
|
The last step was assembling a cage for the sensor modules. One "flaw" in the design of the sensors I will mention below is that they float at line level, so an external enclosure was absolutely critical to avoid accidentally touching the modules.
|
|
|
|
## PDU 3.0: Sensors
|
|
|
|
As previously mentioned, the original PDU designs used a Hall effect sensor, which only reported current. But for version 3.0, I wanted something more feature-rich and robust. In scouring the Internet for a solution, I came across the perfect device: the HLW8012.
|
|
|
|
|
|
|
|
This module is able to output PWM signals on two interface pins that correspond to the active power (in Watts), and, via a select pin, the voltage or current passing through the sensor. Though there are some flaws with the design, specifically that the voltage is read from the neutral side and thus a large voltage drop in the load will provide bogus readings, and the fact that these units float at line level due to their design, these seemed like the perfect fit.
|
|
|
|
I was able to find a blog post by Xose Pérez, titled [The HLW8012 IC in the new Sonoff POW](https://tinkerman.cat/post/hlw8012-ic-new-sonoff-pow/) which went over the basics of how this sensor worked and how it can be used to make a DIY single-plug power monitor, similar to a Kill-A-Watt. He [even wrote an Arduino library for it](https://github.com/xoseperez/hlw8012)!
|
|
|
|
Armed with this knowledge, I ordered 48 (and later, due to an error, another 24) of the sensors from a Chinese seller called [ElectroDragon](https://www.electrodragon.com/product/energy-meter-hlw8012-breakout-board/) and got to work assembling the last component.
|
|
|
|
|
|
|
|
During my preliminary testing with an Aruino Uno, however, I found some issues with Xose's library - it was clearly not designed to work with more than one module at a time, so I had to come up with another solution. I also ran into an issue that plagued the 2.0 design: how to collect dozens of wires from dozens of sensors back to a central microcontroller to actually power and read data from them.
|
|
|
|
## PDU 3.0: Microcontrollers
|
|
|
|
My first idea was to use the Arduino Mega. This is a monstrous Arduino microcontroller with 54 digital I/Os, more than enough to handle 16 or 18 sensors per circuit, which was my goal - match or slightly excede the 32-port 2.0 design. But it had some fatal flaws: first, this is an 8-bit microcontroller which makes dealing with relatively-large integer and floating point values very cumbersome; second, the microcontroller CPU is very slow; and third and most important to my physical design, I would have to run a total of 20 wires from each module back up to the central Arduino board at the top, an exercise in frustration.
|
|
|
|
I continued to look for a good solution, when finally a discussion with a friend led to an excellent discovery: the STM32 "Black Pill" microcontroller. Forget an 8-bit 16MHz 54-I/O Arduino board; this tiny monster can do 16 I/Os with a 32-bit 100MHz ARM-based core, which is more than enough to read sensors on the order of microseconds. And it has a USB-C output!
|
|
|
|
|
|
|
|
But one microcontroller wouldn't cut it; I'd have a total of 3 leads (plus power) from each sensor, and due to the "float at line voltage" design of the sensors, needed separate microcontrollers for each circuit lest they short. So I would need more than just one. Thus I came up with an even better solution: why not use one STM32 for "each" module (really, one for each "side" of two modules)? This had two benefits; first, I could keep the "each module is separate" mentality even though the compute side; and second, the USB connections would make it trivial to run back to the central controller, a Raspberry Pi - I would just need a USB hub, USB isolator, and some relatively-short USB cables, instead of dozens and dozens of discrete leads. And the distance between each sensor and the microcontroller was small enough to use normal 6" jumper wires. A match made in heaven!
|
|
|
|
|
|
|
|
## PDU 3.0: Programming
|
|
|
|
With all the physical assembly completed and the units ready for live testing, I was able to get started with the programming aspect.
|
|
|
|
The first task was to calibrate the sensors. Looking through Xose's library code, I was able to determine that each sensor would need 3 multiplier values (one each for wattage, voltage, and amperage), which was then divided by the resulting PWM value to produce a usable reading. But how to find those multipliers?
|
|
|
|
To solve this problem, I used on of the free STM32's to build a sensor calibrator. The idea is quite simple: I would connect up the sensor to the calibrator, attach a known resistive load (a 40W and 60W lightbulb in parallel totaling ~100W of load), and connect everything to a no-name Kill-A-Watt to give me a reference. I could then enter the reference value the Kill-A-Watt showed, and let the calibrator read the modules and calculate the correct multiplier.
|
|
|
|
This process took a lot of iteration to get right, and in the end I settled on code that would run a large number of scans trying to determine the exact value that matched my input values. But it was worth the time, and the results turned out to be perfect - I was able to use the calibrator on each sensor to determine what their multiplier should be, and then store this for later use inside the live code on each microcontroller, giving me nearly to-the-watt accuracy.
|
|
|
|
```C++
|
|
[calibrator code]
|
|
```
|
|
|
|
|
|
|
|
With the calibration values in hand, I turned to writing code to handle the actual module microcontrollers. The code here ended up being extremely simple once I had the calibration: simply poll the PWM of each sensor in turn, calculate the output, then display it on the serial console for reading by the Raspberry Pi using JSON formatting. Note the `struct` for the sensor modules, which contain the individual multipiers found during the calibration step for that given module, as well as the identifier of each plug.
|
|
|
|
```C++
|
|
[microcontroller code]
|
|
```
|
|
|
|
The final step was to write the controller software for the Raspberry Pi side. This turned out to be a bit complex, due to needing to read from a total of 10 microcontrollers in quick succession.
|
|
|
|
[TBD]
|
|
|
|
```Python
|
|
[pdusensord code]
|
|
```
|
|
|
|
## Putting everything together
|
|
|
|
With all the programming and module assembly done, I could begin assembling the final PDU. Here is the final result:
|
|
|
|
|
|
|
|
I spent a few days load testing it with a resistive heater in my garage to be sure everything was in safe working order, and it passed all my tests. I then let it run with 3 servers attached for 3 full months to do a final burn-in, occasionally switching which outlets they were attached to, without incident.
|
|
|
|
The last step was final installation into my rack:
|
|
|
|
|
|
|
|
And the readings by my monitoring software are exactly what I wanted - accurate to the Watt, at least in theory:
|
|
|
|
|
|
|
|
I hope you found this interesting!
|