blog/content/post/custom-pdu-3.md

+++ tags = ["diy","homelab","buildlog"] title = "Build a Custom Power Monitoring PDU" description = "Building a custom power monitoring PDU for fun and profit" type = "post" weight = 1 draft = true +++

As a veteran homelabber, one thing that always comes up is power usage. Electricity is, unfortunately, not free. And servers can use a lot of it. Some systems provide on-demand power monitoring via their IPMI/BMC interfaces, but not all do. And figuring out how much power these systems use can be a hassle.

The most obvious solution is what is normally called a "per-port monitored PDU". Typically used by colocation providers and large enterprises, these power distribution units (PDUs) allow the administrator to see the actual power usage out of each individual port at a given time, both for billing and monitoring purposes. They're the perfect solution to the problem of not knowing how much power you are using.

But these PDUs are not cheap. New, the cheapest ones I've been able to find run over $1400 USD, and they're gigantic 5-foot monsters. Add in dual circuits/UPSes, and your cost doubles. There has to be a better way.

Well, there is. With a decent amount of electrical know-how, some programming, 3D printing, and a lot of patience, I've been able to build myself several custom PDUs. Read on to know how!

PDU 1.0 and 2.0 - Hall Effect sensors

My first two forrays into the custom PDU project were simple devices that used the ACS714 Hall effect current sensors alone. These units were built out of plastic wall boxes with the sensors held in series with the hot lines connecting to each plug.

This design had a lot of downsides however:

1. In terms of monitoring, only getting the current was somewhat problematic. Current is only one part of the energy equation, and voltage and power factor are other important components which the sensor does not provide.

2. The sensors were, in my experience, notoriously unreliable. They were nearly impossible to calibrate (due partly to #1) and would sometimes report wildly inaccurate values due to interference.

3. The physical design of the PDU was cumbersome. Each box had to be wired in a very tight space with very tight tolerances on wire length, leading to many a scraped and cut finger.

4. Due to the through runs of the power busses, made out of normal 14-2 Romex wire, the boxes were permanently attached. This was fine at the start, but connect 8 of these boxes together and the unit became cumbersome to work with.

In setting out to design version 3 of the PDU, I wanted to solve all 4 issues, making something more robust and easier to service and maintain, as well as more accurate.

PDU 3.0: Physical design

Solving issues 3 and 4 turned out to be fairly easy - the solution was a 3D Printer, specifically my new Ender 3 v2. Instead of using pre-made 3-gang plastic wall boxes, I could design individual "modules" one at a time, print their components, and then assemble them together using some sort of quick-release between them.

I began with a plan to create a CAD design of a full box, but ultimately this ended up going nowhere, not least due to my lack of experience (and patience!) with 3D modeling software. Instead, I was able to find two smaller components with which I could build out larger boxes: a 2-gang wallplate, and a 120mm "honeycomb" fan grill. These two components could easily be combined with a bit of superglue and electrical tape to form a 2-outlet cubic box which would hold all the wiring, the sensors, and the plugs, while providing ample room to work, as well as an open visual apperance to allow easy inspection of the internal components.

To connect the electrical portion of the modules together, I avoided the 1.0 and 2.0 method of using marette connectors to join multiple leads, and instead went with a busbar concept. I was able to find two designs of busbar: a dual-bus, 4-post version which would be used for the hot and neutral leads, and a raised 6-post version for the ground leads. This would greatly simplify the assembly by allowing me to use Y-connectors and securely screw down the leads, keeping everything very neat.

These busbars were then mounted on the bus sidepeace pictured above, to give a secure base to work off of. This piece alone was enough to assemble the core electrical components easily with plenty of working room.

Connecting the leads was then a trivial exercise of cutting exactly-length pieces of 14-gauge wire, stripping the right amount off each end, and bending them into position. Each sensor - we'll cover these in the next section - required 4 leads: two hot, in and out, and two neutral, in and out (though bridged internally), so all 4 leads met up in a level location towards the back of the module.

The final step was a method to connect the modules to each other. Rather than fixed, through wire, I settled on a relatively-recent innovation, which is very common in Europe but virtually unknown in North America: clamp-down connectors. These are absolutely fantastic for this purpose, able to handle a full 32A through them while being absolutely trivial to connect and disconnect. And it turned out that the clearances between modules were absolutely perfect for these. Thus, I can easily connect and disconnect modules during assembly.

And voila, a finished module, ready to accept the four sensor modules. I could then attach the other side plate and the back when I was ready to connect the modules and microcontroller.

The input section proved to be equally simple to assemble. I was able to find a set of combination IEC-13 input, fuse, and switch units from Amazon fairly easily. While technically rated only for 10A, I feel comfortable with this since each circuit should only ever be expected to run just under 10A load, and this is a derated value. I simply replaced the stock fuses with a 15A fast-blow variety for safety, and assembled a final segment to hold them. Input can now be handled by a simple IEC cord, along with switching, power control, and an indicator light, protected by a fuse should anything ever short out.

PDU 3.0: Sensors

As previously mentioned, the original PDU designs used a Hall effect sensor, which only reported current. But for version 3.0, I wanted something more feature-rich and robust. In scouring the Internet for a solution, I came across the perfect device: the HLW8012.

This module is able to output PWM signals on two interface pins that correspond to the active power (in Watts), and, via a select pin, the voltage and current passing through the sensor. Though there are some flaws with the design, specifically that the voltage is read from the neutral side and thus a large voltage drop in the load will provide bogus readings, these seemed like the perfect fit.

I was able to find a blog post by Xose Pérez, titled The HLW8012 IC in the new Sonoff POW which went over the basics of how this sensor worked and how it can be used to make a DIY single-plug power monitor, similar to a Kill-A-Watt. He even wrote an Arduino library for it!

Armed with this knowledge, I ordered 48 of the sensors from a Chinese seller called ElectroDragon and got to work assembling the last component.

During my preliminary testing with an Aruino Uno, however, I found some issues with Xose's library - it was clearly not designed to work with more than one module at a time, so I had to come up with another solution. I also ran into an issue that plagued the 2.0 design: how to collect dozens of wires from dozens of sensors back to a central microcontroller to actually power and read data from them.

PDU 3.0: Microcontrollers

My first idea was to use the Arduino Mega. This is a monstrous Arduino microcontroller with 54 digital I/Os, more than enough to handle 16 or 18 sensors per circuit, which was my goal - match or slightly excede the 32-port 2.0 design. But it had some fatal flaws: first, this is an 8-bit microcontroller which makes dealing with relatively-large integer and floating point values very cumbersome; second, the microcontroller CPU is very slow; and third and most important to my physical design, I would have to run a total of 20 wires from each module back up to the central Arduino board at the top, an exercise in frustration.

I continued to look for a good solution, when finally a discussion with a friend led to an excellent discovery: the STM32 "Black Pill" microcontroller. Forget an 8-bit 16MHz 54-I/O Arduino board; this tiny monster can do 16 I/Os with a 32-bit 100MHz ARM-based core, which is more than enough to read sensors on the order of microseconds. And it has a USB-C output!

But one microcontroller wouldn't cut it; I'd have a total of 3 leads (plus power) from each sensor, so I would need more than just two. Thus I came up with an even better solution: why not use one STM32 for each module? This had two benefits; first, I could keep the "each module is separate" mentality even though the compute side; and second, the USB connections would make it trivial to run back to the central controller, a Raspberry Pi - I would just need a USB hub and some relatively-short USB cables, instead of dozens and dozens of discrete leads. And the distance between each sensor and the microcontroller was small enough to use normal 6" jumper wires. A match made in heaven!

PDU 3.0: Programming

With all the physical assembly completed and the units ready for live testing, I was able to get started with the programming aspect.

The first task was to calibrate the sensors. Looking through Xose's library code, I was able to determine that each sensor would need 3 multiplier values (one each for wattage, voltage, and amperage), which was then divided by the resulting PWM value to produce a usable reading. But how to find those multipliers?

To solve this problem, I used on of the free STM32's to build a sensor calibrator. The idea is quite simple: I would connect up the sensor to the calibrator, attach a known resistive load (a 40W and 60W lightbulb in parallel totaling ~100W of load), and connect everything to a no-name Kill-A-Watt to give me a reference. I could then enter the reference value the Kill-A-Watt showed, and let the calibrator read the modules and calculate the correct multiplier. The results turned out to be perfect - I was able to use the calibrator on each sensor to determine what their multiplier should be, and then store this for later.

[calibrator code]

With the calibration values in hand, I turned to writing code to handle the actual module microcontrollers. The code here ended up being extremely simple: simply poll the PWM of each sensor in turn, calculate the output, then display it on the serial console for reading by the Raspberry Pi. Note the struct for the sensor modules, which contain the individual multipiers found during the calibration step for that given module, as well as the identifier of each plug.

[microcontroller code]

The final step was to write the controller software for the Raspberry Pi side.

[tbd]

Putting everything together

With all the programming and module assembly done, I could begin assembling the final PDU. Here is the final result:

I spent a few days load testing it with a resistive heater in my garage to be sure everything was in safe working order, and it passed all my tests. The last step was final installation into my rack:

And the readings by my monitoring software are exactly what I wanted - accurate to the Watt: