Tweaks to the PDU document
This commit is contained in:
parent
b5e6bc8c17
commit
ffc4de480d
|
|
@ -8,35 +8,37 @@ weight = 1
|
|||
draft = true
|
||||
+++
|
||||
|
||||
As a veteran homelabber, one thing that always comes up is power usage. Electricity is, unfortunately, not free. And servers can use a lot of it. Some systems provide on-demand power monitoring via their IPMI/BMC interfaces, but not all do. And figuring out how much power these systems use can be a hassle.
|
||||
As a veteran homelabber, one thing that always comes up is power usage. Electricity is, unfortunately, not free, even if I do get half of it from "too cheap to meter" nuclear power here in Ontario. And servers can use a lot of it. Some systems provide on-demand power monitoring via their IPMI/BMC interfaces, but not all do. And figuring out how much power the other, non-intelligent, systems use can be a hassle.
|
||||
|
||||
The most obvious solution is what is normally called a "per-port monitored PDU". Typically used by colocation providers and large enterprises, these power distribution units (PDUs) allow the administrator to see the actual power usage out of each individual port at a given time, both for billing and monitoring purposes. They're the perfect solution to the problem of not knowing how much power you are using.
|
||||
|
||||
But these PDUs are not cheap. New, the cheapest ones I've been able to find run over $1400 USD, and they're gigantic 5-foot monsters. Add in dual circuits/UPSes, and your cost doubles. There has to be a better way.
|
||||
But these PDUs are not cheap. New, the cheapest ones I've been able to find run over $1400 USD, and they're gigantic 5-foot monsters designed for full-sized 42U+ racks. Add in dual circuits/UPSes, and the cost doubles. There has to be a better way.
|
||||
|
||||
Well, there is. With a decent amount of electrical know-how, some programming, 3D printing, and a lot of patience, I've been able to build myself several custom PDUs. Read on to know how!
|
||||
|
||||
**DISCLAIMER/WARNING:** While I am not a professional licensed electrician, I've spent a large portion of my life working with A/C electricity, pretty much from the time I could walk and hold a screwdriver, including a year as an Electrical associate at The Home Depot, as well as numerous home projects and two previous PDUs. I know what I'm doing. **Working with mains electricity is very dangerous, especially if you do not know what you're doing.** This post is provided as a curiosity for most, and a build log/guide only for those who are well-versed in working with this sort of thing. **Do not try this at home. I will not provide advice or guidance on any aspect of any similar project(s) outside of the scope of this document. Contact an electrician if in doubt.**
|
||||
|
||||
## PDU 1.0 and 2.0 - Hall Effect sensors
|
||||
|
||||
My first two forrays into the custom PDU project were simple devices that used the ACS714 [Hall effect](https://en.wikipedia.org/wiki/Hall_effect) current sensors alone. These units were built out of plastic wall boxes with the sensors held in series with the hot lines connecting to each plug.
|
||||
|
||||
The first iteration worked well, but was quite small, with only 16 outlets (4 boxes), which I quickly outgrew. The second iteration was a fair bit larger, with 28 outlets (7 boxes), which was more than enough for my rack even now.
|
||||
|
||||
|
||||
|
||||
This design had a lot of downsides however:
|
||||
|
||||
1. In terms of monitoring, only getting the current was somewhat problematic. Current is only one part of the energy equation, and voltage and power factor are other important components which the sensor does not provide.
|
||||
1. In terms of monitoring, only getting the current was somewhat problematic. Current, measured in amperes, is only one part of the energy equation, and voltage and power factor are other important components which the ACS714 sensor does not provide.
|
||||
|
||||
2. The sensors were, in my experience, notoriously unreliable. They were nearly impossible to calibrate (due partly to #1) and would sometimes report wildly inaccurate values due to interference.
|
||||
2. The sensors were, in my experience, notoriously unreliable. They were nearly impossible to calibrate (due partly to #1) and would sometimes report wildly inaccurate values due to interference. This was especially pronounced with low loads, since I needed to use 20A sensors for safety which have a correspondingly low threshold. Under 0.1A (about 12W) they were completely useless, and under 0.5A (about 60W) they were often +/- 15-20% out from a Kill-A-Watt's readings.
|
||||
|
||||
3. The physical design of the PDU was cumbersome. Each box had to be wired in a very tight space with very tight tolerances on wire length, leading to many a scraped and cut finger. Maintenance was also a hassle for this reason.
|
||||
3. The physical design of the PDU was cumbersome. Each box had to be wired in a very tight space with very tight tolerances on wire length, leading to many a scraped and cut finger. Maintenance was also a hassle for this reason. If a sensor died, which thankfully has not happened, replacing it would be a massive chore. And due to the through runs of the power busses, made out of normal 14-2 Romex wire, the boxes were permanently attached to each other. This was fine at the start, but connect 8 of these boxes together and the unit became cumbersome to work with.
|
||||
|
||||
4. Due to the through runs of the power busses, made out of normal 14-2 Romex wire, the boxes were permanently attached. This was fine at the start, but connect 8 of these boxes together and the unit became cumbersome to work with.
|
||||
|
||||
In setting out to design version 3 of the PDU, I wanted to solve all 4 issues, making something more robust and easier to service and maintain, as well as more accurate.
|
||||
In setting out to design version 3 of the PDU, I wanted to solve all 3 issues, making something more robust and easier to service and maintain, as well as more accurate.
|
||||
|
||||
## PDU 3.0: Physical design
|
||||
|
||||
Solving issues 3 and 4 turned out to be fairly easy - the solution was a 3D Printer, specifically my new Ender 3 v2. Instead of using pre-made 3-gang plastic wall boxes, I could design individual "modules" one at a time, print their components, and then assemble them together using some sort of quick-release between them.
|
||||
Solving issue 3 turned out to be fairly easy - the solution was a 3D Printer, specifically my new Ender 3 v2. Instead of using pre-made 3-gang plastic wall boxes, I could design individual "modules" one at a time, print their components, and then assemble them together using some sort of quick-release between them.
|
||||
|
||||
I began with a plan to create a CAD design of a full box, but ultimately this ended up going nowhere, not least due to my lack of experience (and patience!) with 3D modeling software. Instead, I was able to find two smaller components with which I could build out larger boxes: a 2-gang wallplate, and a 120mm "honeycomb" fan grill. These two components could easily be combined with a bit of superglue and electrical tape to form a 2-outlet cubic box which would hold all the wiring, the sensors, and the plugs, while providing ample room to work, as well as an open visual apperance to allow easy inspection of the internal components.
|
||||
|
||||
|
|
@ -54,11 +56,11 @@ Connecting the leads was then a trivial exercise of cutting exactly-length piece
|
|||
|
||||
|
||||
|
||||
The final step was a method to connect the modules to each other. Rather than fixed, through wire, I settled on a relatively-recent innovation, which is very common in Europe but virtually unknown in North America: clamp-down connectors. These are absolutely fantastic for this purpose, able to handle a full 32A through them while being absolutely trivial to connect and disconnect. And it turned out that the clearances between modules were absolutely perfect for these. Thus, I can easily connect and disconnect modules during assembly.
|
||||
The final step was a method to connect the modules to each other. Rather than fixed, through wire, I settled on a relatively-recent innovation, which is very common in Europe but virtually unknown in North America: clamp-down connectors. These are absolutely fantastic for this purpose, able to handle a full 32A through them while being absolutely trivial to connect and disconnect. And it turned out that the clearances between modules were absolutely perfect for these. Thus, I could easily connect and disconnect modules during assembly or maintenance.
|
||||
|
||||
|
||||
|
||||
And voila, a finished module, ready to accept the four sensor modules. I could then attach the other side plate and the back when I was ready to connect the modules and microcontroller.
|
||||
And *voilà*, a finished module, ready to accept the four sensor modules. I could then attach the other side plate and the back when I was ready to connect the modules and microcontroller.
|
||||
|
||||
|
||||
|
||||
|
|
@ -66,17 +68,21 @@ The input section proved to be equally simple to assemble. I was able to find a
|
|||
|
||||
|
||||
|
||||
While I ultimately did need to permanently attach the modules physically to keep the movement from fatiguing the wires, breaking this connection in the future would simply be a matter of cutting some cable ties and breaking some glue bonds to replace the module. I could also extend the unit arbitrarily, with another module or two should I eventually need more than 32 ports.
|
||||
|
||||
The last step was assembling a cage for the sensor modules. One "flaw" in the design of the sensors I will mention below is that they float at line level, so an external enclosure was absolutely critical to avoid accidentally touching the modules.
|
||||
|
||||
## PDU 3.0: Sensors
|
||||
|
||||
As previously mentioned, the original PDU designs used a Hall effect sensor, which only reported current. But for version 3.0, I wanted something more feature-rich and robust. In scouring the Internet for a solution, I came across the perfect device: the HLW8012.
|
||||
|
||||
|
||||
|
||||
This module is able to output PWM signals on two interface pins that correspond to the active power (in Watts), and, via a select pin, the voltage and current passing through the sensor. Though there are some flaws with the design, specifically that the voltage is read from the neutral side and thus a large voltage drop in the load will provide bogus readings, these seemed like the perfect fit.
|
||||
This module is able to output PWM signals on two interface pins that correspond to the active power (in Watts), and, via a select pin, the voltage or current passing through the sensor. Though there are some flaws with the design, specifically that the voltage is read from the neutral side and thus a large voltage drop in the load will provide bogus readings, and the fact that these units float at line level due to their design, these seemed like the perfect fit.
|
||||
|
||||
I was able to find a blog post by Xose Pérez, titled [The HLW8012 IC in the new Sonoff POW](https://tinkerman.cat/post/hlw8012-ic-new-sonoff-pow/) which went over the basics of how this sensor worked and how it can be used to make a DIY single-plug power monitor, similar to a Kill-A-Watt. He [even wrote an Arduino library for it](https://github.com/xoseperez/hlw8012)!
|
||||
|
||||
Armed with this knowledge, I ordered 48 of the sensors from a Chinese seller called [ElectroDragon](https://www.electrodragon.com/product/energy-meter-hlw8012-breakout-board/) and got to work assembling the last component.
|
||||
Armed with this knowledge, I ordered 48 (and later, due to an error, another 24) of the sensors from a Chinese seller called [ElectroDragon](https://www.electrodragon.com/product/energy-meter-hlw8012-breakout-board/) and got to work assembling the last component.
|
||||
|
||||
|
||||
|
||||
|
|
@ -90,7 +96,7 @@ I continued to look for a good solution, when finally a discussion with a friend
|
|||
|
||||
|
||||
|
||||
But one microcontroller wouldn't cut it; I'd have a total of 3 leads (plus power) from each sensor, so I would need more than just two. Thus I came up with an even better solution: why not use one STM32 for each module? This had two benefits; first, I could keep the "each module is separate" mentality even though the compute side; and second, the USB connections would make it trivial to run back to the central controller, a Raspberry Pi - I would just need a USB hub and some relatively-short USB cables, instead of dozens and dozens of discrete leads. And the distance between each sensor and the microcontroller was small enough to use normal 6" jumper wires. A match made in heaven!
|
||||
But one microcontroller wouldn't cut it; I'd have a total of 3 leads (plus power) from each sensor, and due to the "float at line voltage" design of the sensors, needed separate microcontrollers for each circuit lest they short. So I would need more than just one. Thus I came up with an even better solution: why not use one STM32 for "each" module (really, one for each "side" of two modules)? This had two benefits; first, I could keep the "each module is separate" mentality even though the compute side; and second, the USB connections would make it trivial to run back to the central controller, a Raspberry Pi - I would just need a USB hub, USB isolator, and some relatively-short USB cables, instead of dozens and dozens of discrete leads. And the distance between each sensor and the microcontroller was small enough to use normal 6" jumper wires. A match made in heaven!
|
||||
|
||||
|
||||
|
||||
|
|
@ -100,7 +106,9 @@ With all the physical assembly completed and the units ready for live testing, I
|
|||
|
||||
The first task was to calibrate the sensors. Looking through Xose's library code, I was able to determine that each sensor would need 3 multiplier values (one each for wattage, voltage, and amperage), which was then divided by the resulting PWM value to produce a usable reading. But how to find those multipliers?
|
||||
|
||||
To solve this problem, I used on of the free STM32's to build a sensor calibrator. The idea is quite simple: I would connect up the sensor to the calibrator, attach a known resistive load (a 40W and 60W lightbulb in parallel totaling ~100W of load), and connect everything to a no-name Kill-A-Watt to give me a reference. I could then enter the reference value the Kill-A-Watt showed, and let the calibrator read the modules and calculate the correct multiplier. The results turned out to be perfect - I was able to use the calibrator on each sensor to determine what their multiplier should be, and then store this for later.
|
||||
To solve this problem, I used on of the free STM32's to build a sensor calibrator. The idea is quite simple: I would connect up the sensor to the calibrator, attach a known resistive load (a 40W and 60W lightbulb in parallel totaling ~100W of load), and connect everything to a no-name Kill-A-Watt to give me a reference. I could then enter the reference value the Kill-A-Watt showed, and let the calibrator read the modules and calculate the correct multiplier.
|
||||
|
||||
This process took a lot of iteration to get right, and in the end I settled on code that would run a large number of scans trying to determine the exact value that matched my input values. But it was worth the time, and the results turned out to be perfect - I was able to use the calibrator on each sensor to determine what their multiplier should be, and then store this for later.
|
||||
|
||||
```C++
|
||||
[calibrator code]
|
||||
|
|
@ -114,9 +122,9 @@ With the calibration values in hand, I turned to writing code to handle the actu
|
|||
[microcontroller code]
|
||||
```
|
||||
|
||||
The final step was to write the controller software for the Raspberry Pi side.
|
||||
The final step was to write the controller software for the Raspberry Pi side. This turned out to be a bit complex, due to needing to read from a total of 10 microcontrollers in quick succession.
|
||||
|
||||
[tbd]
|
||||
[TBD]
|
||||
|
||||
## Putting everything together
|
||||
|
||||
|
|
@ -124,10 +132,14 @@ With all the programming and module assembly done, I could begin assembling the
|
|||
|
||||
|
||||
|
||||
I spent a few days load testing it with a resistive heater in my garage to be sure everything was in safe working order, and it passed all my tests. The last step was final installation into my rack:
|
||||
I spent a few days load testing it with a resistive heater in my garage to be sure everything was in safe working order, and it passed all my tests. I then let it run with 3 servers attached for 3 full months to do a final burn-in, occasionally switching which outlets they were attached to, without incident.
|
||||
|
||||
The last step was final installation into my rack:
|
||||
|
||||
|
||||
|
||||
And the readings by my monitoring software are exactly what I wanted - accurate to the Watt:
|
||||
And the readings by my monitoring software are exactly what I wanted - accurate to the Watt, at least in theory:
|
||||
|
||||
|
||||
|
||||
I hope you found this interesting!
|
||||
|
|
|
|||
Loading…
Reference in New Issue