joshuaboniface
/
blog
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Major tweaks to various parts of the post.

This commit is contained in:
Joshua Boniface 2017-03-28 00:56:40 -04:00
parent 095b5d1d6d
commit b7565a3ec4
1 changed files with 78 additions and 63 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

141
content/post/a-raspberry-pi-bmc.md
View File

@ -13,7 +13,7 @@ IPMI BMCs are pretty ubiquitous in the datacenter and enterprise computing, beca
## Enter the Raspberry Pi ## Enter the Raspberry Pi
If you don't know what it is, the Rapberry Pi is a small single-board computer, featuring an ARM SOC, Ethernet, USB, video, audio, and most importantly, GPIO, powered by MicroUSB and running the Debian distribution '[Raspbian](https://www.raspberrypi.org/downloads/raspbian/)'. The GPIO pins allow, with a simple utility or Python library, one to control or read information from various devices, including a serial console interface. These features make the Raspberry Pi a perfect fit for a BMC, and really not that far from the "real" BMCs found in most server-grade motherboards. If you don't know what it is, the [Rapberry Pi](https://www.raspberrypi.org) is a small single-board computer, featuring an ARM SOC, Ethernet, USB, video, audio, and most importantly, GPIO (General-Purpose Input and Output) pins, which is powered by MicroUSB and running the Debian distribution '[Raspbian](https://www.raspberrypi.org/downloads/raspbian/)'. The GPIO pins allow one to control or read information with a simple utility or Python library, from various devices, including a serial console interface. These features make the Raspberry Pi a perfect fit for a BMC, and really not that far from the "real" BMCs found in most server-grade motherboards.
(Pictured: A Raspberry Pi) (Pictured: A Raspberry Pi)
[Picture - RPi] [Picture - RPi]
@ -24,101 +24,116 @@ If you don't know what it is, the Rapberry Pi is a small single-board computer,
One of the main ideas behind a BMC is that it should be accessible even if the host system is off. So we need some sort of power independent of the host state. The first thought when using a Raspberry Pi as a BMC would be to power it from some sort of external USB power brick, but that's messy: another cable running out of the case that must be dealt with, and another power brick. Luckily for us however, the ATX power supply standard has a solution! One of the main ideas behind a BMC is that it should be accessible even if the host system is off. So we need some sort of power independent of the host state. The first thought when using a Raspberry Pi as a BMC would be to power it from some sort of external USB power brick, but that's messy: another cable running out of the case that must be dealt with, and another power brick. Luckily for us however, the ATX power supply standard has a solution!
The purple wire on a standard 24-pin ATX connector provides a standby +5V power supply, rated in the spec for up to 50mA but in reality today often supporting close to 1A. This is more than enough to run the motherboard standby power as well as a Raspberry Pi. This has the added bennefit of working just like a real BMC: when the system is unplugged, the BMC also turns off, and turns back on as soon as power is reapplied. With that in mind, I made a simple "adapter" out of a piece of solid-core CAT5 cable, which is inserted directly into the ATX motherboard connector, with the other end attached to a standard MicroUSB cable. The result is consistent, reliable, in-case power for the BMC without any trickery. The purple wire on a standard 24-pin ATX connector provides a standby +5V power supply, rated in the spec for up to 50mA but in reality today often supporting close to 1A. This is more than enough to run the motherboard standby power as well as a Raspberry Pi. This has the added benefit of working just like a real BMC: when the system is unplugged, the BMC also turns off, and turns back on as soon as power is reapplied. With that in mind, I made a simple "adapter" out of a piece of solid-core CAT5 cable and carefully encased in hot glue, which is inserted directly into the ATX motherboard connector, with the other end attached to a standard MicroUSB cable. The result is consistent, reliable, in-case power for the BMC without any trickery, and a cable tie keeps it safe. [Editor's note: WARNING - This may burn down your computer - I will eventually make proper connectors, but I'm not letting that hold up the project!]
*(Pictured: The power interface to the motherboard)*
(Pictured: The power interface to the motherboard, attached to a MicroUSB cable)
[Picture - power stuff] [Picture - power stuff]
### GPIO to rule them all
A BMC has one main function: controlling and determining the power state of the system. This basic functionality allows you to, for instance, hard reset a crashed machine, or start it back up after a power failure. No more running for the box to press the power button!
The Raspberry Pi's GPIO pins provide a nice simple method for interfacing with the raw system management headers on any motherboard. For determining the power state, a GPIO pin (in my case, pin 21) is connected with a small resistor directly to the Power LED connector on the motherboard. When this pin is read high, we know the system is online; when it's low, we know it's offline. Probably the simplest circuit in the project!
The power and reset switches are a little more complex. While you can direct the GPIO directly to the switch headers, this will not work as you would expect, and could in fact risk blowing out your motherboard by sending 3.3V into the switch headers. The solution is to use a transistor, and from my basic understanding any would do: connect the base pin of the transistor to your GPIO pin, and the collector and emitter pins to the switch header (emitter to ground). The result is an electrically controlled switch, which turns on when the GPIO is set high for a second, and turns off again when the GPIO is set low. You can now safely control the power and reset switches with your Raspberry Pi.
*(Pictured: the wiring layout for a first-generation model-B Raspberry Pi BMC)*
[Picture - wiring diagram]
### Serial - USB or TTL? ### Serial - USB or TTL?
The most critical component of the BMC, at least to me, is the ability to manage the host system without requiring a connected monitor+keyboard. Luckily just about every motherboard, and especially "server-grade" motherboards, still feature one or two DB9 COM ports. With some BIOS configurations, they will almost always support redirecting console output through the serial port, giving us exactly what we need for console access. However getting that console into the Raspberry Pi is a bit tricky. Another critical function of a BMC is to manage the host system without requiring a connected monitor+keyboard. Luckily just about every motherboard, and especially "server-grade" motherboards that otherwise lack a BMC, still feature one or two DB9 COM ports, and console redirection. With some BIOS configurations, console redirection lets you send the console output through the serial port, giving us an old-school VTY terminal: exactly what we need for console access. However getting that console into the Raspberry Pi is a bit tricky.
The most common way to do this is to use a USB to Serial adapter, but like the original power suggestion, it's messy in terms of cabling. Enter the TTL serial port on the Raspberry Pi. Built in to [GPIO pins 14 and 15](http://codeandlife.com/2012/07/01/raspberry-pi-serial-console-with-max3232cpe/), the Raspberry Pi features a TTL 3.3V serial interface. Normally you would use this to get a console to the Raspberry Pi itself, but for our purposes, a simple flip of the TXD and RXD lines and some [reconfiguration of the Raspbian image](http://www.hobbytronics.co.uk/raspberry-pi-serial-port), we can use this interface to directly communicate with the host system. The most common way to do this is to use a USB to Serial adapter, but like the original power suggestion, it's messy in terms of cabling, requiring a header, crossover cable, and then the adapter itself. Enter the TTL serial port on the Raspberry Pi. Built in to [GPIO pins 14 and 15](http://codeandlife.com/2012/07/01/raspberry-pi-serial-console-with-max3232cpe/), the Raspberry Pi features a TTL 3.3V serial interface. Normally you would use this to get a console into the Raspberry Pi itself, but with a simple flip of the TXD and RXD lines and some [reconfiguration of the Raspbian image](http://www.hobbytronics.co.uk/raspberry-pi-serial-port), we can use this interface to communicate with the host system instead.
This method does require a special chip, as you can see in the first link above. Luckily, the converter chips can be had for about $1.50 for 5 on eBay from the right Chinese sellers. And with the tiny little board and some small jumpers, we can connect the chip not to the rear DB9 port, but directly to the motherboard COM2 header; no messy crossover cables or USB to serial adapters! The resulting device is `/dev/ttyAMA0` in Raspbian and works flawlessly with your terminal emulator of choice; I'm using screen to allow persistence and clean disconnection without terminating the serial session (more on that later!) This method does require a special chip, as you can see in the first link above, called the MAX3232, which will convert between TTL serial from the Raspberry Pi and the RS232 serial of the motherboard. Luckily, the converter chips can be had for about $1.50 for 5 on eBay from the right Chinese sellers, including the capacitors you need to make the circuit work. And with some jumpers, we can connect the chip directly to the motherboard COM2 header; no messy crossover cables or USB to serial adapters! The resulting device is `/dev/ttyAMA0` in Raspbian and works flawlessly with your terminal emulator of choice; I'm using screen to allow persistence and clean disconnection without terminating the serial session (more on that later!)
The one downside of this method is the lack of proper VGA graphics support. Your OS needs to be text-based with support for console redirection, such as any Unix-like OS or possibly Windows Core (I don't know), and it needs to be configured with this support to get anywhere past the BIOS. As I'm using Debian on all my systems this is perfectly fine by me and I include the tweaks into my system images, but keep this in mind when setting up your BMC.
(Pictured: the MAX3232 signal converter board) (Pictured: the MAX3232 signal converter board)
[Picture - MAX3232 board] [Picture - MAX3232 board]
### GPIO to rule them all
No BMC would be complete without two more functions: controlling and determining the power state of the system. This basic functionality allows you to, for instance, hard reset a crashed machine, or start it back up after a power failure. No more running for the box to press the power button!
The Raspberry Pi's GPIO pins provide a nice simple method for interfacing with the raw system management headers on any motherboard. For determining the power state, a GPIO pin (in my case, pin 21) is connected with a small resistor directly to the Power LED connector on the motherboard. When this pin is high, we know the system is online; when it's low, we know it's offline. Dead-simple.
The power and reset switches are a little more complex. While you can direct the GPIO directly to the switch headers, this will not work as you would expect, and could in fact risk blowing out your motherboard by sending 3.3V into the switch headers. The solution is to use a transistor: connect the gate pin of the transistor to your GPIO pin, and the anode and cathode pins to the switch header. The result is an electrically controlled switch, which turns on when the GPIO is set high, and turns off again when the GPIO is set low. You can now safely control the power and reset switches with your Raspberry Pi.
(Pictured: the wiring layout for a first-generation model-B Raspberry Pi BMC)
[Picture - wiring diagram]
### Cabling it up ### Cabling it up
The actual cabling is a little more complex. Mainly because, even with a BMC-managed system, you sometimes want to press the physical power button, or see the system is on via the chassis power LED. This is easily done by using a combination of female-female and male-male jumper cables. Insert a male-male into top side of a female-female cable at one end, and voila! A male-ended lead for the chassis switch or LED, and a female-ended lead for connecting to your resistor/transistor/Raspberry Pi as you wish. Some electrical tape to keep everything snug and secure, and we have the perfect DIY out-of-band management solution! The actual cabling is kept simple using a solderable breadboard and way too many hours. Each cell features a total of 26 header pins for female-to-female jumper cables, along with the two transistors, two resistors, and the MAS3232 daughter chip. This unit keeps all the cabling neat and consistent between all three of my systems, and makes documenting the connections a breeze!
(Pictured: the cabling of the Raspberry Pi BMC) The board layout is straightforward and rendered here in ASCII for convenience, where each line (a connection), character (Transistor, Resistor), or number (header pin) represents one hole on the breadboard. The transistor leads are labeled Emitter, Base, and Collector.
```
[01][02][03][04][05][06][07][08][09][10]
| | | | | | | | | |
[11][12][13][14] | | . . . .
| | | | | | |---|---|---|
E \ / C E \ / C | | | |
T T | | | MAX |
B / /---/ B | | | 3232 |
| | | | | BOARD |
| | [15][16] | | | |
| | | | | | |---|---|---|
| | . | . | . . . .
| | R | R | | | | |
[17][18][19][20][21][22][23][24][25][26]
Motherboard connectors:
01: Reset ground 02: Reset signal
02: Power ground 04: Power signal
05: Power LED ground 06: Power LED signal
07: RS232 +3.3v 08: RS232 TX
09: RS232 RX 10: RS232 ground
Chassis connectors:
11: Reset switch ground 12: Reset switch signal
13: Power switch ground 14: Power switch signal
15: Chassis LED ground 16: Chassis LED signal
Raspberry Pi connectors:
17: Reset GPIO [0] 18: Power GPIO [1]
19: GPIO ground 20: Power LED GPIO [3]
21: GPIO ground 22: Power State GPIO [2]
23: TTL +3.3v 24: TTL RX
25: TTL TX 26: TTL ground
```
The finished product is a small board that keeps all the cabling neat and tidy in the case, and is easily mounted. While my soldering job is attrocious, they work!
(Pictured: the finished breadboards and the cabling of the Raspberry Pi BMC)
[Picture - the cabling] [Picture - the cabling]
## The software ## The software
Most traditional BMCs are managed via a (terrible) Web UI, which is often prone to breakage or other failures. And often these BMCs are incredibly complex, featuring dozens of menus and gigantic Flash monstrosities just to view their state. By using the Raspberry Pi, a seeming limitation - SSH-only access - becomes a benefit: it becomes trivial to connect to the BMC from a terminal, check the power state or serial console, turn on/off/reset a host, and then disconnect. No crummy web page or point-and-click menu system! Most traditional BMCs are managed via a (usually-terrible) Web UI, which is often prone to breakage of various kinds. And often these Web UIs are incredibly complex, featuring dozens of menus and gigantic Flash monstrosities just to view their state. By using the Raspberry Pi, a seeming limitation - SSH-only access - becomes a benefit: it becomes trivial to connect to the BMC from a terminal, check the power state or serial console, turn on/off/reset a host, and then disconnect. No crummy Flash web page, Java plugins, or slow point-and-click menu system!
The software side started out as a basic Raspbian system, however I wanted to make it a little more "BMC-like", in a sense stripped-down and easy-to-use with a small set of commands. I started by writing a simple "shell" emulator in BASH, and using a constant loop and hostname prompt along with `stty` to keep it focused and running. Each command triggers a function which performs its specific job and then returns to the "shell". While `bash` is available also, it should rarely be needed. The software side started out as a basic Raspbian system, however I wanted to make it a little more "BMC-like", in a sense stripped-down and easy-to-use with a small set of commands. I started by writing a simple "shell" emulator in BASH, and using a constant loop and hostname prompt along with `stty` to keep it focused and running. Each command triggers a function which performs its specific job and then returns to the "shell". While `bash` is available also, it should rarely be needed.
The actual software doing the heavy lifting is a combination of `screen`, to view the host system serial console, and the `gpio` utility by WiringPi (Debian package `wiringpi`). The `screen` session is configured to start automatically at BMC boot to ensure all serial output is captured and stored for later analysis - a major problem with the (few) SSH-based BMCs I've tried! The `gpio` program makes writing and reading the GPIO pins simple and easy, returning 0 or 1 for the low/high states and easily writing states. By writing the BMB shell in `bash`, I was able to get all the flexibility I wanted without any programming overhead; the whole thing is under 200 lines. The actual software doing the heavy lifting is a combination of `screen`, to view the host system serial console, and the `gpio` utility by WiringPi (Debian package `wiringpi`). The `screen` session is configured to start automatically at BMC boot via `rc.local`, to ensure all serial output is captured and stored for later analysis, even from cold boot - a major problem with the few SSH-based BMCs I've tried! The `gpio` program makes writing and reading the GPIO pins simple and easy, returning 0 or 1 for the low/high states and easily writing states. By writing the BMC shell in `bash`, I was able to get all the flexibility I wanted without any programming overhead; the whole thing is under 200 lines including all the functions and a little prep work ensuring the packages are active. The whole code of the `bmc.sh` utility script can be found [on my GitHub](https://github.com/joshuaboniface/rpibmc), and is fairly self-explanatory.
(Pictured: an example session with `bmc.sh`) The BMC is accessible via a user called `bmc`, which has the `bmc.sh` script as its login shell, as well as the groups required to run the various functions (`sudo` and `gpio` specifically). The `/etc/sudoers` file has also been edited to allow sudo without a password, which is used within the `bmc.sh` script. The new user can be created and configured on fresh install of Raspbian using these commands, and assuming `bmc.sh` is in `/bin`:
![Shell example](/images/bmcshell.png)
For example, here is the `powersw` function, which manages the power button for the `powersw` and `kill` commands. By introducing a delay, we can even simulate a power-button force-off event for the `kill` task and avoid messing with the raw power:
``` ```
powersw() { sudo sed -i 's/%sudo\tALL=(ALL:ALL) ALL/%sudo\tALL=(ALL:ALL) NOPASSWD: ALL/' /etc/sudoers
if [ "$1" == "hard" ]; then sudo useradd -g sudo -G gpio -s /bin/bmc.sh -d /home/bmc -m bmc
delay=10 sudo chpasswd <<<"bmc:password"
echo -n "Holding power switch... "
else
delay=0
echo -n "Pressing power switch... "
fi
gpio mode 1 out
gpio write 1 1
sleep $delay
gpio write 1 0
sleep 1
echo "done."
}
``` ```
Simply by setting the GPIO high, we simulate a button press, and then by setting the GPIO low, we release the simulated button. The result is a very convincing replica of the standard IPMI/BMC power control functionality. Another feature is to obtain the current system power status (shown multiple times in the image above). This function reads instead of writes the GPIO, in this case the front power LED, and reports the result, colour and all: Finally we're able to set the host system's name (for display when logging in) via the file `/etc/bmchost`, which makes deploying an image and setting the name trivial. Log in as the 'bmc' user via SSH, and observe:
``` *(Pictured: an example session with `bmc.sh`)*
readpower() {
gpio mode 2 in
powerstate_raw=$(gpio read 2)
if [ "${powerstate_raw}" -eq 1 ]; then
powerstate="\e[32mOn\e[0m"
else
powerstate="\e[31mOff\e[0m"
fi
}
```
The entire code of the BMC "shell" can be found on my [GitHub](https://github.com/joshuaboniface), and is called via a small hack to the `/etc/passwd` file for the `bmc` user, starting by default and terminating the SSH session on exit; to set this when creating a new user, simply pass `useradd` the `-S /path/to/bmc.sh` flag instead: ![BMC example](/images/bmc.sh.png)
``` *(Pictured: Debian Live via the serial console)*
bmc:x:0:0:bmc:/home/bmc:/bin/bmc.sh
``` ![Console example](/images/console.png)
```
useradd -g sudo -G gpio -S /bin/bmc.sh bmc
```
Finally we're able to set the host system's name (for display when logging in) via the file `/etc/bmchost`, which makes deploying an image and setting the name trivial.
## Conclusion ## Conclusion
I hope you've found this post interesting and useful - if you have some IPMI-less systems you want to manage remotely, I definitely recommend you try it out. So far my testing has been extremely positive; for under $30 (if you can find the Raspberry Pi for the right price), you can build youself out-of-band BMC management for any motherboard you can find. And the remote management makes even the most irritating host messups ("Oh broke my udev network rules, again!?") trivial to recover from. A small price to pay for the peace of mind of being able to manage your system from almost anywhere, even a cruise ship! I hope you've found this post interesting and useful - if you have some IPMI-less systems you want to manage remotely, I definitely recommend you try it out. So far my testing has been extremely positive; for under $40 (if you can find the Raspberry Pi for the right price; a first generation is more than powerful enough), you can build youself out-of-band BMC management for any motherboard you can find. And the remote management makes even the most irritating host messups ("Oh broke my udev network rules, again!?") trivial to recover from. A small price to pay for the peace of mind of being able to manage your system from almost anywhere, even a cruise ship!
(Pictured: what you might have to do without a Raspberry Pi BMC!) *(Pictured: what you might have to do on a cruise ship without a BMC!)*
[Picture - texting my dad to reboot ceph2 from a cruise ship]
![No BMC fail](/images/nobmcfail.png)
If you have any questions or comments, shoot me an e-mail! If you have any questions or comments, shoot me an e-mail!