329 lines
15 KiB
Markdown
329 lines
15 KiB
Markdown
# OpenBMC Server Power Recovery
|
|
|
|
Author: Andrew Geissler (geissonator)
|
|
|
|
Other contributors:
|
|
|
|
Created: October 11th, 2021
|
|
|
|
## Problem Description
|
|
|
|
Modern computer systems have a feature, automated power-on recovery, which in
|
|
essence is the ability to tell your system what to do when it hits issues with
|
|
power to the system. If the system had a black out (i.e. power was completely
|
|
cut to the system), should it automatically power the system on? Should it leave
|
|
it off? Or maybe the user would like the system to go to whichever state it was
|
|
at before the power loss.
|
|
|
|
There are also instances where the user may not want automatic power recovery to
|
|
occur. For example, some systems have op-panels, and on these op-panels there
|
|
can be a pin hole reset. This is a manual mechanism for the user to force a hard
|
|
reset to the BMC in situations where it is hung or not responding. In these
|
|
situations, the user may wish for the system to not automatically power on the
|
|
system, because they want to debug the reason for the BMC error.
|
|
|
|
During blackout scenarios, system owners may have a set of services they need
|
|
run once the power is restored. For example, IBM requires all LED's be toggled
|
|
to off in a blackout. OpenBMC needs to provide a mechanism for system owners to
|
|
run services in this scenario.
|
|
|
|
A brownout is another scenario that commonly utilizes automated power-on
|
|
recovery features. A brownout is a scenario where BMC firmware detects (or is
|
|
told) that chassis power can no longer be supported, but power to the BMC will
|
|
be retained. On some systems, it's desired to utilize the automated power-on
|
|
feature to turn chassis power back on as soon as the brownout condition ends.
|
|
|
|
Some system owners may chose to attach an Uninterrupted Power Supply (UPS) to
|
|
their system. A UPS continues to provide power to a system through a blackout or
|
|
brownout scenario. A UPS has a limited amount of power so it's main purpose is
|
|
to handle brief power interruptions or to allow for an orderly shutdown of the
|
|
host firmware.
|
|
|
|
The goal of this design document is to describe how OpenBMC firmware will deal
|
|
with these questions.
|
|
|
|
## Background and References
|
|
|
|
The BMC already implements a limited subset of function in this area. The
|
|
[PowerRestorePolicy][pdi-restore] property out in phosphor-dbus-interface
|
|
defines the function capability.
|
|
|
|
In smaller servers, this feature is commonly found within the Advanced
|
|
Configuration and Power Interface (ACPI).
|
|
|
|
[openbmc/phosphor-state-manager][state-mgr] supports this property as defined in
|
|
the phosphor-dbus-interface.
|
|
|
|
## Requirements
|
|
|
|
### Automated Power-On Recovery
|
|
|
|
OpenBMC software must ensure it persists the state of power to the chassis so it
|
|
can know what to restore it to if necessary
|
|
|
|
OpenBMC software must provide support for the following options:
|
|
|
|
- Do nothing when power is lost to the system (this will be the default)
|
|
- Always power the system on and boot the host
|
|
- Always power the system off (previous power was on, power is now off, run all
|
|
chassis power off services to ensure a clean state of software and hardware)
|
|
- Restore the previous state of the chassis power and host
|
|
|
|
These options are only checked and enforced in situations where the BMC does not
|
|
detect that chassis power is already on to the system when it comes out of
|
|
reboot.
|
|
|
|
OpenBMC software must also support the concept of a one_time power restore
|
|
policy. This is a separate instance of the `PowerRestorePolicy` which will be
|
|
hosted under a D-Bus object path which ends with "one_time". If this one_time
|
|
setting is not the default, `None`, then software will execute the policy
|
|
defined under it, and then reset the one_time property to `None`. This one_time
|
|
feature is a way for software to utilize automated power-on recovery function
|
|
for other areas like firmware update scenarios where a certain power on behavior
|
|
is desired once an update has completed.
|
|
|
|
### BMC and System Recovery Paths
|
|
|
|
In situations where the BMC or the system have gotten into a bad state, and the
|
|
user has initiated some form of manual reset which is detectable by the BMC as
|
|
being user initiated, the BMC software must:
|
|
|
|
- Fill in appropriate `RebootCause` within the [BMC state interface][bmc-state]
|
|
- At a minimum, `PinholeReset` will be added. Others can be added as needed
|
|
- Log an error indicating a user initiated forced reset has occurred
|
|
- Not log an error indicating a blackout has occurred if chassis power was on
|
|
prior to the pin hole reset
|
|
- Not implement any power recovery policy on the system
|
|
- Turn power recovery back on once BMC has a normal reboot
|
|
|
|
### Blackout
|
|
|
|
A blackout occurs when AC power is cut from the system, resulting in a total
|
|
loss of power if there is no UPS installed to keep the system on. To identify
|
|
this scenario after a BMC reboot, chassis-state-manager will check to see what
|
|
the last power state was before the loss of power and compares it against the
|
|
pgood pin. Blackouts can be intentionally triggered by a user (i.e a pinhole
|
|
reset) or in severe cases occur when there is some sort of an external outage.
|
|
In either case the BMC must take into account this detrimental state. When this
|
|
condition occurs, the BMC may(depending on configuration):
|
|
|
|
- Provide a generic target, `obmc-chassis-blackout@.target` to be called when a
|
|
blackout is detected
|
|
- Adhere to the current power restore policy
|
|
|
|
BMC firmware must also be able to:
|
|
|
|
- Discover why the system is in a blackout situation. From either loss of power
|
|
or user actions.
|
|
|
|
### Brownout
|
|
|
|
As noted above, a brownout condition is when AC power can not continue to be
|
|
supplied to the chassis, but the BMC can continue to have power and run.
|
|
|
|
When this condition occurs, the BMC must:
|
|
|
|
- Power system off as quickly as situations requires (or gracefully handle the
|
|
loss of power if it occurred without warning)
|
|
- Log an error indicating the brownout event has occurred
|
|
- Support the ability for host firmware to indicate a one-time power restore
|
|
policy if they wish for when the brownout completes
|
|
- Identify when a brownout condition has completed
|
|
- Wait for the brownout to complete and implement the one-time power restore
|
|
policy. If no one-time policy is defined then run the standard power restore
|
|
policy defined for the system
|
|
|
|
BMC firmware must also be able to:
|
|
|
|
- Discover if system is in a brownout situation
|
|
- Run when the BMC first comes up to know if it should implement any automated
|
|
power-on recovery
|
|
- Not run any power-on recovery logic when a brownout is occurring
|
|
- Tell the host firmware that it is a automated power-on recovery initiated boot
|
|
when that firmware is what boots the system
|
|
|
|
### Uninterruptible Power Supply (UPS)
|
|
|
|
When a UPS is present and a blackout or brownout condition occurs, the BMC must:
|
|
|
|
- Log an error to indicate the condition has occurred
|
|
- If host firmware is running, notify the host firmware of this utility failure
|
|
condition (this behavior is build-time configurable)
|
|
- If the UPS battery power becomes low and if host firmware is running, notify
|
|
the host firmware of the condition, indicating a quick power off is required
|
|
(this behavior is build-time configurable)
|
|
- Log an error if the UPS battery power becomes low and a power loss to the
|
|
entire system is imminent(i.e. a blackout scenario where BMC will also lose
|
|
power and UPS is about to run out of power)
|
|
- Not execute any automated power-on recovery logic to prevent power on/off
|
|
thrasing (this behavior is build-time configurable)
|
|
|
|
## Proposed Design
|
|
|
|
### Automated Power-On Recovery
|
|
|
|
An application will be run after the chassis and host states have been
|
|
determined which will only run if the chassis power is not on.
|
|
|
|
This application will look for the one_time setting and use it if its value is
|
|
not `None`. If it does use the one_time setting then it will reset it to `None`
|
|
once it has read it. Otherwise the application will read the persistent value of
|
|
the `PowerRestorePolicy`. The application will then run the logic as defined in
|
|
the Requirements above.
|
|
|
|
This function will be hosted in phosphor-state-manger and potentially
|
|
x86-power-control.
|
|
|
|
### BMC and System Recovery Paths
|
|
|
|
The BMC state manager application currently looks at a file in the sysfs to try
|
|
and determine the cause of a BMC reboot. It then puts this reason in the
|
|
`RebootCause` property.
|
|
|
|
One possible cause of a BMC reset is an external reset (EXTRST). There are a
|
|
variety of reasons an external reset can occur. Some systems are adding GPIOs to
|
|
provide additional detail on these types of resets.
|
|
|
|
A new GPIO name will be added to the [device-tree-gpio-naming.md][dev-tree]
|
|
which reports whether a pin hole reset has occurred on the previous reboot of
|
|
the BMC. The BMC state manager application will enhance its support of the
|
|
`RebootCause` to look for this GPIO and if present, read it and set
|
|
`RebootCause` accordingly when it can either not determine the reason for the
|
|
reboot via the sysfs or sysfs reports a EXTRST reason (in which case the GPIO
|
|
will be utilized to enhance the reboot reason).
|
|
|
|
If the power recovery software sees the `PinholeReset` reason within the
|
|
`RebootCause` then it will not implement any of its policy. Future BMC reboots
|
|
which are not pin hole reset caused, will cause `RebootCause` to go back to a
|
|
default and therefore power recovery policy will be re-enabled on that BMC boot.
|
|
|
|
The phosphor-state-manager chassis software will not log a blackout error if it
|
|
sees the `PinholeReset` reason (or any other reason that indicates a user
|
|
initiated a reset of the system).
|
|
|
|
### Blackout
|
|
|
|
A new systemd target `obmc-chassis-blackout.target` should be added to allow
|
|
system maintainers to call services in this condition. This new target will be
|
|
called when the BMC detects a blackout. The target will allow for system owners
|
|
to add their own specific services to this new target.
|
|
Phosphor-chassis-state-manager will ensure `obmc-chassis-blackout.target` will
|
|
be called after a blackout.
|
|
|
|
### Brownout
|
|
|
|
The existing `xyz.openbmc_project.State.Chassis` interface will be enhanced to
|
|
support a `CurrentPowerStatus` property. The existing
|
|
phosphor-chassis-state-manager, which is instantiated per instance of chassis in
|
|
the system, will support a read of this property. The following will be the
|
|
possible returned values for the power status of the target chassis:
|
|
|
|
- `Undefined`
|
|
- `BrownOut`
|
|
- `UninterruptiblePowerSupply`
|
|
- `Good`
|
|
|
|
The phosphor-psu-monitor application within the phosphor-power repository will
|
|
be responsible for monitoring for brownout conditions. It will support a
|
|
per-chassis interface which represents the status of the power going into the
|
|
target chassis. This interface will be generic in that other applications could
|
|
host it to report the status of the power. The state-manager software will
|
|
utilize mapper to look for all implementations of the interface for its chassis
|
|
and aggregate the status (i.e. if any reports a brownout, then `BrownOut` will
|
|
be returned). This interface will be defined in a later update to this document.
|
|
|
|
The application(s) responsible for detecting and reporting chassis power will
|
|
run on startup and discover the correct state for their property. These
|
|
applications will log an error when a brownout occurs and initiate the fast
|
|
power off.
|
|
|
|
If the system design needs it, the existing one-time function provided by
|
|
phosphor-state-manager for auto power on policy will be utilized for when the
|
|
brownout completes.
|
|
|
|
When the phosphor-power application detects that a brownout condition has
|
|
completed it will reset its interface representing power status to good and
|
|
start the state-manager service which executes the automated power-on logic.
|
|
|
|
phosphor-state-manager will ensure automated power-on recovery logic is only run
|
|
when the power supply interface reports the power status is good. If there are
|
|
multiple chassis and/or host instances in the system then the host instances
|
|
associated with the chassis(s) with a bad power status will be the only ones
|
|
prevented from booting.
|
|
|
|
### Uninterruptible Power Supply (UPS)
|
|
|
|
A new phosphor-dbus-interface will be defined to represent a UPS. A BMC
|
|
application will implement one of these per UPS attached to the system. This
|
|
application will monitor UPS status and monitor for the following:
|
|
|
|
- UPS utility fail (system power has failed and UPS is providing system power)
|
|
- UPS battery low (UPS is about to run out of power)
|
|
|
|
If the application sees power has been lost and the system is running on UPS
|
|
battery power then it will monitor for the power remaining in the UPS and notify
|
|
the host that a shutdown is required if needed. This application will also be
|
|
responsible for logging an error indicating the UPS backup power has been
|
|
switched to and set the appropriate property in their interface to indicate the
|
|
scenario is present when the system can no longer remain on.
|
|
phosphor-state-manager will query mapper for implementation of this new UPS
|
|
interface and utilize them in combination with power supply brownout status when
|
|
determining the value to return for its `CurrentPowerStatus`.
|
|
|
|
Similar to the above brownout scenario, phosphor-state-manager will ensure
|
|
automated power-on recovery logic is not run if `PowerStatus` is not set to
|
|
`Good`. This behavior will be build-time configurable within
|
|
phosphor-state-manager.
|
|
|
|
## Alternatives Considered
|
|
|
|
None, this is a pretty basic feature that does not have a lot of alternatives
|
|
(other then just not doing it).
|
|
|
|
## Impacts
|
|
|
|
None
|
|
|
|
## Testing
|
|
|
|
The control of this policy can already bet set via the Redfish API.
|
|
|
|
```
|
|
# Power Restore Policy
|
|
curl -k -H "Content-Type: application/json" -X PATCH -d '{"PowerRestorePolicy":"AlwaysOn"}' https://${bmc}/redfish/v1/Systems/system
|
|
curl -k -H "Content-Type: application/json" -X PATCH -d '{"PowerRestorePolicy":"AlwaysOff"}' https://${bmc}/redfish/v1/Systems/system
|
|
curl -k -H "Content-Type: application/json" -X PATCH -d '{"PowerRestorePolicy":"LastState"}' https://${bmc}/redfish/v1/Systems/system
|
|
```
|
|
|
|
For testing, each policy should be set and verified. The one_time aspect should
|
|
also be checked for each possible value and verified to only be used once.
|
|
|
|
Validate that when multiple black outs occur, the firmware continues to try and
|
|
power on the system when policy is `AlwaysOn` or `Restore`.
|
|
|
|
On supported systems, a pin hole reset should be done with a system that has a
|
|
policy set to always power on. Tester should verify system does not
|
|
automatically power on after a pin hole reset. Verify it does automatically
|
|
power on when a normal reboot of the BMC is done.
|
|
|
|
A brownout condition should be injected into a system and appropriate paths
|
|
should be verified:
|
|
|
|
- Error log generated
|
|
- Host notified (if running and notification possible)
|
|
- System quickly powered off
|
|
- Power recovery function is not run while a brownout is present
|
|
- System automatically powers back on when brownout condition ends (assuming a
|
|
one-time or system auto power-on recovery policy of `AlwaysOn` or `Restore`)
|
|
|
|
Plug a UPS into a system and ensure when power is cut to the system that an
|
|
error is logged and the host is notified and allowed to power off.
|
|
|
|
[pdi-restore]:
|
|
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/yaml/xyz/openbmc_project/Control/Power/RestorePolicy.interface.yaml
|
|
[state-mgr]: https://github.com/openbmc/phosphor-state-manager
|
|
[bmc-state]:
|
|
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/yaml/xyz/openbmc_project/State/BMC.interface.yaml
|
|
[dev-tree]:
|
|
https://github.com/openbmc/docs/blob/master/designs/device-tree-gpio-naming.md
|