Configuring IPMI / BMC management

Every managed node on the SC//Reliant™ platform has a Baseboard Management Controller (BMC) — the out-of-band management processor that responds to Intelligent Platform Management Interface (IPMI) commands. Depending on your hardware vendor you may know it as the BMC, XCC (Lenovo), or iLO (HPE). SC//Reliant Platform Manager (RPM) manages that controller declaratively through a single property named IPMI.

This page covers what the IPMI property configures, the prerequisites your BMC must meet, how the IPMI credential is stored and rotated, and how to verify the result. For background on what the BMC network is for, how it is addressed, and why it is segmented the way it is, see The out-of-band management network.

Sections below describe behavior common to all supported hardware, except for Hardware-specific notes and the Applies to column of the troubleshooting table, which call out vendor-specific items. Do not apply a fix from one vendor's subsection to another vendor's hardware.

What the IPMI property manages

When the IPMI property is set on a node group, every node in that group has its BMC brought into the state the property describes on each configuration run — that is, on each Puppet run on that node. For the ways to trigger one, see Deploying changes to your environment. Specifically, the platform manages:

• The BMC user account — the username, user ID, password, and channel of the account that should exist.
• BMC LAN configuration — the management IP address, netmask, gateway, operating mode, channel, and address type (static or DHCP).

Because the configuration is applied from inside the operating system on each node, no separate network path to the BMC is required to push the settings. The BMC does need to be reachable over the network afterwards for the platform to use it, which is what the prerequisites in the next section cover.

The credential defined here is also the credential a cluster uses to fence (STONITH) its peers. That makes the IPMI property a dependency of cluster high availability, not just a convenience for remote console access. For background on fencing, see Pacemaker and Corosync.

Before you begin: BMC prerequisites

On most platforms the interfaces the configuration process needs are disabled by default. Confirm the following in the BMC's own web interface before assigning the IPMI property, otherwise the configuration run fails.

Requirements for the fencing user

These three requirements apply to every supported BMC, regardless of vendor. For a cluster to fence a node over IPMI, all three must be true:

1. IPMI over LAN must be enabled. Fencing is an out-of-band operation — one node sends an IPMI command across the network to the target node's BMC. If IPMI over LAN is disabled, the BMC does not answer, and fencing fails silently from the caller's point of view.
2. KCS must be enabled. The Keyboard Controller Style interface is the local, in-band channel between the host operating system and its own BMC. RPM uses KCS to create and update the BMC user in the first place. With KCS disabled, the platform cannot configure the user at all.
3. The user must have permission to reset the device and read its status. In IPMI terms, that means chassis control (power off, power cycle, reset) and chassis status. RPM creates its managed user at the Administrator privilege level, which satisfies this.

KCS and IPMI over LAN are separate settings and are usually found on separate pages of the BMC interface. Enabling one does not enable the other.

General settings worth checking

Hardware-specific notes

The items below are vendor-specific. They are not general IPMI behavior and should not be applied to hardware other than the vendor they are listed under.

Lenovo XClarity Controller (XCC), including ThinkEdge SE100

Lenovo XCC ships in a more restrictive state than the other supported BMCs and needs additional configuration beyond the two interface toggles above. All of the following are specific to XCC:

Supermicro

Supermicro BMC firmware is comparatively permissive and needs no special handling. In practice, only the two interface requirements in Requirements for the fencing user need attention, and no security-mode change is required.

HPE iLO

iLO needs IPMI over LAN enabled as described above, and no security-mode change. It has one behavior of its own:

Password requirements

The IPMI password must satisfy two separate sets of rules at once: the limits of the IPMI protocol itself, and whatever complexity policy your BMC vendor enforces. Pick a password that satisfies both.

Protocol limits

These are properties of the IPMI protocol and apply to all hardware.

A password longer than 20 characters is not rejected by every BMC — some accept it and store only the first 20 bytes. The password then appears to be set, but authentication fails. Never exceed 20 characters.

Recommended minimum

The following baseline satisfies the protocol limit and the published complexity policies of every supported BMC vendor, so a single password standard works across a mixed fleet:

• 12 to 20 characters.
• At least one uppercase letter, one lowercase letter, and one number.
• No spaces.
• Not the username, a repetition of the username, or the username reversed.
• No more than two consecutive identical characters (for example, aaa is rejected).

Special characters and per-vendor policy

Special characters are the one place where vendor guidance actively conflicts, so treat them with care.

Practical guidance. A 12–20 character password using only uppercase letters, lowercase letters, and digits satisfies both policies above and avoids the shell-quoting and firmware-parsing problems that special characters introduce in an IPMI password. If your own security policy mandates a special character, prefer characters with no shell meaning — for example -, _, ., =, + — and avoid #, $, !, &, *, backtick, and quotation marks.

The IPMI property

The IPMI property is assigned to a node group and inherited by every node in it. For where it sits among the other node group properties, see Managing node groups.

The property is a nested structure with two top-level sections, Users and Networks. Each has its own Enable this option toggle; a section that is toggled off is not configured at all. Each section holds exactly one entry — one BMC account and one BMC network — and the title of that entry carries meaning: it is not a label you are free to choose arbitrarily.

Editing and saving the property

The property is edited as a single unit. Select Edit on the IPMI property to open the edit screen, make your changes across both sections, then Save and complete the confirmation. There is one Save for the entire property, not one per section or per field. For what the confirmation covers on a node group change, see Updating a node group.

Saving stores the configuration; it does not apply it. Nothing reaches a node's BMC until the next configuration run on that node.

Users

The entry title under Users is the username of the BMC account being created. If that title is reladmin, the account created on the BMC is named reladmin. Beneath the title are the account's remaining attributes.

The Users section is authoritative. Every BMC user slot other than the one declared here is explicitly disabled on each configuration run. Because the section holds a single entry, the managed account is the only BMC account that persists — an account created by hand in the BMC web interface is disabled the next time configuration runs.

The account declared in this section is created at the Administrator privilege level, which is what satisfies requirement 3 in Requirements for the fencing user.

Networks

As with Users, the entry title under Networks carries the value: it is the IP address assigned to the BMC's management interface, written without a CIDR suffix. Everything below the title describes how that address is configured.

The Channel value ties the two sections together, so the channel set here and the channel set on the user should match. The user is configured only after the network on its channel has been configured successfully. If the network entry fails, the user on that channel is skipped rather than half-applied.

Property references in the IPMI property

Each field in the IPMI property can hold either a Raw Value, typed in directly, or a Property Reference that points at another property and resolves to that property's value when configuration is applied. Use the dropdown on the field to choose which one you are providing. For how the two differ and how the property hierarchy resolves them, see Using raw values vs property references.

Three fields in this property are normally references rather than raw values:

The remaining fields — id, Channel, Mode, Network Type, and Netmask — are typically raw values, and the prefilled platform defaults are usually correct. Any of them can take a property reference instead, which is worth considering where the value has to vary from node to node.

Both references in the Networks section follow the same two-part pattern, because the values they point at belong to individual nodes:

1. On the node group's IPMI property, set the entry title to reference ipmi ip and Gateway to reference Bridge0 Gateway.
2. Add both properties to each node in that group, setting them to that node's BMC address and gateway.

A reference with no value on the node has nothing to resolve to. Note that ipmi ip can only be added to a node in the first place, so its value never lives at the node group level.

The IPMI Password property

The password itself is held in a dedicated property. This is the property that the Password field of the IPMI property references, and it is exposed for scripted access as my_ipmi_password.

The value is masked in the interface. Generate Password produces a compliant random value so you do not have to construct one by hand; use it unless you have a specific reason to supply your own.

Rotating the IPMI password

Because both the BMC user and the cluster's fencing configuration read this one property, rotation is a single change rather than a coordinated multi-step operation:

1. Set a new value on the my_ipmi_password property at the appropriate scope, either in SC//Reliant Platform Manager or programmatically through the RPM API. See Rotating through the RPM API.
2. Trigger a configuration run against the affected nodes. See Deploying changes to your environment.
3. On that run, the BMC user's password is updated to the new value and the fencing configuration is rewritten to match, in the same pass.

Always rotate through the property, never directly on the BMC. Changing the password in the BMC web interface leaves the value stored in RPM stale. Fencing keeps presenting the old credential and fails — and because fencing failures surface only when a node actually needs to be fenced, the problem tends to be discovered at the worst possible moment. The next configuration run also reverts the BMC to the value held in the property.

Rotate the property and run configuration across all nodes in a cluster together. A cluster whose nodes hold different IPMI passwords cannot fence consistently.

The password requirements in Password requirements apply to a rotated password exactly as they do to the initial one, whichever method you use. The API stores whatever value it is given. A 30-character password with shell metacharacters in it is accepted by the property and then silently truncated or rejected by the BMC.

Rotating through the RPM API

Setting my_ipmi_password through the API is what makes rotation scriptable and schedulable. A fleet-wide rotation becomes a loop rather than a series of manual edits, and the same call can run from a pipeline or a scheduled job on a fixed interval. Property writes made this way are recorded in the audit trail with the item, field, previous value, current value, user, and timestamp, the same as a change made in the interface.

For the authentication token, secret key, signature, and required headers, see API access requirements. Everything below is specific to rotating the IPMI password.

Choosing the scope

The property can be set on a node group or on an individual node. The choice determines how many BMCs the change reaches.

Both scopes write to the same audit trail, and the entry identifies which one was used.

The request

Both endpoints take the same body and return the same response shape.

Note that the node endpoint keys on the node hostname, not on a numeric ID.

The body sets the one property:

{"properties":{"my_ipmi_password":"<new_password>"}}

A successful call returns HTTP 200 along with the updated object, including its properties block. Confirm the new value in that block rather than relying on the status code alone.

Examples

Rotate the password for every node in a node group:

ruby api.rb <customer_code> put api/node_groups/<node_group_id> '{"properties":{"my_ipmi_password":"<new_password>"}}'

Rotate the password on a single node:

ruby api.rb <customer_code> put api/nodes/<node_hostname> '{"properties":{"my_ipmi_password":"<new_password>"}}'

Scripting the rotation

The following script rotates the password on a node group, reading the API token and secret key from the environment and confirming the change against the value the API echoes back. The single-node equivalent is identical apart from the path, which keys on hostname rather than a numeric ID.

#!/bin/bash
CUSTOMER="<customer_code>"
NODE_GROUP_ID="$1"
NEW_PASSWORD="$2"
PATH_ONLY="/api/node_groups/${NODE_GROUP_ID}"
BODY=$(printf '{"properties":{"my_ipmi_password":"%s"}}' "${NEW_PASSWORD}")
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%S+00:00")
SIGNATURE=$(printf '%s %s' "${PATH_ONLY}" "${TIMESTAMP}" \
  | openssl dgst -sha256 -hmac "${SECRET_KEY}" -r | awk '{print $1}')
RAW=$(curl -sS -w "\n%{http_code}" -X PUT "https://platform.reliant.io${PATH_ONLY}" \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "X-Auth-Signature: ${SIGNATURE}" \
  -H "X-Timestamp: ${TIMESTAMP}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.reliantsec-v1+json" \
  -H "X-Customer: ${CUSTOMER}" \
  -d "${BODY}")
HTTP_CODE=$(echo "${RAW}" | tail -n1)
RESPONSE=$(echo "${RAW}" | sed '$d')
if [ "${HTTP_CODE}" != "200" ]; then
  echo "Request failed (HTTP ${HTTP_CODE})"
  echo "${RESPONSE}"
  exit 1
fi
RETURNED_PASSWORD=$(echo "${RESPONSE}" | jq -r '.properties.my_ipmi_password')
if [ "${RETURNED_PASSWORD}" == "${NEW_PASSWORD}" ]; then
  echo "Updated successfully: my_ipmi_password matches"
else
  echo "Mismatch: API returned '${RETURNED_PASSWORD}', expected '${NEW_PASSWORD}'"
  exit 1
fi

Run it with the token and secret key exported and the target and new password as arguments:

export TOKEN='<api_token>'
export SECRET_KEY='<api_secret_key>'
./ipmi_group_update.sh <node_group_id> '<new_password>'

Passing the password as an argument puts it in shell history and in the process table while the script runs. On a shared host, read it from a file or prompt for it instead.

Three details of the signature account for most authentication failures during a scripted rotation:

• Sign the path only. The signed value is the URI path (for example, /api/nodes/<node_hostname>) with a leading slash and no scheme, host, or query string.
• Compute the timestamp once and use it twice. Store it in a variable and reuse it for both the signature and the X-Timestamp header. Calling date a second time eventually produces a different string and an intermittent, hard-to-diagnose rejection.
• Keep the clock accurate. The timestamp is part of the signed material and is checked for freshness, so a host with drifting time fails authentication with credentials that are otherwise correct.

What a rotation looks like on the node

On the next configuration run against an affected node, the new value is applied to both consumers in the same pass:

Notice: /Stage[main]/Profiles::Ipmi/Ipmi::User[user-reladmin]/Ipmiuser[user-reladmin]/password: changed [redacted] to [redacted]
Notice: /Stage[main]/Profiles::Cluster::Fence/Cs_primitive[fence_<node_hostname>]/parameters: parameters changed {
  'hostname' => '<node_hostname>',
  'interface' => 'lan2',
  'ipaddr' => '<bmc_ip>',
  'passwd' => 'Sensitive [value redacted]',
  'pcmk_delay_max' => '5s',
  'userid' => 'reladmin'
} to {
  'hostname' => '<node_hostname>',
  'ipaddr' => '<bmc_ip>',
  'passwd' => 'Sensitive [value redacted]',
  'pcmk_delay_max' => '5s',
  'userid' => 'reladmin',
  'interface' => 'lan2'
}

The Cs_primitive notice prints the entire parameter hash, so the change that matters (passwd, which is masked) is easy to miss. Note also that interface moves from the top of the hash to the bottom. That reordering is cosmetic: the hash is unordered, and the values are unchanged. It is not a second change and it does not indicate a resource flapping between runs.

Both notices appear only on a run where the value actually differs. Once every node has converged, subsequent runs report no changes.

Verifying a rotation

Three checks, in the order they become available:

Then confirm the new credential authenticates over the network on every node in the cluster, using ipmiutil health as described in Verifying the configuration. That is the check that most closely mirrors what fencing does, and it is the one worth running after any rotation.

What happens when configuration runs

On each configuration run, for every node in the group:

1. The BMC network settings for the declared channel are applied first — address type, address, netmask, gateway, mode, and channel.
2. The declared user is then created or updated on its channel at Administrator privilege, with the password resolved from the referenced property.
3. Every other BMC user slot is disabled.

The order matters: the user depends on the network for its channel having been configured successfully. If the network step fails, the user step is skipped rather than partially applied, which keeps a node from ending up with a user that cannot be reached.

Configuration is idempotent. Once a node's BMC matches the property, subsequent runs report no changes.

On HPE iLO, a successful run is not the last step. iLO accepts the network settings but does not activate them until the management controller is reset, so a run can report success while the BMC is still on its old address. Follow the run with a cold BMC reset as described in HPE iLO.

Verifying the configuration

Connect to the node with ssh <node_hostname> and run the following after a configuration run has completed. Both commands apply to all supported hardware.

Confirm the BMC LAN settings and the managed user:

sudo ipmiutil lan print

The output lists the configured IP address, netmask, and gateway, along with a per-channel user access line for each enabled account. The managed account should appear with Admin privilege on its channel.

Confirm the credential authenticates over the network:

sudo ipmiutil health -N <BMC_IP> -U <username> -F lan2 -Y

A successful run reports the BMC manufacturer, firmware version, IPMI version, and a selftest status of OK. This is the check that most closely mirrors what fencing does, so it is the one worth running after any password rotation. Use the command exactly as written; -F lan2 selects the IPMI 2.0 / RMCP+ connection and works on all supported hardware.

Troubleshooting

The Applies to column indicates which hardware a given symptom and fix are relevant to. A fix marked for one vendor should not be applied to another.

Related articles

• Deploying changes to your environment — how to trigger the configuration run that applies the IPMI property.
• Pacemaker and Corosync — how STONITH fencing uses the IPMI credential.
• Managing node groups — the full node group property list.
• Understanding configuration data — how properties and property references work.
• API access requirements — the token, secret key, signature, and headers needed to rotate the IPMI password through the API.
• The out-of-band management network — what the BMC network is for, its addressing model, and its security posture.
• Configuring networking — communication paths, including IPMI ports 623/UDP and 443/TCP.