[OpenNMS/opennms] d94d2e: NMS-20109: Active/passive HA for OpenNMS core (#8700)

  Branch: refs/heads/release-36.x
  Home:   https://github.com/OpenNMS/opennms
  Commit: d94d2e6ee87dbebf2b12df1581099710ead893a5
      https://github.com/OpenNMS/opennms/commit/d94d2e6ee87dbebf2b12df1581099710ead893a5
  Author: Chandra Gorantla <[email protected]>
  Date:   2026-08-05 (Wed, 05 Aug 2026)

  Changed paths:
    M container/features/src/main/resources/features.xml
    M container/karaf/src/main/filtered-resources/etc/custom.properties
    M core/daemon/pom.xml
    M core/daemon/src/main/java/org/opennms/netmgt/vmmgr/Invoker.java
    M core/daemon/src/main/java/org/opennms/netmgt/vmmgr/Manager.java
    M core/daemon/src/main/java/org/opennms/netmgt/vmmgr/Starter.java
    A core/daemon/src/main/java/org/opennms/netmgt/vmmgr/StartupLifecycleHook.java
    M docs/modules/operation/nav.adoc
    A docs/modules/operation/pages/deep-dive/admin/high-availability.adoc
    M docs/modules/reference/pages/karaf-shell/karaf-shell.adoc
    M features/events/syslog/src/test/java/org/opennms/netmgt/syslogd/SyslogMessageTest.java
    A features/ha-management/ha-api/pom.xml
    A features/ha-management/ha-api/src/main/java/org/opennms/netmgt/ha/HaConfiguration.java
    A features/ha-management/ha-api/src/main/java/org/opennms/netmgt/ha/HaInstanceState.java
    A features/ha-management/ha-api/src/main/java/org/opennms/netmgt/ha/HaMode.java
    A features/ha-management/ha-api/src/main/java/org/opennms/netmgt/ha/HaRole.java
    A features/ha-management/ha-daemon/pom.xml
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/DbConnectionFactory.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaConfigSyncer.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaHeartbeatWriter.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaStartupCoordinator.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaStartupLifecycleHook.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaStatusSchema.java
    A features/ha-management/ha-daemon/src/main/java/org/opennms/netmgt/ha/HaSyncFiles.java
    A features/ha-management/ha-daemon/src/main/resources/META-INF/services/org.opennms.netmgt.vmmgr.StartupLifecycleHook
    A features/ha-management/ha-daemon/src/test/java/org/opennms/netmgt/ha/DbConnectionFactoryTest.java
    A features/ha-management/ha-daemon/src/test/java/org/opennms/netmgt/ha/HaConfigSyncerTest.java
    A features/ha-management/ha-daemon/src/test/java/org/opennms/netmgt/ha/HaStartupCoordinatorTest.java
    A features/ha-management/ha-daemon/src/test/java/org/opennms/netmgt/ha/HaStartupLifecycleHookTest.java
    A features/ha-management/ha-rest/pom.xml
    A features/ha-management/ha-rest/src/main/java/org/opennms/netmgt/ha/rest/HaRestService.java
    A features/ha-management/ha-rest/src/main/java/org/opennms/netmgt/ha/rest/dto/HaInstanceStatusDto.java
    A features/ha-management/ha-rest/src/main/java/org/opennms/netmgt/ha/rest/dto/HaStatusCollectionDto.java
    A features/ha-management/ha-rest/src/main/java/org/opennms/netmgt/ha/rest/impl/HaRestServiceImpl.java
    A features/ha-management/ha-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml
    A features/ha-management/ha-shell/pom.xml
    A features/ha-management/ha-shell/src/main/java/org/opennms/netmgt/ha/shell/HaConfigCommand.java
    A features/ha-management/ha-shell/src/main/java/org/opennms/netmgt/ha/shell/HaFailoverCommand.java
    A features/ha-management/ha-shell/src/main/java/org/opennms/netmgt/ha/shell/HaStatusCommand.java
    A features/ha-management/pom.xml
    M features/pom.xml
    M opennms-base-assembly/pom.xml
    A opennms-base-assembly/src/main/filtered-meridian/etc/ha-configuration.xml
    M opennms-base-assembly/src/main/filtered/etc/opennms.service
    M opennms-full-assembly/pom.xml
    M opennms-webapp/src/main/webapp/WEB-INF/applicationContext-spring-security.xml

  Log Message:
  -----------
  NMS-20109: Active/passive HA for OpenNMS core (#8700)

* WIP; testing

(cherry picked from commit 99ab08c00edc9df5453e7f3c90b3fe62a456d527)

* WIP

(cherry picked from commit b81400fbf988a99bf2512d8b2512ea365c23de14)

* test and iterate, test and iterate, test and

(cherry picked from commit 203bf32e3b4cbba44e2117f11f7d4593effadf36)

* WIP

(cherry picked from commit 853cf8a9ec43f42814304754d047ea887eefc236)

* don't rely on local clock

(cherry picked from commit 7b39e322211f2b90905e2d7a0a97fa1a17c0de55)

* periodically re-read the ha-configuration.xml sow e can change values on the fly

(cherry picked from commit d04e0a9f0b7a52919f5f19f8b32c847faecf70ca)

* skip syncing files in examples/ dir

(cherry picked from commit a98638e6d45c78f75315461cc59c62997c37c2f1)

* rest and shell refinements, correct some log messages

(cherry picked from commit 77b60c30a44409184ebfea73204366b451b5a208)

* Fixups for the config shell command

(cherry picked from commit e7bc80fce5b76c37fb7d2fb974972fbdb4995682)

* terminate immediately on split-brain detected instead of manager shutdown

(cherry picked from commit 5f54bea669cbe69c67bbb578e94a66837aaa6d6d)

* split-brain improvements; add active_since timestamp for tracking who became active when, and expose that value in both karaf shell and in ha rest api.

(cherry picked from commit 726f4cad4f4deae0b806af35cebdcc9acc7897bd)

* Convert to ShellTable for niceness

(cherry picked from commit 2b0ef35822c4f3747aace4ddbc015c812e37047d)

* NMS-20109: Make HA management an optional feature

- Create ha_instance_status from the coordinator at startup (advisory-lock
  serialized, includes agent_last_seen) instead of the core Liquibase
  changelog
- Write only last_heartbeat from the heartbeat cycle; state columns change
  on transitions only
- Add <mode>heartbeat-only</mode>: an external supervisor owns the state
  machine and OpenNMS only publishes liveness, never gating startup
- Move the HA REST endpoints out of opennms-webapp-rest into the ha-rest
  OSGi whiteboard bundle; enforce authorization via Spring Security
  intercept-urls
- Replace the filesystem-API config sync with a binary manifest transfer
  (sha256 diff, atomic apply, deletion propagation, sync-excludes); the
  hasync account no longer needs ROLE_FILESYSTEM_EDITOR

* NMS-20109: Handle some corner cases

Fail-closed startup
Partner check before claiming ACTIVE
No assume-inactive-on-error

* NMS-20109: Fail closed on HA config errors and harden sync path handling

- Abort startup instead of proceeding standalone when the HA config is
  unreadable, required fields are missing, or coordination throws;
  coordinator mode now requires a distinct partner-instance-id (enforced
  at load, REST update, and reload)
- Clamp failover-threshold to at least 2x the heartbeat interval and
  enforce a minimum sync interval before scheduling
- Suppress config sync in heartbeat-only mode, including config reloads
- Halt the JVM when failover cannot stop services after the step-down
  is advertised, instead of running on undetectable
- Config sync: support the ${scv:alias:attr|default} fallback syntax and
  resolve credentials via the stock vault (honors the configured keystore
  type); advertise the serving node's exclusions in the manifest and use
  the union of both nodes' lists for deletion propagation; canonicalize
  paths before exclusion matching; reject symlink escapes from etc/ and
  skip symlinks when building manifests; re-check HA state before every
  file write and deletion so a mid-cycle promotion aborts the pass
- Copy mode and sync-excludes in the ha-config shell command
- Use subtree intercept-url patterns for /rest/ha so trailing-slash
  variants keep requiring ROLE_ADMIN
- Document featuresBoot.d feature activation in the config template

* NMS-20109: Publish HA state after shutdown completes; lifecycle SPI

- Defer the STANDBY/FAILED row write until the service Invoker has
  finished stopping, so the partner can never promote alongside a node
  that is still draining; a hung drain falls back to heartbeat staleness
  at the failover threshold. The split-brain yield keeps its immediate
  write since the halt is instantaneous.
- Arbitrate split-brain only against a partner whose heartbeat is within
  the failover threshold: an ACTIVE row with a dead heartbeat means the
  partner stopped while holding the role, so the survivor continues as
  the sole active instance instead of halting in deference to it.
- Replace the reflective HA hooks in Starter and Manager with a
  StartupLifecycleHook SPI discovered via ServiceLoader; ha-daemon
  registers the implementation, and its absence is a no-op.
- Clarify the sync-excludes example in the config template.

* NMS-20109: Add docs for HA feature

* NMS-20109: Bound JDBC reads, mask sync password, unify heartbeat task

* NMS-20109: Survive status signals while gated; heartbeat through the drain

- register the SIGUSR1 status handler before the startup gate can block,
  so "opennms status" no longer kills a waiting standby; an empty status
  file is written when no services exist yet
- keep the heartbeat publishing liveness until the terminal state is
  written post-drain: the partner promotes on the state write, never on
  staleness beside a still-draining node (a hung drain is escalated by
  the service manager's stop timeout)
- promote when the partner row stays missing beyond the failover
  threshold instead of waiting forever
- serialize config-sync passes so a reschedule cannot overlap a
  still-running pass

* NMS-20109 : Update docs

* Restart in systemd unit and note in docs

* Fix conditional in nav while I'm here

* NMS-20109: Read partner identity once per monitor cycle

- Snapshot partner-instance-id in each monitor path so a cycle reasons about
  one partner throughout; a change during the anti-flap wait now restarts the
  verification instead of concluding about a different node.
- Reset the missing-row timer when partner-instance-id changes, so a new
  partner gets the full threshold before its absence counts toward promotion.
- Invoke the Manager MBean on the platform server instead of the first entry
  of MBeanServerFactory.findMBeanServer(null), whose ordering is unspecified;
  a miss would turn a planned failover into an immediate halt.
- Do not serve sync files through symlinks, which could otherwise alias an
  excluded file; manifests never advertise them.
- Cancel the superseded monitor and sync tasks before releasing the startup
  gate on promotion.

* NMS-20109: Report heartbeat staleness separately from role and state

- Rename the derived status flag from "degraded" to heartbeat-stale (STALE in
  the shell table) and compute it from heartbeat age alone. It also flagged a
  SECONDARY holding ACTIVE and the DEGRADED state, so both rows read
  DEGRADED=YES after a healthy failover, which reads as total failure.
- Align the class defaults with the shipped template — failover threshold 60s,
  sync interval 60s — and name the enforced minimums in the javadoc.

* NMS-20109: Handle some more edge case and docs

 - Manifest self-identification (#ha-manifest 1) refusing unmarked responses before any fetch or delete
 - partner-rest-url restart-only, with tests on both the REST and reload paths
 - Template defaults to https, plus a WARN when sync runs over http
 - Loud ERROR when a node's own status row has vanished
 - ROLE_REST dropped from the template, javadoc, and docs; this comment condensed
 - Docs: STANDBY vs DEGRADED distinction, automatic return as DEGRADED with a restart policy, and systemctl stop for maintenance

* Fix unrelated bug in syslog tests

* NMS-20109: Update docs on second node installation

---------

Co-authored-by: Dino <[email protected]>



To unsubscribe from these emails, change your notification settings at https://github.com/OpenNMS/opennms/settings/notifications


_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/wiki/index.php?page=MailingListFaq
opennms-cvs mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-cvs