alarm-recipient-scan — what happens when the far end is down

Install and run

# download it next to wherever you are working
curl -O https://plantroomlabs.com/tools/alarm-recipient-scan.py

# read the recipient machinery out of an installation
python3 alarm-recipient-scan.py /opt/Niagara/Niagara-4.15.5.22

What it printed here

$ python3 alarm-recipient-scan.py $NIAGARA_HOME
Alarm recipients: what happens when the far end is down
======================================================================

read from /opt/Niagara/Niagara-4.15.5.22
javap:    1.8.0_504

The one method a vendor implements:
  protected abstract boolean sendAlarm(javax.baja.alarm.BAlarmRecord) throws java.lang.Exception;

Property defaults, read out of the static initialiser, not the docs:
  retryInterval  15000 ms (15 s), with a 'min' facet of 1000 ms
  persistent     true
  so out of the box a failed alarm is written to disk and retried
  every 15 seconds, for as long as the station runs.

Step 1. handleAlarm calls sendAlarm on the alarm thread, at offset 82.
        The return value is tested once:
          ifne 91   -> true: set lastSendTime, status ok
          90: return  <- false: RETURN, and nothing else at all

  So returning false does not mean 'not sent, please retry'. It means
  'forget this alarm'. The record is not queued, the status is not set
  to fault, lastFailureCause is not written, the retry thread is not
  started, and nothing is logged. The alarm is gone, and the station
  looks healthy.

Step 2. A thrown Exception is the only way into the recovery machinery.
        The handler (ranges 40-90, 91-145) does, in order:
          status := fault, lastFailureTime := now,
          lastFailureCause := the exception's toString (its cause, if
          it is a BajaRuntimeException),
          then if persistent: write <uuid>.xml under the recipient's
          persistence directory, inside AccessController.doPrivileged,
          and set queuedAlarmCount from a listing of that directory;
          else: Queue.enqueue the record in memory,
          then start the RetryThread if it is not already running.

Step 3. The in-memory queue is the no-argument javax.baja.util.Queue
        (the recipient calls "<init>":()V), whose default bound is
        2,147,483,647 - Integer.MAX_VALUE.
        public synchronized boolean enqueue(java.lang.Object) throws javax.baja.util.QueueFullException;
        throws only at that bound, and handleAlarm does not catch it.
        On a JACE the real bound is the heap, not a setting.

Step 4. The retry loop, in a thread named 'alarm:RecipRetryThread':
          sleep(Math.max(retryInterval, 1000 ms)) - so a retryInterval
          under 1 s is clamped up to 1 s - then poll().
          poll(): persistent and queuedAlarmCount > 0 -> dequeueDisk,
          otherwise dequeueMemory. When queuedAlarmCount reaches 0 the
          thread kills itself and the field is nulled, so the loop
          exists exactly as long as there is something to retry.
          dequeueMemory takes a size snapshot, dequeues that many and
          calls handleAlarm on each, so a record that fails again is
          re-queued at the tail: the retry order is not the alarm order.
          dequeueDisk lists the directory, sorts it with a comparator,
          and for each file decodes the record, deletes the file, then
          calls handleAlarm - which re-writes the same file on failure.

Step 5. What reaches the station log: 6 FINE sites and 1 WARNING.
        The messages in the class are:
          RecoverableRecipient: Failed to delete queue file
          RecoverableRecipient: failed @
          RecoverableRecipient: failed to persist alarm
          RecoverableRecipient: handleAlarm
          RecoverableRecipient: sending ...
          RecoverableRecipient: sent @
        Only the persistence failure is a WARNING. Every send failure,
        every retry and every poll error is FINE, which is off by
        default - so the evidence lives in the recipient's own
        properties (status, lastFailureTime, lastFailureCause,
        queuedAlarmCount), not in the log a site will send you.

Three consequences worth designing for
----------------------------------------------------------------------
  1. return false only for 'this alarm is not mine'. For a failure,
     throw. Returning false on a timeout loses alarms silently.
  2. There is no give-up and no dead letter. A record the far end will
     never accept is retried every 15 s for as long as the station
     runs. With persistent=true that is one delete and one write per
     retry - about 11520 a day - on a controller's flash.
  3. queuedAlarmCount and lastFailureCause are the only operator-
     visible evidence. A product that does not surface them leaves a
     site with nothing to look at.

Three checks one station settles in an afternoon
----------------------------------------------------------------------
  1. Point the recipient at a dead endpoint, raise one alarm, and read
     queuedAlarmCount: zero means your sendAlarm returned false and the
     alarm is gone; one means it threw and the alarm is queued.
  2. Leave it failing for ten minutes and list the persistence
     directory: one <uuid>.xml per queued alarm, rewritten every 15 s.
  3. Bring the endpoint back and watch the order the records arrive in.

What it reads, and what that does not cover

The repository

The file you are downloading