Agent Extensions

An agent extension is a separate process that adds metrics, lists, tables, and actions to the NetXMS agent. The agent and the extension communicate over a TCP connection using JSON-RPC 2.0, one JSON message per line, so an extension can be written in any language. The vcenter extension (see VMware vCenter Agent Extension) is an example.

For agent configuration file syntax, see Agent Configuration.

Extensions, Subagents, and External Metrics

The agent has several ways to add data sources:

  • Subagents are shared libraries loaded into the agent process. They use the agent’s C API, and an error in a subagent can stop the agent.

  • External subagents (ExternalSubagent) are separate processes that connect to the agent through a named pipe or local socket, using the agent’s NXCP protocol.

  • External metrics and actions run a command for each request, or periodically for data providers.

  • Extensions are long-running processes that keep their state between requests and answer each request over an open connection. An extension can also push values to Push DCIs and generate events at any time.

The names of the metrics, lists, tables, and actions come from the extension: after it connects, the agent asks the extension for its list of capabilities. The agent asks the extensions only when its own metrics, subagents, external metrics, and external subagents report that a metric is unknown. Give extension metrics names that no other source uses. While an extension is not connected, the agent does not know its metrics, and requests for them fail as unsupported.

Spawn and Connect Modes

Each extension runs in one of two modes:

Spawn mode

The agent starts the extension process and supervises it. For each start, the agent selects a free TCP port on the loopback interface and generates a random authentication token, and passes them to the process in the environment variables NXAGENT_EXTENSION_PORT and NXAGENT_EXTENSION_TOKEN. The extension listens on 127.0.0.1 at that port, and the agent connects to it. If the process exits or the connection fails, the agent stops the process and starts it again after a delay.

Connect mode

The extension runs on its own, for example as a systemd service or in a container, and listens at a fixed address. The agent connects to the configured Endpoint and authenticates with a token that both sides share. If the connection fails, the agent connects again after a delay.

Configuration

Configuration Section

Each extension has its own section in the agent configuration file. The section name is extensions/ followed by the extension name:

*extensions/temperature
Mode = spawn
Command = /usr/bin/python3 /opt/myext/temperature.py
Environment = SENSOR_BUS=/dev/i2c-1
EnvironmentFile = /etc/nxagentd.d/temperature.env

*extensions/legacyapp
Mode = connect
Endpoint = tcp://127.0.0.1:9876
Token = file:/etc/nxagentd.d/legacyapp.token

The agent reads extension configuration at startup; restart the agent to apply changes.

Key Mode Default Description

Mode

Both

 — 

spawn or connect. Required.

Command

Spawn

 — 

Command line that starts the extension. Required in spawn mode. The agent starts the command directly, not through a shell, so give the full path to the executable. Arguments are separated by spaces; enclose an argument that contains spaces in double quotes.

Environment

Spawn

 — 

Environment variable for the extension process, as NAME=value. Can be specified multiple times. The process also inherits the agent’s environment.

EnvironmentFile

Spawn

 — 

File with environment variables for the extension process, one NAME=value per line; lines that start with # are comments. The agent reads the file each time it starts the process, so a changed value takes effect when the extension restarts, without an agent restart. Values from this file override Environment values with the same name. If the agent cannot read the file, the start fails and the agent tries again after the restart delay.

RunAs

Spawn

 — 

Reserved. The agent accepts this key but ignores it: the extension process always runs under the agent’s account.

RestartDelay

Spawn

5

Delay in seconds before the first restart.

RestartDelayMax

Spawn

300

Maximum restart delay in seconds.

ShutdownTimeout

Spawn

10

Time in seconds that the process has to exit after the agent asks it to stop. After this time the agent kills the process.

Endpoint

Connect

 — 

Address of the extension, as tcp://host:port or host:port. Enclose an IPv6 address in brackets, for example tcp://[::1]:9876. Required in connect mode.

Token

Connect

 — 

Authentication token, given as the token itself or as file:<path> to read it from a file. Required in connect mode. The agent reads the token file once, at startup. In spawn mode the agent generates its own token and does not use this key; omit it, because a file: value that cannot be read makes the section invalid in either mode.

AllowRemote

Connect

no

Allow an Endpoint that is not a loopback address. See Security.

ReconnectDelay

Connect

5

Delay in seconds before the first reconnection attempt.

ReconnectDelayMax

Connect

300

Maximum reconnection delay in seconds.

RequestTimeout

Both

5

Time in seconds to wait for the value of a metric, list, or table.

ActionTimeout

Both

60

Time in seconds to wait for an action to complete.

DebugTag

Both

extension.<name>

Debug tag for the messages of this extension in the agent log.

If a section has an error, for example a missing required key or an Endpoint that the agent cannot resolve, the agent writes an error to its log and does not register the extension.

Short Form

A spawn mode extension with default settings can be registered with one line:

Extension = temperature: /usr/bin/python3 /opt/myext/temperature.py

The format is name: command; the word spawn before the command is optional. Extension can be specified multiple times. Connect mode needs a token, so it is available only in a configuration section.

Lifecycle

The agent starts all extensions when it starts. A connection is ready for requests after two steps: the agent sends its token and protocol version, and then asks the extension for its capabilities.

The agent sends a ping to the extension every 30 seconds. If three pings in sequence get no reply within 10 seconds, the agent closes the connection, stops a spawned process, and does not start or connect to the extension again until the agent restarts.

In other cases, when a spawned process exits, its connection fails, or the agent cannot connect, the agent waits and tries again. The first delay is RestartDelay (spawn mode) or ReconnectDelay (connect mode), and each next delay is two times longer, up to RestartDelayMax or ReconnectDelayMax. The delay returns to its initial value each time the agent opens a TCP connection to the extension, before the token check, so an extension that rejects the token is retried at the initial delay. Each retry increments the restart count of the extension.

When the agent stops a spawned extension:

  • On UNIX-like systems, the agent closes the connection, sends SIGTERM to the process group of the extension, and waits up to ShutdownTimeout seconds; if the process is still running, the agent kills it.

  • On Windows, the agent terminates the process immediately.

An extension in connect mode keeps running when the agent stops; the agent only closes the connection.

Log Messages

The agent writes the messages of an extension to its log with the extension’s debug tag (extension.<name> unless DebugTag is set). A few messages that are not specific to one connection, such as the registration of an extension, use the debug tag extension. These messages are debug messages, so they appear only when the debug level for that tag is high enough:

Message Debug level

Errors that the extension logs through the protocol

1

Warnings that the extension logs through the protocol

2

Start and stop requests

2

Supervision and retry messages of the agent

3

Informational messages that the extension logs through the protocol

4

Lines that a spawned process writes to its standard output or standard error

5

Debug messages that the extension logs through the protocol

6

For example, to see the output of the temperature extension:

DebugTags = extension.temperature:5

Errors in the configuration section are not debug messages: the agent always writes them to its log.

Security

  • The connection between the agent and the extension is not encrypted. The agent accepts an Endpoint that is not a loopback address only when AllowRemote = yes is set. If the extension runs on another host, protect the connection, for example with an SSH tunnel or a VPN.

  • The token prevents other processes from connecting to the extension in place of the agent. In connect mode, keep the token in a file that only the agent and the extension can read, and refer to the same file from both sides with Token = file:<path>.

  • A spawned extension runs under the agent’s account, with the same privileges as the agent.

  • Values given with Environment are part of the agent configuration and can be shown in configuration dumps. Put passwords and other secrets in an EnvironmentFile; its values do not go into the configuration.

Monitoring Extensions

The agent provides these metrics about its extensions:

Writing an Extension

An extension is a TCP server, and the agent is its client. In spawn mode, an extension does these steps:

  1. Read the port and token from the NXAGENT_EXTENSION_PORT and NXAGENT_EXTENSION_TOKEN environment variables.

  2. Listen on 127.0.0.1 at that port and accept the connection from the agent.

  3. Check the token in the first request from the agent, and send the list of metrics, lists, tables, and actions when the agent asks for it.

  4. Answer value requests, action requests, and pings until the connection closes.

  5. Exit when the connection closes or the process gets SIGTERM. Do not rely on a shutdown notification from the agent.

The specification of the protocol, with the message formats and error codes, is in the product source repository: doc/AGENT_EXTENSION_PROTOCOL.md. A sample extension in Python is in contrib/extensions/sample-python.