Agent Extensions
An agent extension is a separate process that adds metrics, lists, tables, and actions to the NetXMS agent.
The agent and the extension communicate over a TCP connection using JSON-RPC 2.0, one JSON message per line, so an extension can be written in any language.
The vcenter extension (see VMware vCenter Agent Extension) is an example.
For agent configuration file syntax, see Agent Configuration.
Extensions, Subagents, and External Metrics
The agent has several ways to add data sources:
-
Subagents are shared libraries loaded into the agent process. They use the agent’s C API, and an error in a subagent can stop the agent.
-
External subagents (
ExternalSubagent) are separate processes that connect to the agent through a named pipe or local socket, using the agent’s NXCP protocol. -
External metrics and actions run a command for each request, or periodically for data providers.
-
Extensions are long-running processes that keep their state between requests and answer each request over an open connection. An extension can also push values to Push DCIs and generate events at any time.
The names of the metrics, lists, tables, and actions come from the extension: after it connects, the agent asks the extension for its list of capabilities. The agent asks the extensions only when its own metrics, subagents, external metrics, and external subagents report that a metric is unknown. Give extension metrics names that no other source uses. While an extension is not connected, the agent does not know its metrics, and requests for them fail as unsupported.
Spawn and Connect Modes
Each extension runs in one of two modes:
- Spawn mode
-
The agent starts the extension process and supervises it. For each start, the agent selects a free TCP port on the loopback interface and generates a random authentication token, and passes them to the process in the environment variables
NXAGENT_EXTENSION_PORTandNXAGENT_EXTENSION_TOKEN. The extension listens on127.0.0.1at that port, and the agent connects to it. If the process exits or the connection fails, the agent stops the process and starts it again after a delay. - Connect mode
-
The extension runs on its own, for example as a systemd service or in a container, and listens at a fixed address. The agent connects to the configured
Endpointand authenticates with a token that both sides share. If the connection fails, the agent connects again after a delay.
Configuration
Configuration Section
Each extension has its own section in the agent configuration file.
The section name is extensions/ followed by the extension name:
*extensions/temperature
Mode = spawn
Command = /usr/bin/python3 /opt/myext/temperature.py
Environment = SENSOR_BUS=/dev/i2c-1
EnvironmentFile = /etc/nxagentd.d/temperature.env
*extensions/legacyapp
Mode = connect
Endpoint = tcp://127.0.0.1:9876
Token = file:/etc/nxagentd.d/legacyapp.token
The agent reads extension configuration at startup; restart the agent to apply changes.
| Key | Mode | Default | Description |
|---|---|---|---|
|
Both |
— |
|
|
Spawn |
— |
Command line that starts the extension. Required in spawn mode. The agent starts the command directly, not through a shell, so give the full path to the executable. Arguments are separated by spaces; enclose an argument that contains spaces in double quotes. |
|
Spawn |
— |
Environment variable for the extension process, as |
|
Spawn |
— |
File with environment variables for the extension process, one |
|
Spawn |
— |
Reserved. The agent accepts this key but ignores it: the extension process always runs under the agent’s account. |
|
Spawn |
|
Delay in seconds before the first restart. |
|
Spawn |
|
Maximum restart delay in seconds. |
|
Spawn |
|
Time in seconds that the process has to exit after the agent asks it to stop. See [extension-stop]. |
|
Connect |
— |
Address of the extension, as |
|
Connect |
— |
Authentication token, given as the token itself or as |
|
Connect |
|
Allow an |
|
Connect |
|
Delay in seconds before the first reconnection attempt. |
|
Connect |
|
Maximum reconnection delay in seconds. |
|
Both |
|
Time in seconds to wait for the value of a metric, list, or table. |
|
Both |
|
Time in seconds to wait for an action to complete. |
|
Both |
|
Debug tag for the messages of this extension in the agent log. |
If a section has an error, for example a missing required key or an Endpoint that the agent cannot resolve, the agent writes an error to its log and does not register the extension.
Short Form
A spawn mode extension with default settings can be registered with one line:
Extension = temperature: /usr/bin/python3 /opt/myext/temperature.py
The format is name: command; the word spawn before the command is optional.
Extension can be specified multiple times.
Connect mode needs a token, so it is available only in a configuration section.
Lifecycle
The agent starts all extensions when it starts. A connection is ready for requests after two steps: the agent sends its token and protocol version, and then asks the extension for its capabilities.
The agent sends a ping to the extension every 30 seconds. If three pings in sequence get no reply within 10 seconds, the agent closes the connection, stops a spawned process, and does not start or connect to the extension again until the agent restarts.
In other cases, when a spawned process exits, its connection fails, or the agent cannot connect, the agent waits and tries again.
The first delay is RestartDelay (spawn mode) or ReconnectDelay (connect mode), and each next delay is two times longer, up to RestartDelayMax or ReconnectDelayMax.
The delay returns to its initial value each time the agent opens a TCP connection to the extension, before the token check, so an extension that rejects the token is retried at the initial delay.
Each retry increments the restart count of the extension.
When the agent stops a spawned extension that is still connected, it sends the shutdown notification, closes its sending side of the connection, and waits up to ShutdownTimeout seconds for the process to exit.
If the process is still running after that, the agent sends SIGTERM to the process group of the extension and kills it 2 seconds later; on Windows, it terminates the process.
If the connection was already lost, there is no notification: the agent sends SIGTERM right away and kills the process if it is still running after ShutdownTimeout seconds; on Windows, it terminates the process right away.
When the agent itself stops, it stops all extensions at the same time, so their waiting times run in parallel.
An extension in connect mode keeps running when the agent stops; the agent only closes the connection.
Log Messages
The agent writes the messages of an extension to its log with the extension’s debug tag (extension.<name> unless DebugTag is set).
A few messages that are not specific to one connection, such as the registration of an extension, use the debug tag extension.
These messages are debug messages, so they appear only when the debug level for that tag is high enough:
| Message | Debug level |
|---|---|
Errors that the extension logs through the protocol |
1 |
Warnings that the extension logs through the protocol |
2 |
Start and stop requests |
2 |
Supervision and retry messages of the agent |
3 |
Informational messages that the extension logs through the protocol |
4 |
Lines that a spawned process writes to its standard output or standard error |
5 |
Debug messages that the extension logs through the protocol |
6 |
For example, to see the output of the temperature extension:
DebugTags = extension.temperature:5
Errors in the configuration section are not debug messages: the agent always writes them to its log.
Security
-
The connection between the agent and the extension is not encrypted. The agent accepts an
Endpointthat is not a loopback address only whenAllowRemote = yesis set. If the extension runs on another host, protect the connection, for example with an SSH tunnel or a VPN. -
The token prevents other processes from connecting to the extension in place of the agent. In connect mode, keep the token in a file that only the agent and the extension can read, and refer to the same file from both sides with
Token = file:<path>. -
A spawned extension runs under the agent’s account, with the same privileges as the agent.
-
Values given with
Environmentare part of the agent configuration and can be shown in configuration dumps. Put passwords and other secrets in anEnvironmentFile; its values do not go into the configuration.
Monitoring Extensions
The agent provides these metrics about its extensions:
-
Agent.Extension.IsConnected(name)—1if the extension is connected -
Agent.Extension.Uptime(name)— seconds since the extension connected -
Agent.Extension.RestartCount(name)— number of restart or reconnection attempts -
Agent.Extensions— table with the mode, state, process ID, port, restart count, connection time, and last error of each extension
Writing an Extension
An extension is a TCP server, and the agent is its client. In spawn mode, an extension does these steps:
-
Read the port and token from the
NXAGENT_EXTENSION_PORTandNXAGENT_EXTENSION_TOKENenvironment variables. -
Listen on
127.0.0.1at that port and accept the connection from the agent. -
Check the token in the first request from the agent, and send the list of metrics, lists, tables, and actions when the agent asks for it.
-
Answer value requests, action requests, and pings until the connection closes.
-
Exit when the agent sends the
shutdownnotification, when the connection closes, or when the process getsSIGTERM.
The specification of the protocol, with the message formats and error codes, is in the product source repository:
doc/AGENT_EXTENSION_PROTOCOL.md.
A sample extension in Python is in
contrib/extensions/sample-python.