External Metrics, Actions, and Data Providers

This guide covers how to extend the NetXMS agent with custom metrics, actions, and data providers. For agent configuration file syntax and parameter reference, see Agent Configuration.

External Metrics

External metrics allow the agent to execute commands and return their output as metric values. This is the primary mechanism for collecting custom metrics. Only the first line of command output is returned as the metric value.

The ExternalParameter directive is a deprecated alias for ExternalMetric. Both work identically, but ExternalMetric is the preferred name.

Shell Form vs Exec Form

External commands (ExternalMetric, ExternalList, ExternalTable, and Action) support two execution forms:

Shell form passes the command through the system shell (/bin/sh -c on UNIX, cmd.exe /C on Windows):

ExternalMetric = Disk.UsedPercent(*): df --output=pcent $1 | tail -1 | tr -d ' %'
ExternalMetric = Process.Memory(*): ps -o rss= -p $1

Shell form supports pipes, redirections, variable expansion, and command substitution.

Exec form launches the executable directly without a shell:

ExternalMetric = Disk.UsedPercent(*): ["/usr/bin/df", "--output=pcent", "$1"]
ExternalMetric = MyCheck: ["/opt/scripts/check.sh", "param1", "param2"]

On Linux, exec form requires the full path to the executable. On Windows, the system PATH is searched but the file extension (e.g., .exe) must be included.

When command arguments come from untrusted sources (e.g., DCI parameters passed as $1, $2), use exec form to prevent command injection. Shell form is vulnerable because parameter values undergo full shell expansion — a value like $(rm -rf /) would be executed by the shell. Exec form passes arguments directly to the executable without shell interpretation.

The parameter name follows the standard NetXMS naming convention. Arguments passed from the DCI are substituted as $1, $2, etc.

Exit Code Metric

Each ExternalMetric automatically provides a companion metric with the .ExitCode suffix that returns the exit code of the executed command:

ExternalMetric = MyCheck: /opt/scripts/health_check.sh
# This creates two metrics:
#   MyCheck          - first line of command output
#   MyCheck.ExitCode - exit code (0 = success)

For parameterized metrics, the .ExitCode suffix works the same way:

ExternalMetric = Service.Status(*): /opt/scripts/check_service.sh $1
# Creates: Service.Status(*) and Service.Status(*).ExitCode

The .ExitCode metric is not available for ExternalList, BackgroundExternalMetric, ExternalMetricProvider, or ExternalDataProvider.

Background External Metrics

For metrics where you want to avoid blocking the data collection thread, use BackgroundExternalMetric. The command runs asynchronously at the given interval and the cached result is returned immediately:

BackgroundExternalMetric = Slow.Check:300:/opt/scripts/slow_check.sh

The format is name:interval:command, with the execution interval in seconds. All three parts are required — an entry without an interval is not registered.

External Metric Providers

For metrics that are expensive to compute, use an external metric provider. The provider script runs periodically in the background and caches results for multiple metrics:

ExternalMetricProvider = /opt/scripts/collect_all.sh:60

The format is command:interval[,timeout]. The interval is in seconds and defaults to 60 when omitted; the optional timeout is in milliseconds. The script must output key-value pairs, one per line:

CPU.Temperature=62
Memory.Available=4096
Disk.Health(sda)=OK
Disk.Health(sdb)=WARNING

Metrics from the provider are accessed by their names as printed by the script. Parameterized metrics are also supported (as shown with Disk.Health(sda) above).

When no per-provider timeout is given, ExternalMetricProviderTimeout applies; if that is not set, DefaultExecutionTimeout (5000 ms by default) is used.

External Data Providers

External data providers execute a command periodically and cache its structured output (JSON, XML, or plain text). Individual values are then extracted from the cached output using jq expressions (JSON), XPath queries (XML), or PCRE regular expressions (plain text).

This is useful when a single command produces multiple related metrics — the command runs once and all metrics are extracted from the cached result.

Configuration

External data providers use INI-style section syntax:

[ExternalDataProvider/myApp]
Command = /opt/scripts/app_status.sh
PollingInterval = 300
Timeout = 30000
Description = Application status information
Parameter Default Description

Command

(required)

Command to execute (shell form or exec form)

PollingInterval

60

Execution interval in seconds

Timeout

(see description)

Execution timeout in milliseconds. If not set, ExternalMetricProviderTimeout applies, falling back to DefaultExecutionTimeout (5000 ms by default)

Description

(none)

Text description for this provider

ForcePlainTextParser

false

Force plain text parsing even if output looks like JSON or XML

The agent auto-detects the output format:

  • Output starting with { or [ is parsed as JSON (extracted with jq syntax)

  • Output starting with < is parsed as XML (extracted with XPath)

  • All other output is parsed as plain text (extracted with PCRE regex containing one capture group)

Set ForcePlainTextParser = true to override auto-detection.

Generic Metric Access

Without any additional configuration, the provider name itself becomes a parameterized metric. Pass the extraction query as the parameter:

Suppose the command outputs:

{
  "version": "1.2.5",
  "queues": { "in": 10, "out": 42 }
}

You can query values directly:

  • myApp(.version) returns 1.2.5

  • myApp(.queues.out) returns 42

  • myApp(.queues | add) returns 52

Any valid jq expression can be used as the parameter (for JSON output), or XPath query (for XML), or regex (for text).

Named Metrics

Define named metrics to create fixed metric names with predefined extraction queries:

[ExternalDataProvider/myApp/Metrics]
myApp.Version = .version
myApp.Queue.Inbound = .queues.in
myApp.Queue.Outbound = .queues.out
myApp.Queue.Outbound.description = Number of files in outbound queue
myApp.Queue.Outbound.dataType = int32

Each metric entry maps a metric name to an extraction query. Optional .description and .dataType suffixes set the metric description and data type respectively.

Parameterized named metrics are supported (v6.1+):

[ExternalDataProvider/myApp/Metrics]
myApp.Queue(*) = .queues.$1
myApp.Queue(*).description = Number of files in given queue
myApp.Queue(*).dataType = int32

List Metrics

Define list metrics to extract arrays from the cached output:

[ExternalDataProvider/myApp/Lists]
myApp.QueueNames = .queues | keys[]
myApp.QueueNames.description = List of available queue names

Each entry maps a list name to an extraction query that returns multiple values.

Data Types

The following data types can be used in .dataType and ColumnType settings:

Type Description

string

Text string (default for metrics)

int32

32-bit signed integer

uint32

32-bit unsigned integer

int64

64-bit signed integer

uint64

64-bit unsigned integer

float

Floating-point number

counter32

32-bit counter (monotonically increasing, wraps at 232)

counter64

64-bit counter (monotonically increasing, wraps at 264)

The type name null (no data type) is also accepted, as are the legacy aliases int and uint for int32 and uint32.

Complete Example

A script /opt/scripts/app_metrics.sh outputs JSON:

{
  "version": "3.1.0",
  "uptime": 86421,
  "connections": { "active": 150, "idle": 23, "max": 500 },
  "queues": { "inbound": 42, "outbound": 7, "failed": 0 }
}

Agent configuration:

[ExternalDataProvider/appMetrics]
Command = /opt/scripts/app_metrics.sh
PollingInterval = 60

[ExternalDataProvider/appMetrics/Metrics]
App.Version = .version
App.Version.dataType = string
App.Uptime = .uptime
App.Uptime.dataType = uint64
App.Connections.Active = .connections.active
App.Connections.Active.dataType = int32
App.Connections(*) = .connections.$1
App.Connections(*).dataType = int32
App.Queue(*) = .queues.$1
App.Queue(*).dataType = int32

[ExternalDataProvider/appMetrics/Lists]
App.QueueNames = .queues | keys[]
App.ConnectionTypes = .connections | keys[]

This creates metrics like App.Version, App.Uptime, App.Connections.Active, App.Connections(idle), App.Queue(inbound), and lists like App.QueueNames.

Execution Timeouts

# Global timeout for all external executions (milliseconds)
DefaultExecutionTimeout = 30000

# Override for external metrics only
ExternalMetricTimeout = 15000

# Override for metric providers
ExternalMetricProviderTimeout = 60000

# Override for actions
ExternalCommandTimeout = 120000

Environment Variables

The server can push environment variables to the agent for use by externally executed commands. The AcceptedEnvironmentVariables parameter defines which server-provided variables the agent accepts, as a comma-separated list of name patterns:

AcceptedEnvironmentVariables = NX_*, MYAPP_*

The default is NX_*, NETXMS_*. The literal value none clears the list, so no server-provided variables are accepted.

This parameter only filters variables pushed by the server — it does not sanitize the environment that child processes inherit from the agent process.

External Lists

External lists return multi-line output where each line is a list item:

ExternalList = System.ActiveUsers: who | cut -d' ' -f1 | sort -u

Exec form is also supported:

ExternalList = System.Mounts: ["/usr/bin/findmnt", "-rno", "TARGET"]
ExternalList = Dir.Contents(*): ["/usr/bin/ls", "$1"]

External lists can be used as an instance source for instance discovery DCIs.

External Tables

External tables return tabular data from command output. The command must print column names on the first line and data rows on subsequent lines, with columns separated by a delimiter.

Section-Based Configuration

The preferred configuration format uses INI-style sections:

[ExternalTable/Disk.Usage]
Command = df -P | tail -n +2
Separator = \s
MergeSeparators = true
InstanceColumns = mount_point
Description = Disk space usage
Parameter Default Description

Command

(required)

Command to execute (shell form or exec form)

Separator

,

Column separator character

MergeSeparators

false

Treat consecutive separators as one (v6.1+)

InstanceColumns

(none)

Comma-separated list of columns that uniquely identify each row

Description

(none)

Table description

PollingInterval

(none)

If set, run command in background at this interval (seconds). If not set, run synchronously on each request.

Timeout

(see description)

Execution timeout in milliseconds. If not set, ExternalMetricProviderTimeout applies, falling back to DefaultExecutionTimeout (5000 ms by default)

ColumnType

(none)

Column data type in format columnName:dataType. Can be specified multiple times.

DefaultColumnDataType

int32

Default data type for columns without an explicit ColumnType (v6.1+)

Parameterized tables use (*) in the section name:

[ExternalTable/File.Data(*)]
Command = ["/usr/bin/cat", "/data/files/$1"]
Separator = ;
InstanceColumns = id

Separator Special Characters

The Separator parameter supports escape sequences:

Sequence Meaning

\s

Space

\t

Tab

\n

Newline

\r

Carriage return

\u115

Unicode character by decimal code point

Legacy One-Line Configuration

The older single-line format is still supported:

ExternalTable = Disk.Usage::df -P
ExternalTable = Disk.Usage:instanceColumns=mount_point;separator=\s;mergeSeparators=true:df -P | tail -n +2

The format is Name:options:command where options is a semicolon-separated list of key=value pairs. Supported option keys: instanceColumns, separator, description, mergeSeparators, backgroundPolling, pollingInterval, timeout, defaultColumnDataType.

Example

A script outputs comma-separated data with a header row:

name,pid,cpu,memory
httpd,1234,2.5,150000
postgres,5678,8.1,524288
java,9012,15.3,1048576

Configuration:

[ExternalTable/Process.Stats]
Command = /opt/scripts/process_stats.sh
Separator = ,
InstanceColumns = name
ColumnType = name:string
ColumnType = cpu:float
ColumnType = memory:uint64
Description = Process resource usage
PollingInterval = 30

Actions

Actions define commands that the server can execute on the agent remotely. They are used in event processing policy rules, as alarm responses, or triggered manually:

Action = RestartService: systemctl restart $1
Action = CleanLogs: find /var/log -name "*.log" -mtime +30 -delete

The server calls actions through the management console, EPP rules, or NXSL scripts using the AgentExecuteCommand() and AgentExecuteCommandWithOutput() functions.

Actions run with the same permissions as the agent process. Restrict the MasterServers and ControlServers lists to trusted servers only.

Runtime Management

Agent Control Commands

Use nxagentd with control options for runtime management:

Command Description

nxagentd -f

Run in foreground (do not detach)

nxagentd -D <level>

Set debug level (0-9)

nxagentd -c <file>

Use specified configuration file

nxagentd -l

Show log file location

Querying Agent Remotely

From the server, use nxget to query any agent metric:

nxget <agent-ip> <metric-name>
nxget -s <shared-secret> 10.0.0.50 System.CPU.Usage

Use nxaction to execute an action remotely:

nxaction 10.0.0.50 RestartService httpd

Complete Configuration Example

# Server access control
MasterServers = 10.0.0.1
ControlServers = 10.0.1.0/24

# Security
RequireEncryption = yes
RequireAuthentication = yes
SharedSecret = MyS3cretPhrase

# Logging
LogFile = /var/log/nxagentd.log
LogRotationMode = 2
MaxLogSize = 33554432
DebugLevel = 0
DebugTags = tunnel:4

# Network
ListenAddress = *
ListenPort = 4700

# Tunnel connection to server
ServerConnection = server.example.com

# Proxy functions
EnableSNMPProxy = yes
EnableSNMPTrapProxy = yes
SNMPTrapPort = 162

# Zone assignment
ZoneUIN = 3

# Subagents (platform subagent is loaded automatically)
SubAgent = logwatch.nsm
SubAgent = filemgr.nsm

# Custom metrics
ExternalMetric = Hardware.SerialNumber: dmidecode -s system-serial-number
ExternalMetric = Disk.UsedPercent(*): df --output=pcent $1 | tail -1 | tr -d ' %'

# Actions
Action = RestartService: systemctl restart $1

Additional configuration files can be placed in the configuration include directory (nxagentd.conf.d) — see Configuration Include Directory.