Prometheus Monitoring
The prometheus agent subagent brings metrics in Prometheus format into NetXMS.
It works in two ways, which can be used at the same time:
-
Scraping — the agent reads metrics endpoints (Prometheus exporters, or applications with a built-in
/metricsendpoint) at a fixed interval and keeps the last result in memory. DCIs read single series with the agent metricsPrometheus.ValueandPrometheus.LabelValue, and lists and tables of the scraped data support instance discovery. -
Receiving — the agent accepts data sent with the Prometheus remote write protocol, for example by a Prometheus server.
Metric mappings in the agent configuration push selected samples from both sources to push DCIs on the server. Received remote write data is available only through metric mappings.
How It Works
Scraping Endpoints
Each scrape target has its own section in the agent configuration and its own scraper thread. The first scrape runs when the agent starts. The next scrape starts one scrape interval after the start of the previous one; if a scrape takes longer than the interval, the next one starts 1 second after it ends.
The agent sends an HTTP GET request to the target URL. The request asks for the OpenMetrics text format 1.0.0 or the Prometheus text exposition format 0.0.4, accepts compressed responses, and follows up to 8 redirects. Only HTTP and HTTPS URLs are supported.
A scrape succeeds when the endpoint answers with an HTTP status code from 200 to 299 within the timeout and the response is not larger than 64 MB.
After a successful scrape, the parsed samples replace the samples of the previous scrape.
After a failed scrape, the samples of the last successful scrape remain available, and Prometheus.Target.Status returns 0.
The agent metrics read the samples in memory and do not contact the endpoint. A DCI therefore gets a new value at most once per scrape interval, whatever its own polling interval is.
The parser reads one sample per line in the form metric_name{label="value",…} value:
-
Lines that start with
(HELP,# TYPE,# EOF, and other comments) are skipped. -
Sample timestamps and OpenMetrics exemplars are ignored.
-
Values are floating-point numbers, including
NaN,+Inf, and-Inf. -
Metric and label names can contain letters, digits,
_,:, and.. Lines that use the quoted metric name syntax of Prometheus 3 ({"metric.name",label="value"} 1) cannot be parsed. -
Lines that cannot be parsed are skipped.
Histograms and summaries are exposed by endpoints as separate series (_bucket, _sum, _count, and series with a quantile label).
Read them like any other series, for example Prometheus.Value(app,http_request_duration_seconds_bucket,le=0.5).
Selecting a Series
A Prometheus metric usually has several series that differ in their labels, for example one series per network device.
Prometheus.Value and Prometheus.LabelValue select one series with label filters, given as additional metric arguments in the form label=value:
Prometheus.Value(node,node_network_receive_bytes_total,device=eth0)
Prometheus.Value(node,node_filesystem_avail_bytes,mountpoint=/,fstype=ext4)
Prometheus.LabelValue(node,node_uname_info,release)
The series is selected as follows:
-
The metric name must be equal to the name in the endpoint response, including any suffix such as
_total. The comparison is case-sensitive. -
A series matches a filter when it has the label and the label value is exactly the given value (case-sensitive). A series without the label does not match.
-
A series must match all filters. There are no regular expressions and no negative filters.
-
If several series match, the first one in the order of the endpoint response is used. Without filters, this is the first series of the metric. Give enough filters to identify one series;
Prometheus.Seriesshows all series of a metric with their labels.
Arguments are separated by commas, and spaces at the start and end of each argument are removed.
If a label value contains a comma or a parenthesis, put the argument or only the value in double or single quotes, for example "path=/a,b" or path="/a,b".
To include the quote character itself in a quoted string, double it.
Receiving Remote Write Data
The subagent runs an HTTP listener (without TLS and without authentication) on the configured address and port. It accepts POST requests to the configured endpoint path and answers other paths with HTTP status 404 and other methods with 405.
The request body must be a Prometheus remote write 1.0 message: a snappy-compressed protocol buffers prometheus.WriteRequest.
Remote write 2.0 messages (io.prometheus.write.v2.Request) are not supported.
The listener answers with HTTP status 200 when it has decoded the message, and with 400 when the body is empty or cannot be decoded.
From each time series, the subagent reads the metric name (label __name__), the other labels, and the float samples; native histogram samples are ignored.
Each sample is passed to the metric mappings, and samples of metrics without a mapping are discarded.
The received data is not stored by the subagent.
Metric Mappings
A metric mapping pushes the samples of one Prometheus metric to the server as push DCI values. Mappings apply to samples received through remote write and to samples of every successful scrape. Each sample is pushed separately and stored with the time when the server receives it; sample timestamps are not used.
A mapping is a Metric entry in the [Prometheus] section, in one of two forms:
Metric = prometheus_name:netxms_name:name_labels
Metric = prometheus_name:netxms_name:name_labels:value_labels
| Field | Description |
|---|---|
|
Prometheus metric name, compared exactly with the metric name of the sample. |
|
Base name of the pushed metric. |
|
Comma-separated label names. Their values, in this order, become the arguments of the pushed metric name. |
|
Optional, comma-separated label names. If given, the pushed value is the values of these labels, separated by spaces, and the sample value is ignored. If not given, the pushed value is the sample value with six decimal places. |
The pushed metric name is netxms_name(value,value,…); the parentheses are always present.
If a sample does not have a label from name_labels or value_labels, an empty string is used for it.
A label value that contains (, ), ,, ", or ' is enclosed in double quotes in the metric name, with " doubled.
The fields are separated by colons and are not trimmed, so do not put spaces around the colons and commas.
Prometheus metric names that contain a colon (for example, names of recording rule results such as job:http_requests:rate5m) cannot be mapped.
If several mappings have the same prometheus_name, only the first one is used.
For example, this mapping:
Metric = node_network_transmit_drop_total:Net.Interface.TxDrops:nodename,device
turns the sample
node_network_transmit_drop_total{device="eth0",instance="server1.local",job="node",nodename="server1"} 42
into the pushed value Net.Interface.TxDrops(server1,eth0) = 42.000000.
This mapping uses labels as the value:
Metric = node_uname_info:Node.Uname:nodename:sysname,release
It turns the sample node_uname_info{nodename="server1",release="5.14.0-570.el9.x86_64",sysname="Linux"} 1 into the pushed value Node.Uname(server1) = Linux 5.14.0-570.el9.x86_64.
All pushed values go to the node of the agent that runs the subagent, whatever host the samples describe.
When the data covers several hosts, as remote write data from a Prometheus server usually does, include a label that identifies the host (such as instance or nodename) in name_labels.
The server stores a pushed value only in an active DCI with origin Push whose metric name is the pushed name. If there is no such DCI, the server checks DCIs with instance discovery method Push (see Collecting Received and Mapped Data); values that match none are discarded.
Enabling Prometheus Monitoring
The Windows agent installer includes prometheus.nsm.
The agent packages for Linux do not include it; to use it on a UNIX-like system, build the agent from source with the configure option --with-prometheus, which requires the libmicrohttpd, Protocol Buffers, and snappy libraries.
Load the subagent in nxagentd.conf:
SubAgent = prometheus
The remote write receiver is enabled by default and listens on 127.0.0.1:9090, also when no [Prometheus] section exists.
If the receiver cannot start (for example, because the port is in use or the address is invalid), the subagent fails to load, and scraping does not work either.
Port 9090 is also the default port of the Prometheus server; if both run on one host, set another Port or set EnableReceiver = no.
The subagent reads its configuration only when the agent starts. Restart the agent after every change.
[Prometheus] Section
| Parameter | Default | Description |
|---|---|---|
|
|
Enable the remote write receiver. Set to |
|
|
IP address the receiver listens on (an address, not a host name). Use |
|
|
TCP port the receiver listens on. With |
|
|
URL path that accepts remote write requests. |
|
(none) |
Metric mapping (see Metric Mappings). Repeat the parameter for each mapping. |
The receiver has no authentication and no TLS.
Keep the default address 127.0.0.1 if the sender runs on the same host; otherwise, restrict access to the port with a firewall.
Scrape Target Sections
Each scrape target is a section [Prometheus/Targets/target].
The target (up to 63 characters) is the target name used as the first argument of the Prometheus.* metrics.
A section without URL is skipped, with a warning in the agent log.
| Parameter | Default | Description |
|---|---|---|
|
(required) |
HTTP or HTTPS URL of the metrics endpoint. The endpoint can be on the agent host or on another host. |
|
|
Scrape interval in seconds. The minimum is 1. |
|
|
Timeout for the whole request in seconds. The minimum is 1. |
|
(none) |
User name for HTTP basic authentication. |
|
(none) |
Password for HTTP basic authentication, plain text or obfuscated with |
|
(none) |
Token for HTTP bearer authentication. If set, |
|
|
Verify the server certificate for HTTPS. |
|
|
Verify that the server certificate matches the host name in the URL. |
|
(none) |
Path to a file with trusted CA certificates in PEM format. If set, it is the only trust source for the target. If not set, the default trust store of libcurl is used; on Windows, the Windows certificate store. |
When BearerToken is set and the agent uses libcurl older than 7.69, the agent does not follow redirects for the target, so the token is not sent to another host.
The TLS backend of libcurl on Windows is selected with the agent parameter CurlSSLBackend (see Agent Configuration).
Example Configuration
SubAgent = prometheus
[Prometheus]
EnableReceiver = no
[Prometheus/Targets/node]
URL = http://127.0.0.1:9100/metrics
ScrapeInterval = 30
[Prometheus/Targets/etcd]
URL = https://etcd1.example.com:2379/metrics
CACertificate = /etc/ssl/etcd-ca.pem
BearerToken = exampletoken
[Prometheus/Targets/haproxy]
URL = http://lb1.example.com:8404/metrics
Login = monitoring
Password = secret
Check the targets with nxget:
nxget -T <agent-address> Prometheus.Targets
nxget <agent-address> "Prometheus.Value(node,node_load1)"
Collecting Scraped Data
Create DCIs with origin NetXMS Agent on the node of the agent.
For a single series, use Prometheus.Value with label filters, or Prometheus.LabelValue for data carried in labels:
| Metric | Collects |
|---|---|
|
Load average (a metric with one series) |
|
Free space of the root file system |
|
Kernel release from the |
|
1 if the last scrape succeeded |
For counters (metrics with the suffix _total), set a delta calculation method on the DCI to get a rate (see Delta Calculation Methods).
A DCI on Prometheus.Target.Status with a threshold on value 0 detects failed scrapes; while scrapes fail, the other metrics keep returning the data of the last successful scrape.
Instance Discovery
Lists and tables of the subagent create one DCI per series (see Configuring Instance Discovery):
-
One label identifies the series — use method Agent List with the list
Prometheus.LabelValues(node,node_network_receive_bytes_total,device)and the metricPrometheus.Value(node,node_network_receive_bytes_total,device={instance}). If label values can contain commas or parentheses, quote the filter:"device={instance}". -
Several labels identify the series — use method Agent Table with the table
Prometheus.Series(node,node_filesystem_avail_bytes)and the metricPrometheus.Value(node,node_filesystem_avail_bytes,{instance}). The instance is the LABELS column, which contains all labels of the series inlabel=value,…form, already quoted where needed, so it expands to one filter per label.
With the Agent Table method, the instance contains all labels of the series.
If the value of any label changes (for example, a version label after an upgrade), the series gets a new instance and a new DCI.
Collecting Received and Mapped Data
-
Configure metric mappings in the
[Prometheus]section (see Metric Mappings) and restart the agent. -
For remote write, enable the receiver and configure the sender. For a Prometheus server, add a
remote_writeentry toprometheus.yml, and limit the sent series to the mapped metrics withwrite_relabel_configs:remote_write: - url: "http://192.0.2.10:9091/api/v1/write" write_relabel_configs: - source_labels: [__name__] regex: "node_network_transmit_drop_total|node_uname_info" action: keepThe agent configuration for this sender:
[Prometheus] Address = 192.0.2.10 Port = 9091 Metric = node_network_transmit_drop_total:Net.Interface.TxDrops:nodename,device Metric = node_uname_info:Node.Uname:nodename:sysname,release -
On the node of the agent, create DCIs with origin Push and the pushed names as metric names, for example
Net.Interface.TxDrops(server1,eth0). Choose a numeric data type for mappings that push the sample value, and String for mappings with value labels.
To create the DCIs automatically, use instance discovery method Push on a prototype DCI with origin Push:
-
Metric name — the pushed name with
{instance}in place of the varying argument, for exampleNet.Interface.TxDrops(server1,{instance}). -
Pattern — a regular expression that matches the pushed names (case-insensitive); the first capturing group is the instance, for example
^Net\.Interface\.TxDrops\(server1,(.+)\)$.
When a value arrives for a name without a DCI, the server matches the name against the pattern, creates the DCI for the instance (if the instance filter accepts it), and stores the value in it.
Reference
Agent Metrics
These metrics are provided by the prometheus subagent.
They read the samples that the subagent keeps in memory from the last successful scrape of each scrape target; a metric request does not contact the endpoint.
Data received through the remote write receiver is not available through these metrics.
The first argument of every metric is the name of a scrape target, as configured in its [Prometheus/Targets/target] section; it is matched case-insensitively.
An unknown target name results in the error no such instance.
Metrics that select a series take optional label=value arguments after the fixed arguments.
A series matches when it has every given label with exactly the given value (case-sensitive); the first matching series in the order of the endpoint response is used.
Metric names are matched exactly (case-sensitive).
Put an argument in double or single quotes if it contains a comma or a parenthesis, for example "path=/a,b" or path="/a,b".
For details and examples, see Selecting a Series.
Series Metrics
Prometheus.LabelValue(target, metric, label[, label=value …])
Value of label label of the first series of metric that matches the label filters.
Use it for info-style metrics that carry their data in labels, for example node_uname_info or node_exporter_build_info.
Returns no such instance when no series matches or the matching series does not have the label label, and an error when the target has had no successful scrape since the agent started.
Data Type |
String |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
metric |
String |
Metric name as exposed by the endpoint, for example |
label |
String |
Name of the label whose value is returned |
label=value |
String |
Optional, any number: label filters that select the series |
Prometheus.Value(target, metric[, label=value …])
Value of the first series of metric that matches the label filters.
Without label filters, the first series of the metric is used.
Returns no such instance when no series matches, and an error when the target has had no successful scrape since the agent started.
Data Type |
Float |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
metric |
String |
Metric name as exposed by the endpoint, including any suffix such as |
label=value |
String |
Optional, any number: label filters that select the series |
Target Metrics
Prometheus.Target.LastScrapeTime(target)
Time of the last scrape attempt of the target, successful or not, as UNIX time. 0 before the first scrape attempt.
Data Type |
Unsigned Integer 64 |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
Prometheus.Target.SampleCount(target)
Number of samples kept from the last successful scrape of the target (one sample per series). A failed scrape does not change it; 0 before the first successful scrape.
Data Type |
Unsigned Integer |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
Lists
Prometheus.LabelValues(target, metric, label) (List)
Distinct values of label label across all series of metric, in the order of first appearance in the endpoint response.
Series without the label are skipped.
The list is empty when the metric is not in the last scrape.
Use it as the source of instance discovery with method Agent List when one label identifies a series.
Data Type |
List |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
metric |
String |
Metric name |
label |
String |
Label name |
Tables
Prometheus.Series(target, metric) (Table)
All series of metric from the last successful scrape, one row per series.
The table is empty when the metric is not in the last scrape.
Use it as the source of instance discovery with method Agent Table when several labels identify a series.
Data Type |
Table |
Platforms |
All ( |
| Argument | Type | Description |
|---|---|---|
target |
String |
Scrape target name |
metric |
String |
Metric name |
| Column | Type | Description |
|---|---|---|
LABELS |
String |
All labels of the series in the form |
VALUE |
Float |
Sample value |
label name |
String |
One column for each label name that appears in any series of the metric, named after the label, in the order of first appearance. Empty for a series without that label. A label named |
Prometheus.Targets (Table)
Configured scrape targets with the result of their last scrape.
Data Type |
Table |
Platforms |
All ( |
| Column | Type | Description |
|---|---|---|
NAME |
String |
Target name (instance key) |
URL |
String |
Endpoint URL |
INTERVAL |
Unsigned Integer |
Scrape interval in seconds |
STATUS |
Integer |
Result of the last scrape attempt: 1 = success, 0 = failure or no attempt yet |
LAST_SCRAPE |
Unsigned Integer 64 |
Time of the last scrape attempt (UNIX time); 0 before the first attempt |
SAMPLES |
Unsigned Integer |
Number of samples kept from the last successful scrape |
Troubleshooting
-
Prometheus.*metrics are not supported — check that the subagent is loaded (nxget -l <agent-address> Agent.SubAgentListlistsPROMETHEUS). If it is not, check the agent log: the subagent does not load when the remote write receiver cannot start. -
Prometheus.Valuereturns an error — the target has had no successful scrape since the agent started. CheckPrometheus.Target.Statusand the debug output of theprometheustag. -
Prometheus.Valuereturns no such instance — the target name is unknown, the metric is not in the last scrape, or no series matches the label filters. Compare the arguments withnxget -T <agent-address> "Prometheus.Series(<target>,<metric>)". -
Mapped values do not reach the DCIs — check that the pushed name in the agent debug output (tag
prometheus, level 7) is exactly the DCI metric name, and that the DCI has origin Push and is active.
To see details, enable these debug tags (agent tags with DebugTags in nxagentd.conf, server tags with the debug server console command; see Diagnostics):
| Debug tag | Component |
|---|---|
|
Agent: |
|
Agent: values sent to the server |
|
Server: values pushed by agents (level 5) |
|
Server: DCIs created by instance discovery method Push |