Prometheus Monitoring

The prometheus agent subagent brings metrics in Prometheus format into NetXMS. It works in two ways, which can be used at the same time:

  • Scraping — the agent reads metrics endpoints (Prometheus exporters, or applications with a built-in /metrics endpoint) at a fixed interval and keeps the last result in memory. DCIs read single series with the agent metrics Prometheus.Value and Prometheus.LabelValue, and lists and tables of the scraped data support instance discovery.

  • Receiving — the agent accepts data sent with the Prometheus remote write protocol, for example by a Prometheus server.

Metric mappings in the agent configuration push selected samples from both sources to push DCIs on the server. Received remote write data is available only through metric mappings.

How It Works

Scraping Endpoints

Each scrape target has its own section in the agent configuration and its own scraper thread. The first scrape runs when the agent starts. The next scrape starts one scrape interval after the start of the previous one; if a scrape takes longer than the interval, the next one starts 1 second after it ends.

The agent sends an HTTP GET request to the target URL. The request asks for the OpenMetrics text format 1.0.0 or the Prometheus text exposition format 0.0.4, accepts compressed responses, and follows up to 8 redirects. Only HTTP and HTTPS URLs are supported.

A scrape succeeds when the endpoint answers with an HTTP status code from 200 to 299 within the timeout and the response is not larger than 64 MB. After a successful scrape, the parsed samples replace the samples of the previous scrape. After a failed scrape, the samples of the last successful scrape remain available, and Prometheus.Target.Status returns 0.

The agent metrics read the samples in memory and do not contact the endpoint. A DCI therefore gets a new value at most once per scrape interval, whatever its own polling interval is.

The parser reads one sample per line in the form metric_name{label="value",…​} value:

  • Lines that start with ( HELP, # TYPE, # EOF, and other comments) are skipped.

  • Sample timestamps and OpenMetrics exemplars are ignored.

  • Values are floating-point numbers, including NaN, +Inf, and -Inf.

  • Metric and label names can contain letters, digits, _, :, and .. Lines that use the quoted metric name syntax of Prometheus 3 ({"metric.name",label="value"} 1) cannot be parsed.

  • Lines that cannot be parsed are skipped.

Histograms and summaries are exposed by endpoints as separate series (_bucket, _sum, _count, and series with a quantile label). Read them like any other series, for example Prometheus.Value(app,http_request_duration_seconds_bucket,le=0.5).

Selecting a Series

A Prometheus metric usually has several series that differ in their labels, for example one series per network device. Prometheus.Value and Prometheus.LabelValue select one series with label filters, given as additional metric arguments in the form label=value:

Prometheus.Value(node,node_network_receive_bytes_total,device=eth0)
Prometheus.Value(node,node_filesystem_avail_bytes,mountpoint=/,fstype=ext4)
Prometheus.LabelValue(node,node_uname_info,release)

The series is selected as follows:

  • The metric name must be equal to the name in the endpoint response, including any suffix such as _total. The comparison is case-sensitive.

  • A series matches a filter when it has the label and the label value is exactly the given value (case-sensitive). A series without the label does not match.

  • A series must match all filters. There are no regular expressions and no negative filters.

  • If several series match, the first one in the order of the endpoint response is used. Without filters, this is the first series of the metric. Give enough filters to identify one series; Prometheus.Series shows all series of a metric with their labels.

Arguments are separated by commas, and spaces at the start and end of each argument are removed. If a label value contains a comma or a parenthesis, put the argument or only the value in double or single quotes, for example "path=/a,b" or path="/a,b". To include the quote character itself in a quoted string, double it.

Receiving Remote Write Data

The subagent runs an HTTP listener (without TLS and without authentication) on the configured address and port. It accepts POST requests to the configured endpoint path and answers other paths with HTTP status 404 and other methods with 405.

The request body must be a Prometheus remote write 1.0 message: a snappy-compressed protocol buffers prometheus.WriteRequest. Remote write 2.0 messages (io.prometheus.write.v2.Request) are not supported. The listener answers with HTTP status 200 when it has decoded the message, and with 400 when the body is empty or cannot be decoded.

From each time series, the subagent reads the metric name (label __name__), the other labels, and the float samples; native histogram samples are ignored. Each sample is passed to the metric mappings, and samples of metrics without a mapping are discarded. The received data is not stored by the subagent.

Metric Mappings

A metric mapping pushes the samples of one Prometheus metric to the server as push DCI values. Mappings apply to samples received through remote write and to samples of every successful scrape. Each sample is pushed separately and stored with the time when the server receives it; sample timestamps are not used.

A mapping is a Metric entry in the [Prometheus] section, in one of two forms:

Metric = prometheus_name:netxms_name:name_labels
Metric = prometheus_name:netxms_name:name_labels:value_labels
Field Description

prometheus_name

Prometheus metric name, compared exactly with the metric name of the sample.

netxms_name

Base name of the pushed metric.

name_labels

Comma-separated label names. Their values, in this order, become the arguments of the pushed metric name.

value_labels

Optional, comma-separated label names. If given, the pushed value is the values of these labels, separated by spaces, and the sample value is ignored. If not given, the pushed value is the sample value with six decimal places.

The pushed metric name is netxms_name(value,value,…​); the parentheses are always present. If a sample does not have a label from name_labels or value_labels, an empty string is used for it. A label value that contains (, ), ,, ", or ' is enclosed in double quotes in the metric name, with " doubled.

The fields are separated by colons and are not trimmed, so do not put spaces around the colons and commas. Prometheus metric names that contain a colon (for example, names of recording rule results such as job:http_requests:rate5m) cannot be mapped. If several mappings have the same prometheus_name, only the first one is used.

For example, this mapping:

Metric = node_network_transmit_drop_total:Net.Interface.TxDrops:nodename,device

turns the sample

node_network_transmit_drop_total{device="eth0",instance="server1.local",job="node",nodename="server1"} 42

into the pushed value Net.Interface.TxDrops(server1,eth0) = 42.000000.

This mapping uses labels as the value:

Metric = node_uname_info:Node.Uname:nodename:sysname,release

It turns the sample node_uname_info{nodename="server1",release="5.14.0-570.el9.x86_64",sysname="Linux"} 1 into the pushed value Node.Uname(server1) = Linux 5.14.0-570.el9.x86_64.

All pushed values go to the node of the agent that runs the subagent, whatever host the samples describe. When the data covers several hosts, as remote write data from a Prometheus server usually does, include a label that identifies the host (such as instance or nodename) in name_labels.

The server stores a pushed value only in an active DCI with origin Push whose metric name is the pushed name. If there is no such DCI, the server checks DCIs with instance discovery method Push (see Collecting Received and Mapped Data); values that match none are discarded.

Enabling Prometheus Monitoring

The agent packages and the Windows agent installer do not include prometheus.nsm; to use it, build the agent from source with the configure option --with-prometheus, which requires the libmicrohttpd, Protocol Buffers, and snappy libraries.

Load the subagent in nxagentd.conf:

SubAgent = prometheus

The remote write receiver is enabled by default and listens on 127.0.0.1:9090, also when no [Prometheus] section exists. If the receiver cannot start (for example, because the port is in use or the address is invalid), the subagent fails to load, and scraping does not work either. Port 9090 is also the default port of the Prometheus server; if both run on one host, set another Port or set EnableReceiver = no.

The subagent reads its configuration only when the agent starts. Restart the agent after every change.

[Prometheus] Section

Parameter Default Description

EnableReceiver

yes

Enable the remote write receiver. Set to no if you only scrape endpoints.

Address

127.0.0.1

IP address the receiver listens on (an address, not a host name). Use 0.0.0.0 to listen on all IPv4 addresses.

Port

9090

TCP port the receiver listens on. With 0, the subagent fails to load.

Endpoint

/api/v1/write

URL path that accepts remote write requests.

Metric

(none)

Metric mapping (see Metric Mappings). Repeat the parameter for each mapping.

The receiver has no authentication and no TLS. Keep the default address 127.0.0.1 if the sender runs on the same host; otherwise, restrict access to the port with a firewall.

Scrape Target Sections

Each scrape target is a section [Prometheus/Targets/target]. The target (up to 63 characters) is the target name used as the first argument of the Prometheus.* metrics. A section without URL is skipped, with a warning in the agent log.

Parameter Default Description

URL

(required)

HTTP or HTTPS URL of the metrics endpoint. The endpoint can be on the agent host or on another host.

ScrapeInterval

60

Scrape interval in seconds. The minimum is 1.

Timeout

10

Timeout for the whole request in seconds. The minimum is 1.

Login

(none)

User name for HTTP basic authentication.

Password

(none)

Password for HTTP basic authentication, plain text or obfuscated with nxencpasswd (use the Login value as the key: nxencpasswd login password). Used only when Login is set.

BearerToken

(none)

Token for HTTP bearer authentication. If set, Login and Password are not used.

VerifyCertificate

yes

Verify the server certificate for HTTPS.

VerifyHost

yes

Verify that the server certificate matches the host name in the URL.

CACertificate

(none)

Path to a file with trusted CA certificates in PEM format. If set, it is the only trust source for the target. If not set, the default trust store of libcurl is used.

When BearerToken is set and the agent uses libcurl older than 7.61, the agent does not follow redirects for the target, so the token is not sent to another host.

Example Configuration

SubAgent = prometheus

[Prometheus]
EnableReceiver = no

[Prometheus/Targets/node]
URL = http://127.0.0.1:9100/metrics
ScrapeInterval = 30

[Prometheus/Targets/etcd]
URL = https://etcd1.example.com:2379/metrics
CACertificate = /etc/ssl/etcd-ca.pem
BearerToken = exampletoken

[Prometheus/Targets/haproxy]
URL = http://lb1.example.com:8404/metrics
Login = monitoring
Password = secret

Check the targets with nxget:

nxget -T <agent-address> Prometheus.Targets
nxget <agent-address> "Prometheus.Value(node,node_load1)"

Collecting Scraped Data

Create DCIs with origin NetXMS Agent on the node of the agent. For a single series, use Prometheus.Value with label filters, or Prometheus.LabelValue for data carried in labels:

Metric Collects

Prometheus.Value(node,node_load1)

Load average (a metric with one series)

Prometheus.Value(node,node_filesystem_avail_bytes,mountpoint=/)

Free space of the root file system

Prometheus.LabelValue(node,node_uname_info,release)

Kernel release from the release label

Prometheus.Target.Status(node)

1 if the last scrape succeeded

For counters (metrics with the suffix _total), set a delta calculation method on the DCI to get a rate (see Delta Calculation Methods). A DCI on Prometheus.Target.Status with a threshold on value 0 detects failed scrapes; while scrapes fail, the other metrics keep returning the data of the last successful scrape.

Instance Discovery

Lists and tables of the subagent create one DCI per series (see Configuring Instance Discovery):

  • One label identifies the series — use method Agent List with the list Prometheus.LabelValues(node,node_network_receive_bytes_total,device) and the metric Prometheus.Value(node,node_network_receive_bytes_total,device={instance}). If label values can contain commas or parentheses, quote the filter: "device={instance}".

  • Several labels identify the series — use method Agent Table with the table Prometheus.Series(node,node_filesystem_avail_bytes) and the metric Prometheus.Value(node,node_filesystem_avail_bytes,{instance}). The instance is the LABELS column, which contains all labels of the series in label=value,…​ form, already quoted where needed, so it expands to one filter per label.

With the Agent Table method, the instance contains all labels of the series. If the value of any label changes (for example, a version label after an upgrade), the series gets a new instance and a new DCI.

Collecting Received and Mapped Data

  1. Configure metric mappings in the [Prometheus] section (see Metric Mappings) and restart the agent.

  2. For remote write, enable the receiver and configure the sender. For a Prometheus server, add a remote_write entry to prometheus.yml, and limit the sent series to the mapped metrics with write_relabel_configs:

    remote_write:
      - url: "http://192.0.2.10:9091/api/v1/write"
        write_relabel_configs:
          - source_labels: [__name__]
            regex: "node_network_transmit_drop_total|node_uname_info"
            action: keep

    The agent configuration for this sender:

    [Prometheus]
    Address = 192.0.2.10
    Port = 9091
    Metric = node_network_transmit_drop_total:Net.Interface.TxDrops:nodename,device
    Metric = node_uname_info:Node.Uname:nodename:sysname,release
  3. On the node of the agent, create DCIs with origin Push and the pushed names as metric names, for example Net.Interface.TxDrops(server1,eth0). Choose a numeric data type for mappings that push the sample value, and String for mappings with value labels.

To create the DCIs automatically, use instance discovery method Push on a prototype DCI with origin Push:

  • Metric name — the pushed name with {instance} in place of the varying argument, for example Net.Interface.TxDrops(server1,{instance}).

  • Pattern — a regular expression that matches the pushed names (case-insensitive); the first capturing group is the instance, for example ^Net\.Interface\.TxDrops\(server1,(.+)\)$.

When a value arrives for a name without a DCI, the server matches the name against the pattern, creates the DCI for the instance (if the instance filter accepts it), and stores the value in it.

Reference

Agent Metrics

These metrics are provided by the prometheus subagent. They read the samples that the subagent keeps in memory from the last successful scrape of each scrape target; a metric request does not contact the endpoint. Data received through the remote write receiver is not available through these metrics.

The first argument of every metric is the name of a scrape target, as configured in its [Prometheus/Targets/target] section; it is matched case-insensitively. An unknown target name results in the error no such instance.

Metrics that select a series take optional label=value arguments after the fixed arguments. A series matches when it has every given label with exactly the given value (case-sensitive); the first matching series in the order of the endpoint response is used. Metric names are matched exactly (case-sensitive). Put an argument in double or single quotes if it contains a comma or a parenthesis, for example "path=/a,b" or path="/a,b". For details and examples, see Selecting a Series.

Series Metrics

Prometheus.LabelValue(target, metric, label[, label=value …​])

Value of label label of the first series of metric that matches the label filters. Use it for info-style metrics that carry their data in labels, for example node_uname_info or node_exporter_build_info.

Returns no such instance when no series matches or the matching series does not have the label label, and an error when the target has had no successful scrape since the agent started.

Data Type

String

Platforms

All (prometheus)

Table 1. Arguments
Argument Type Description

target

String

Scrape target name

metric

String

Metric name as exposed by the endpoint, for example node_uname_info

label

String

Name of the label whose value is returned

label=value

String

Optional, any number: label filters that select the series

Prometheus.Value(target, metric[, label=value …​])

Value of the first series of metric that matches the label filters. Without label filters, the first series of the metric is used.

Returns no such instance when no series matches, and an error when the target has had no successful scrape since the agent started.

Data Type

Float

Platforms

All (prometheus)

Table 2. Arguments
Argument Type Description

target

String

Scrape target name

metric

String

Metric name as exposed by the endpoint, including any suffix such as _total, _bucket, _sum, or _count

label=value

String

Optional, any number: label filters that select the series

Target Metrics

Prometheus.Target.LastScrapeTime(target)

Time of the last scrape attempt of the target, successful or not, as UNIX time. 0 before the first scrape attempt.

Data Type

Unsigned Integer 64

Platforms

All (prometheus)

Table 3. Arguments
Argument Type Description

target

String

Scrape target name

Prometheus.Target.SampleCount(target)

Number of samples kept from the last successful scrape of the target (one sample per series). A failed scrape does not change it; 0 before the first successful scrape.

Data Type

Unsigned Integer

Platforms

All (prometheus)

Table 4. Arguments
Argument Type Description

target

String

Scrape target name

Prometheus.Target.Status(target)

Result of the last scrape attempt of the target: 1 if it succeeded, 0 if it failed. 0 before the first scrape attempt.

Data Type

Integer

Platforms

All (prometheus)

Table 5. Arguments
Argument Type Description

target

String

Scrape target name

Lists

Prometheus.LabelValues(target, metric, label) (List)

Distinct values of label label across all series of metric, in the order of first appearance in the endpoint response. Series without the label are skipped. The list is empty when the metric is not in the last scrape. Use it as the source of instance discovery with method Agent List when one label identifies a series.

Data Type

List

Platforms

All (prometheus)

Table 6. Arguments
Argument Type Description

target

String

Scrape target name

metric

String

Metric name

label

String

Label name

Prometheus.Targets (List)

Names of the configured scrape targets.

Data Type

List

Platforms

All (prometheus)

Tables

Prometheus.Series(target, metric) (Table)

All series of metric from the last successful scrape, one row per series. The table is empty when the metric is not in the last scrape. Use it as the source of instance discovery with method Agent Table when several labels identify a series.

Data Type

Table

Platforms

All (prometheus)

Table 7. Arguments
Argument Type Description

target

String

Scrape target name

metric

String

Metric name

Table 8. Columns
Column Type Description

LABELS

String

All labels of the series in the form label=value,label=value, in the order of the endpoint response (instance key). A value that contains (, ), ,, ", or ' is enclosed in double quotes, with " doubled, so the column value can be passed unchanged as label filters to Prometheus.Value and Prometheus.LabelValue.

VALUE

Float

Sample value

label name

String

One column for each label name that appears in any series of the metric, named after the label, in the order of first appearance. Empty for a series without that label. A label named LABELS or VALUE (in any letter case) gets no column of its own; it is still part of the LABELS column.

Prometheus.Targets (Table)

Configured scrape targets with the result of their last scrape.

Data Type

Table

Platforms

All (prometheus)

Table 9. Columns
Column Type Description

NAME

String

Target name (instance key)

URL

String

Endpoint URL

INTERVAL

Unsigned Integer

Scrape interval in seconds

STATUS

Integer

Result of the last scrape attempt: 1 = success, 0 = failure or no attempt yet

LAST_SCRAPE

Unsigned Integer 64

Time of the last scrape attempt (UNIX time); 0 before the first attempt

SAMPLES

Unsigned Integer

Number of samples kept from the last successful scrape

Troubleshooting

  • Prometheus.* metrics are not supported — check that the subagent is loaded (nxget -l <agent-address> Agent.SubAgentList lists PROMETHEUS). If it is not, check the agent log: the subagent does not load when the remote write receiver cannot start.

  • Prometheus.Value returns an error — the target has had no successful scrape since the agent started. Check Prometheus.Target.Status and the debug output of the prometheus tag.

  • Prometheus.Value returns no such instance — the target name is unknown, the metric is not in the last scrape, or no series matches the label filters. Compare the arguments with nxget -T <agent-address> "Prometheus.Series(<target>,<metric>)".

  • Mapped values do not reach the DCIs — check that the pushed name in the agent debug output (tag prometheus, level 7) is exactly the DCI metric name, and that the DCI has origin Push and is active.

To see details, enable these debug tags (agent tags with DebugTags in nxagentd.conf, server tags with the debug server console command; see Diagnostics):

Debug tag Component

prometheus

Agent: prometheus subagent. Level 3: loaded targets and mappings; level 5: failed scrapes with the HTTP status or libcurl error; level 6: completed scrapes with sample counts, lines that cannot be parsed, received samples without a mapping; level 7: pushed values, remote write requests that cannot be decoded; level 9: content of received remote write requests.

dc.push

Agent: values sent to the server

agent.conn

Server: values pushed by agents (level 5)

poll.instance

Server: DCIs created by instance discovery method Push