OpenTelemetry (OTLP) Data Collection

NetXMS can receive metrics and log records over the OpenTelemetry protocol (OTLP). Senders push data to the server: an OpenTelemetry Collector, an application instrumented with an OpenTelemetry SDK, or any device with a built-in OTLP exporter. The server matches each sender to a node, stores metric values in DCIs with the OTLP origin, and passes log records to the OpenTelemetry log parser.

For DCI configuration concepts and properties, see DCI Reference. To send NetXMS metrics and events to an OpenTelemetry collector, see OpenTelemetry (OTLP) Export.

Enabling the Receiver

The receiver is provided by the otlp server module. To load the module, add it to netxmsd.conf and restart the server:

Module = otlp

The module adds two endpoints to the built-in web API listener:

Endpoint Data

POST /otlp-backend/v1/metrics

Metrics (OTLP ExportMetricsServiceRequest)

POST /otlp-backend/v1/logs

Log records (OTLP ExportLogsServiceRequest)

The listener address, port (8000 by default), and TLS settings are configured in the [WEBAPI] section of netxmsd.conf; see Enabling the API. By default the listener accepts connections on the loopback interface only, so set Address in the [WEBAPI] section to receive data from other hosts. The webapi module is not required for OTLP.

Only OTLP over HTTP is supported; there is no OTLP/gRPC endpoint and no endpoint for traces.

Encoding and Compression

The request body can be binary protobuf (Content-Type: application/x-protobuf) or JSON (Content-Type: application/json) Since 6.2.3. Requests with any other content type are rejected with status 415. The response uses the same encoding as the request.

Request bodies compressed with Content-Encoding: gzip or deflate are decompressed automatically.

Authentication

Both endpoints require an authentication token in the Authorization header:

Authorization: Bearer <token>

For a sender that runs unattended, issue a persistent token as described in Persistent Tokens. The server accepts a valid token of any enabled user and does not check the user’s access rights to the nodes that receive the data. Use a dedicated user account for OTLP senders, so that the token can be revoked without affecting other integrations.

A request without a valid token is rejected with status 401; a token of a disabled user is rejected with status 403.

Matching Senders to Nodes

Each block of OTLP data carries resource attributes that describe its source, such as host.name or service.name. The server uses matching rules to find the node that a resource belongs to. Data from a resource that matches no node is discarded.

Rules are evaluated in order of their sequence number, and the first rule that finds a node wins. A rule takes the value of one resource attribute, optionally transforms it with a regular expression, and compares the result with one node property:

Node property Comparison

Node name

Object name of the node (case-insensitive)

DNS name

Primary host name of the node (case-insensitive)

IP address

IP address of the node, in any zone

Custom attribute

Value of the named custom attribute of the node (case-insensitive)

If a rule has a regular expression, the value of its first capture group is used (the whole match if the expression has no groups). If the expression does not match the attribute value, the rule is skipped.

A new database contains three rules:

Sequence Attribute Node property

1

host.name

Node name

2

host.name

DNS name

3

host.ip

IP address

Only string, integer, double, and boolean attribute values are used for matching. An array value is treated as empty, so the default host.ip rule matches only when the sender sets host.ip as a single string.

The management client has no editor for matching rules. The rules are stored in the otlp_matching_rules database table and are read when the server starts.

Column Description

id

Unique rule ID

sequence_number

Evaluation order

otel_attribute

Name of the resource attribute

regex_transform

Regular expression applied to the attribute value, or NULL

node_property

0 — node name, 1 — DNS name, 2 — IP address, 3 — custom attribute

custom_attr_name

Name of the custom attribute (for node_property 3)

Match Cache

A successful match is cached for the full set of resource attributes. For the time set in OTLP.MatchCacheTTL (300 seconds by default), data with the same resource attributes goes to the cached node without evaluating the rules again. A change of the node name, host name, IP address, or custom attribute therefore takes effect for a known sender only after the cache entry expires. Failed matches are not cached.

To see which resources do not match, set the OTLP.LogUnmatchedResources server configuration variable to 1 and the otlp debug tag to level 3 or higher (see Enabling Debug Logging). The server log then contains a No matching node found for OTel resource message with the resource attributes of each unmatched resource.

Collecting Metrics

Creating an OTLP DCI

  1. Open the node’s Data Collection view and create a new DCI

  2. Set Origin to OTLP

  3. Set Metric to the OTLP metric name, for example system.memory.usage, or click the selection button to pick one from the metrics already received for the node

  4. Configure thresholds and other properties as for any other DCI

OTLP DCIs have no polling schedule: a value is stored whenever a data point for the metric arrives. The data point time is used as the value timestamp. The Source node setting is not used. OTLP data is accepted only by single-value DCIs, not by table DCIs.

A DCI receives a data point when the DCI is active, has the OTLP origin, and its metric name equals the OTLP metric name (case-insensitive). If a metric has several data points with different attributes (for example, one per memory state), all of them are stored in the same DCI. To get a separate DCI for each attribute combination, use instance discovery (see Instance Discovery).

Metric types are processed as follows:

Gauge, non-monotonic sum

The data point value is stored as is.

Monotonic sum (counter)

The value is converted to a rate per second: the difference from the previous data point divided by the time between the two data points. The first data point, also after a server restart, only sets the starting point. If the counter decreases, it is treated as reset to zero.

Histogram

Each data point is split into several values, each with an extra otel.bucket attribute: le<bound> for each bucket with an explicit upper bound (for example, le0.5), inf for the last bucket, count, and, when present in the data point, sum, min, and max. Use instance discovery with otel.bucket as a key attribute to store them in separate DCIs.

Other metric types (exponential histogram, summary) are ignored.

Metric Selector

The server keeps, for each node, a list of the OTLP metrics it has received: metric name, type, and attribute names (not values). The selection buttons on the General and Instance Discovery pages of the DCI properties show this list. A metric is removed from the list when no data point for it arrives within the time set in OTLP.MetricCatalogRetention (86400 seconds by default).

The list is kept in memory and is empty after a server restart until data arrives again. It contains only metrics from resources that matched the node, so an empty list for a node that should receive data can indicate a matching problem. When a DCI is edited on a template, the list is empty; enter the metric name manually.

The same list is available through the web API as GET /v1/objects/{object-id}/otlp-metrics.

Instance Discovery

The OTLP instance discovery method creates one DCI for each distinct combination of selected data point attributes. Unlike other methods, it does not run during instance discovery polls: an instance DCI is created when the first data point for a new instance arrives.

  1. Create a DCI with the OTLP origin and a metric name that contains {instance} (for example, {instance})

  2. On the Instance Discovery page, set Instance discovery method to OTLP

  3. Set Metric name pattern to the OTLP metric name, or to a pattern with * and ? wildcards (case-insensitive); the selection button lets you pick a received metric and its key attributes

  4. Optionally set Instance key attributes — a comma-separated list of data point attribute names that identify an instance

  5. Optionally set Instance name format — the display name of an instance, with %{attribute} placeholders for data point attribute values (for example, %{state}); when empty, the instance key is used

  6. Optionally set an instance filter script

The prototype DCI must not have the plain OTLP metric name as its metric name: an active OTLP DCI with that name receives the values directly, and instance discovery is not used.

The instance key is the metric name followed by the values of the key attributes, separated by colons (for example, system.memory.usage:used). If no key attributes are set, the values of all data point attributes are used, ordered by attribute name. In the instance DCI, {instance} is replaced with the instance key and {instance-name} with the instance name.

An instance DCI that has been disabled receives no data and is not created again. Instance DCIs are not removed when data for them stops arriving, and the instance retention settings do not apply to them; delete unused instance DCIs manually.

Receiving Logs

Log records sent to /otlp-backend/v1/logs are matched to nodes with the same matching rules as metrics. Records from a resource that matches no node are discarded, and the response reports them as rejected in its partial_success field.

Each record is evaluated by the OpenTelemetry log parser, configured in Configuration > OpenTelemetry log parser (see Events and Log Parser Reference). It uses the same rule format and matching as the syslog parser, with its own set of rules. The parser match conditions use these record fields:

Parser element OpenTelemetry field

<source>, <tag>

service.name resource attribute

<id>, <facility>

Severity number (1 to 24; 0 if not set)

<severity>

Severity band: TRACE, DEBUG, INFO, and unset severity (severity number 0 to 12) set flag 1; WARN (13 to 16) set 2; ERROR (17 to 20) set 8; FATAL (21 to 24) set 16

<logName>

Instrumentation scope name

Match pattern

Log record body

An event generated by the parser has these named parameters, in this order:

  1. The capture groups of the match pattern: named groups keep their names, unnamed groups are named group_1, group_2, and so on, and optional groups that did not match are omitted (see Log Parser Reference).

  2. otelService (service name), otelScope (instrumentation scope name), otelSeverityNumber (severity number, 0 if not set), otelSeverity (the severity band flag 1, 2, 8, or 16 from the table above, not a NetXMS severity), repeatCount, and message (the record body).

  3. One parameter for each resource and record attribute. If a resource attribute and a record attribute have the same name, the resource attribute value is used.

Parameter names are not unique: a name that occurs twice is added twice, and %<name> returns the first one (names are compared without regard to case). A capture group named message therefore hides the record body in %<message>, while an attribute with that name does not. The tag of the matching rule is set as the event tag; there is no eventTag parameter.

When OTLP.Logs.EnableStorage is enabled (the default), the server also stores the records in the database, unless the matching parser rule has doNotSaveToDatabase set. Stored records are kept for OTLP.Logs.RetentionTime days (90 by default) and can be viewed in the OpenTelemetry log in the Logs perspective; viewing them requires the View syslog system right.

For the server configuration variables, see OTLP server variables.

Sending Data from an OpenTelemetry Collector

The otlphttp exporter of the OpenTelemetry Collector sends metrics to <endpoint>/v1/metrics and logs to <endpoint>/v1/logs. Set its endpoint to the /otlp-backend path of the NetXMS web API listener and use it in the metrics and logs pipelines only.

  1. Issue a persistent authentication token for the user account that the collector will use

  2. Make sure the [WEBAPI] listener of the server accepts connections from the collector host

  3. Add an exporter to the collector configuration and use it in the pipelines, for example:

    receivers:
      otlp:
        protocols:
          http:
      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
          memory:
          filesystem:
    
    processors:
      resourcedetection:
        detectors: [system]
        override: false
      batch:
    
    exporters:
      otlphttp/netxms:
        endpoint: http://netxms.example.com:8000/otlp-backend
        headers:
          Authorization: "Bearer <token>"
    
    service:
      pipelines:
        metrics:
          receivers: [otlp, hostmetrics]
          processors: [resourcedetection, batch]
          exporters: [otlphttp/netxms]
        logs:
          receivers: [otlp]
          processors: [resourcedetection, batch]
          exporters: [otlphttp/netxms]

    The resourcedetection processor sets the host.name resource attribute from the collector host, which the default matching rules compare with the node name and DNS name. override: false keeps the host.name that a remote sender already set; without it, data forwarded through the collector would be matched to the collector’s node. For a central collector that receives data from senders without host.name, use separate pipelines so that the collector’s host name is applied only to its own hostmetrics data. Recent Collector releases name these components otlp_http, resource_detection, and host_metrics, and accept the names used here as deprecated aliases. When the TLS listener is enabled, use https:// and the TLS port (8443 by default) in the endpoint.

  4. Restart the collector and wait for the first export

  5. Open the properties of an OTLP DCI on the node and click the metric selection button: the received metrics are listed

  6. Create OTLP DCIs for the metrics to store

An application instrumented with an OpenTelemetry SDK can send data directly, without a collector. Most SDKs read the exporter settings from environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=http://netxms.example.com:8000/otlp-backend
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>
OTEL_TRACES_EXPORTER=none
OTEL_RESOURCE_ATTRIBUTES=host.name=web01

OTEL_RESOURCE_ATTRIBUTES sets the resource attribute used for node matching; set it to the name of the node in NetXMS.

Troubleshooting

No data arrives
  • Check that the otlp module is loaded and the [WEBAPI] listener accepts connections from the sender host

  • Check the HTTP status that the sender reports: 401 or 403 means a token problem, 415 an unsupported content type, 404 a wrong endpoint path, 503 a server in HA standby role

The metric selector is empty for the node

The resources may not be matched to the node. First check that the sender exports a supported metric type and that data arrived since the last server restart. Then enable OTLP.LogUnmatchedResources and check the resource attributes in the server log (see Match Cache).

The metric is listed but the DCI receives no values
  • Check that the DCI is active, has the OTLP origin, and its metric name equals the OTLP metric name

  • For a monotonic sum, no value is stored until a second data point with a later timestamp arrives; the server then stores the rate

For detailed processing messages, set the otlp debug tag to level 6 or higher.