OpenTelemetry (OTLP) Export

The otlp server module sends NetXMS data to an OpenTelemetry collector or any other OTLP receiver:

  • Metric export sends every new numeric DCI value as an OTLP data point.

  • Event forwarding sends selected events as OTLP log records, through an event forwarder and the Forward event action.

Both use OTLP over HTTP with binary protobuf encoding (application/x-protobuf); gRPC and JSON encoding are not supported. Traces are not exported.

The same module also receives OTLP data; see OpenTelemetry (OTLP) Data Collection. Receiving and exporting are configured separately.

Loading the Module

Add the module to netxmsd.conf, before any named section, and restart the server:

Module = otlp

Loading the module does not send anything. Metric export starts when an endpoint is configured, and events are sent only to the event forwarders you create.

Metric Export

Configuration

Set the collector URL in the [OpenTelemetry/MetricExport] section of netxmsd.conf and restart the server:

[OpenTelemetry/MetricExport]
Endpoint = https://collector.example.com:4318/v1/metrics
AuthToken = collector-token
Header = X-Tenant: monitoring

The module registers a performance data storage driver named OTLP when Endpoint is set. Do not set PerfDataStorageDriver for it: that setting loads drivers from separate files.

Parameter Default Description

Endpoint

(empty)

Full URL of the receiver, including the path (usually /v1/metrics); no path is added. Metric export is off while it is empty.

AuthToken

(empty)

Token sent as Authorization: Bearer <token>

Header

(none)

Additional HTTP header, as Name: value. Can be specified multiple times.

VerifyPeer

yes

Verify the receiver’s TLS certificate and host name

ServiceName

(empty)

Value of the service.name resource attribute; when empty, the object’s name is used (see Resource Attributes)

BatchSize

100

Number of queued data points that starts a send

FlushInterval

5

Seconds to wait for more data points before sending what is queued

Timeout

30

Timeout of one HTTP request, in seconds

QueueSizeLimit

10000

Maximum number of data points waiting to be sent; 0 means no limit (see Delivery and Data Loss)

Exported Values

Every new value of a single-value DCI is exported, whatever its origin, after transformation and delta calculation. Store only changed values and the sample saving interval apply as they do to storing the value, but a value is exported even when the DCI keeps no history in the database. Table DCIs, string values, and values that are not numbers are not exported. Integer values are sent as 64-bit signed integers, so unsigned 64-bit values above 263-1 become negative.

To leave out data, add the custom attribute pds:ignore or otel:ignore (any value, even false) to the DCI’s owner object, its template, or its related object.

Metrics and Data Points

The OTLP metric name is the DCI’s metric name. For SNMP and Modbus DCIs, and internal Dummy DCIs, whose names are addresses or placeholders, the DCI description is used instead. The DCI description is sent as the metric description when it differs from the name, and the DCI’s unit as the metric unit.

DCI OTLP metric type

No delta calculation, counter data type

Cumulative monotonic sum

No delta calculation, other numeric type

Gauge

Simple delta

Delta sum (monotonic for counter data types)

Average per second or per minute

Gauge

Data point attributes:

Attribute Description

netxms.dci.id

DCI ID

netxms.data_source

Data source (origin) of the DCI

netxms.instance

Instance, for DCIs created by instance discovery

netxms.related_object

Name of the related object

<name>

One attribute for each custom attribute named tag:<name> or tag_<name> with a value, on the template, the owner object, or the related object

Data points are grouped by the owner object, whose resource attributes are described in Resource Attributes.

Event Forwarding

To send events as OTLP log records:

  1. In Configuration > Event forwarders, create a forwarder with the driver otlp and a JSON configuration, for example:

    {
      "endpoint": "https://collector.example.com:4318/v1/logs",
      "authToken": "collector-token",
      "headers": { "X-Tenant": "monitoring" }
    }
  2. In the event processing policy, add a Forward event action that uses this forwarder to the rules whose events should be sent (see Forward Event). The action’s Recipient, when set, is added to the record’s resource as netxms.recipient.

Key Default Description

endpoint

(required)

Full URL of the receiver, including the path (usually /v1/logs)

authToken

(empty)

Token sent as Authorization: Bearer <token>

headers

(none)

Additional HTTP headers, as an object of header names and string values

verifyPeer

true

Verify the receiver’s TLS certificate and host name

serviceName

(empty)

Value of the service.name resource attribute; when empty, the object’s name is used

batchSize

100

Number of queued records that starts a send

flushInterval

5

Seconds to wait for more records before sending what is queued

timeout

30

Timeout of one HTTP request, in seconds

queueSizeLimit

10000

Maximum number of records waiting to be sent; 0 means no limit

Events that were created from received OpenTelemetry logs are not sent back.

Log Records

The record time is the event time, the body is the event message, and the severity is mapped as follows:

Event severity Severity number Severity text

Normal

9 (INFO)

NORMAL

Warning

13 (WARN)

WARNING

Minor

16 (WARN4)

MINOR

Major

17 (ERROR)

MAJOR

Critical

21 (FATAL)

CRITICAL

Attribute Description

event.id

Event ID

event.code

Event code

event.name

Event name

event.origin

Event origin

event.tags

Event tags, as one comma-separated string

dci.id

ID of the DCI that generated the event, when there is one

param.<name>

Each named event parameter; parameters without a name are param.<number>, counted from 1

The resource is the event’s source object. An event without a source object is sent with host.name set to netxms-server, and with service.name from serviceName or else netxms-server.

Resource Attributes

Metric data points and log records carry the attributes of their object as the OTLP resource:

Attribute Description

service.name

ServiceName (metric export) or serviceName (event forwarder), or else the object’s name

host.name

Object name

host.id

Object GUID

Delivery and Data Loss

Data points and log records are queued in memory and sent in the background, one request per send. A send starts when the queue reaches the batch size or the flush interval has passed.

  • A failed request (an HTTP error status or a connection error) is repeated twice, immediately. If the third attempt fails, the batch is discarded and a warning is written to the server log.

  • When the queue is full, new data is discarded. The queue is kept in memory only: at a normal shutdown the server tries to send it once more, and data not sent then is lost.

  • Data that was not sent is not sent later: there is no backfill after the receiver is reachable again.

  • Data points that the receiver reports as rejected are logged but not sent again.

The metric export queue can be watched with the internal metrics Server.PDS.DriverStat(OTLP,queueSize) and Server.PDS.DriverStat(OTLP,messageDrops) on the management server node. An event forwarder also has its own queue, limited by EventForwarders.MaxQueueSize (see Forward Event).

There are no settings for a custom CA certificate or a client certificate.

Troubleshooting

Set these debug tags (see Debug Console):

  • otlp.metrics — metric export; level 6 logs delivered batches, level 7 every request and the DCIs that were skipped

  • otlp.export — event forwarding

  • pdsdrv — registration of the OTLP driver

  • event.forwarder — the event forwarder queue