OpenTelemetry (OTLP) Export
The otlp server module sends NetXMS data to an OpenTelemetry collector or any other OTLP receiver:
-
Metric export sends every new numeric DCI value as an OTLP data point.
-
Event forwarding sends selected events as OTLP log records, through an event forwarder and the Forward event action.
Both use OTLP over HTTP with binary protobuf encoding (application/x-protobuf); gRPC and JSON encoding are not supported.
Traces are not exported.
The same module also receives OTLP data; see OpenTelemetry (OTLP) Data Collection. Receiving and exporting are configured separately.
Loading the Module
Add the module to netxmsd.conf, before any named section, and restart the server:
Module = otlp
Loading the module does not send anything. Metric export starts when an endpoint is configured, and events are sent only to the event forwarders you create.
Metric Export
Configuration
Set the collector URL in the [OpenTelemetry/MetricExport] section of netxmsd.conf and restart the server:
[OpenTelemetry/MetricExport]
Endpoint = https://collector.example.com:4318/v1/metrics
AuthToken = collector-token
Header = X-Tenant: monitoring
The module registers a performance data storage driver named OTLP when Endpoint is set.
Do not set PerfDataStorageDriver for it: that setting loads drivers from separate files.
| Parameter | Default | Description |
|---|---|---|
|
(empty) |
Full URL of the receiver, including the path (usually |
|
(empty) |
Token sent as |
|
(none) |
Additional HTTP header, as |
|
|
Verify the receiver’s TLS certificate and host name |
|
(empty) |
Value of the |
|
|
Number of queued data points that starts a send |
|
|
Seconds to wait for more data points before sending what is queued |
|
|
Timeout of one HTTP request, in seconds |
|
|
Maximum number of data points waiting to be sent; 0 means no limit (see Delivery and Data Loss) |
Exported Values
Every new value of a single-value DCI is exported, whatever its origin, after transformation and delta calculation. Store only changed values and the sample saving interval apply as they do to storing the value, but a value is exported even when the DCI keeps no history in the database. Table DCIs, string values, and values that are not numbers are not exported. Integer values are sent as 64-bit signed integers, so unsigned 64-bit values above 263-1 become negative.
To leave out data, add the custom attribute pds:ignore or otel:ignore (any value, even false) to the DCI’s owner object, its template, or its related object.
Metrics and Data Points
The OTLP metric name is the DCI’s metric name.
For SNMP and Modbus DCIs, and internal Dummy DCIs, whose names are addresses or placeholders, the DCI description is used instead.
The DCI description is sent as the metric description when it differs from the name, and the DCI’s unit as the metric unit.
| DCI | OTLP metric type |
|---|---|
No delta calculation, counter data type |
Cumulative monotonic sum |
No delta calculation, other numeric type |
Gauge |
Simple delta |
Delta sum (monotonic for counter data types) |
Average per second or per minute |
Gauge |
Data point attributes:
| Attribute | Description |
|---|---|
|
DCI ID |
|
Data source (origin) of the DCI |
|
Instance, for DCIs created by instance discovery |
|
Name of the related object |
<name> |
One attribute for each custom attribute named |
Data points are grouped by the owner object, whose resource attributes are described in Resource Attributes.
Event Forwarding
To send events as OTLP log records:
-
In Configuration > Event forwarders, create a forwarder with the driver
otlpand a JSON configuration, for example:{ "endpoint": "https://collector.example.com:4318/v1/logs", "authToken": "collector-token", "headers": { "X-Tenant": "monitoring" } } -
In the event processing policy, add a Forward event action that uses this forwarder to the rules whose events should be sent (see Forward Event). The action’s Recipient, when set, is added to the record’s resource as
netxms.recipient.
| Key | Default | Description |
|---|---|---|
|
(required) |
Full URL of the receiver, including the path (usually |
|
(empty) |
Token sent as |
|
(none) |
Additional HTTP headers, as an object of header names and string values |
|
|
Verify the receiver’s TLS certificate and host name |
|
(empty) |
Value of the |
|
|
Number of queued records that starts a send |
|
|
Seconds to wait for more records before sending what is queued |
|
|
Timeout of one HTTP request, in seconds |
|
|
Maximum number of records waiting to be sent; 0 means no limit |
Events that were created from received OpenTelemetry logs are not sent back.
Log Records
The record time is the event time, the body is the event message, and the severity is mapped as follows:
| Event severity | Severity number | Severity text |
|---|---|---|
Normal |
9 (INFO) |
|
Warning |
13 (WARN) |
|
Minor |
16 (WARN4) |
|
Major |
17 (ERROR) |
|
Critical |
21 (FATAL) |
|
| Attribute | Description |
|---|---|
|
Event ID |
|
Event code |
|
Event name |
|
Event origin |
|
Event tags, as one comma-separated string |
|
ID of the DCI that generated the event, when there is one |
|
Each named event parameter; parameters without a name are |
The resource is the event’s source object.
An event without a source object is sent with host.name set to netxms-server, and with service.name from serviceName or else netxms-server.
Resource Attributes
Metric data points and log records carry the attributes of their object as the OTLP resource:
| Attribute | Description |
|---|---|
|
|
|
Object name |
|
Object GUID |
Delivery and Data Loss
Data points and log records are queued in memory and sent in the background, one request per send. A send starts when the queue reaches the batch size or the flush interval has passed.
-
A failed request (an HTTP error status or a connection error) is repeated twice, immediately. If the third attempt fails, the batch is discarded and a warning is written to the server log.
-
When the queue is full, new data is discarded. The queue is kept in memory only: at a normal shutdown the server tries to send it once more, and data not sent then is lost.
-
Data that was not sent is not sent later: there is no backfill after the receiver is reachable again.
-
Data points that the receiver reports as rejected are logged but not sent again.
The metric export queue can be watched with the internal metrics Server.PDS.DriverStat(OTLP,queueSize) and Server.PDS.DriverStat(OTLP,messageDrops) on the management server node.
An event forwarder also has its own queue, limited by EventForwarders.MaxQueueSize (see Forward Event).
There are no settings for a custom CA certificate or a client certificate.
Troubleshooting
Set these debug tags (see Debug Console):
-
otlp.metrics— metric export; level 6 logs delivered batches, level 7 every request and the DCIs that were skipped -
otlp.export— event forwarding -
pdsdrv— registration of theOTLPdriver -
event.forwarder— the event forwarder queue