This is the documentation for the development version. There are no official binaries for it; build it from the source code in git. Download links on these pages work only after the release. For the current release, see version 7.0.

OpenTelemetry (OTLP) Export

The otlp server module sends NetXMS data to an OpenTelemetry collector or any other OTLP receiver:

  • Metric export sends every new numeric DCI value as an OTLP data point.

  • Event forwarding sends selected events as OTLP log records, through an event forwarder and the Forward event action.

Both use OTLP over HTTP with binary protobuf encoding (application/x-protobuf); gRPC and JSON encoding are not supported. Traces are not exported.

The same module also receives OTLP data; see OpenTelemetry (OTLP) Data Collection. Receiving and exporting are configured separately.

Loading the Module

Add the module to netxmsd.conf, before any named section, and restart the server:

Module = otlp

Loading the module does not send anything. Metric export starts when an endpoint is configured, and events are sent only to the event forwarders you create.

Metric Export

Configuration

Set the collector URL in the [OpenTelemetry/MetricExport] section of netxmsd.conf and restart the server:

[OpenTelemetry/MetricExport]
Endpoint = https://collector.example.com:4318/v1/metrics
AuthToken = collector-token
Header = X-Tenant: monitoring

The module registers a performance data storage driver named OTLP when Endpoint is set. Do not set PerfDataStorageDriver for it: that setting loads drivers from separate files.

Parameter Default Description

Endpoint

(empty)

Full URL of the receiver, including the path (usually /v1/metrics); no path is added. Metric export is off while it is empty.

AuthToken

(empty)

Token sent as Authorization: Bearer <token>

Header

(none)

Additional HTTP header, as Name: value. Can be specified multiple times.

VerifyPeer

yes

Verify the receiver’s TLS certificate and host name

ServiceName

(empty)

Value of the service.name resource attribute (see Service)

ServiceNameFallbackSource

object-name

What service.name is when neither the object nor ServiceName sets it: object-name (the object’s name) or fixed (the text netxms)

ServiceNamespace

(empty)

Value of the service.namespace resource attribute (see Service)

BatchSize

100

Number of queued data points that starts a send

FlushInterval

5

Seconds to wait for more data points before sending what is queued

Timeout

30

Timeout of one HTTP request, in seconds

QueueSizeLimit

10000

Maximum number of data points waiting to be sent; 0 means no limit (see Delivery and Data Loss)

Exported Values

Every new value of a single-value DCI is exported, whatever its origin, after transformation and delta calculation. Store only changed values and the sample saving interval apply as they do to storing the value, but a value is exported even when the DCI keeps no history in the database. Table DCIs, string values, and values that are not numbers are not exported. Integer values are sent as 64-bit signed integers, so unsigned 64-bit values above 263-1 become negative.

To leave out data, add the custom attribute pds:ignore or otel:ignore (any value, even false) to the DCI’s owner object, its template, or its related object.

Metrics and Data Points

The OTLP metric name is the DCI’s metric name. For SNMP and Modbus DCIs, and internal Dummy DCIs, whose names are addresses or placeholders, the DCI description is used instead. The DCI description is sent as the metric description when it differs from the name, and the DCI’s unit as the metric unit.

DCI OTLP metric type

No delta calculation, counter data type

Cumulative monotonic sum

No delta calculation, other numeric type

Gauge

Simple delta

Delta sum (monotonic for counter data types)

Average per second or per minute

Gauge

Data point attributes:

Attribute Description

netxms.dci.id

DCI ID

netxms.data_source

Data source (origin) of the DCI

netxms.instance

Instance, for DCIs created by instance discovery

netxms.related_object

Name of the related object

netxms.dci.user_tag

DCI tag

netxms.dci.system_tag

DCI system tag (Interpretation)

<name>

One attribute for each custom attribute named tag:<name> or tag_<name> with a value, on the template, the owner object, or the related object

Data points are grouped by the owner object, whose resource attributes are described in Resource Attributes.

Event Forwarding

To send events as OTLP log records:

  1. In Configuration > Event forwarders, create a forwarder with the driver otlp and a JSON configuration, for example:

    {
      "endpoint": "https://collector.example.com:4318/v1/logs",
      "authToken": "collector-token",
      "headers": { "X-Tenant": "monitoring" }
    }
  2. In the event processing policy, add a Forward event action that uses this forwarder to the rules whose events should be sent (see Forward Event). The action’s Recipient, when set, is added to the record’s resource as netxms.recipient.

Key Default Description

endpoint

(required)

Full URL of the receiver, including the path (usually /v1/logs)

authToken

(empty)

Token sent as Authorization: Bearer <token>

headers

(none)

Additional HTTP headers, as an object of header names and string values

verifyPeer

true

Verify the receiver’s TLS certificate and host name

serviceName

(empty)

Value of the service.name resource attribute

serviceNameFallbackSource

object-name

object-name or fixed, as for metric export

serviceNamespace

(empty)

Value of the service.namespace resource attribute

batchSize

100

Number of queued records that starts a send

flushInterval

5

Seconds to wait for more records before sending what is queued

timeout

30

Timeout of one HTTP request, in seconds

queueSizeLimit

10000

Maximum number of records waiting to be sent; 0 means no limit

Events that were created from received OpenTelemetry logs are not sent back.

Log Records

The record time is the event time, the body is the event message, and the severity is mapped as follows:

Event severity Severity number Severity text

Normal

9 (INFO)

NORMAL

Warning

13 (WARN)

WARNING

Minor

16 (WARN4)

MINOR

Major

17 (ERROR)

MAJOR

Critical

21 (FATAL)

CRITICAL

Attribute Description

event.id

Event ID

event.code

Event code

event.name

Event name

event.origin

Event origin

event.tags

Event tags, as an array of strings

dci.id

ID of the DCI that generated the event, when there is one

param.<name>

Each named event parameter; parameters without a name are param.<number>, counted from 1

The resource is the event’s source object. An event without a source object uses the management server node.

Resource Attributes

Metric data points and log records carry the attributes of their object as the OTLP resource.

Service

service.name is taken from the first of:

  1. The custom attribute otel:service.name of the object, its own or inherited.

  2. ServiceName (metric export) or serviceName (event forwarder).

  3. netxms, when the fallback source is fixed.

  4. The object’s name.

service.namespace is the custom attribute otel:service.namespace of the object, its own or inherited, or else ServiceNamespace / serviceNamespace; without either it is not sent.

Object Identity

Attribute Description

netxms.object.id

Object ID

netxms.object.guid

Object GUID

netxms.object.class

Object class, for example Node, Sensor, or Rack

host.name, host.id

Object name and GUID; nodes only

hw.vendor, hw.model, hw.serial_number

Hardware vendor, model, and serial number, when known; nodes, sensors, access points, and mobile devices

netxms.sensor.class

Device class of a sensor, for example water_meter or temperature_humidity

Location

These attributes come from the object and its parents: the nearest facility, room, rack, and cooling zone above the object.

Attribute Description

netxms.facility, netxms.facility.id

Facility name and GUID

netxms.room, netxms.room.id

Room name and GUID

netxms.rack, netxms.rack.id

Rack name and GUID

netxms.rack.position, netxms.rack.units

Top unit and height of the object in the rack, for objects mounted in it

netxms.cooling.zone, netxms.cooling.zone.id

Cooling zone name and GUID

netxms.zone.uin, netxms.zone

Zone UIN and name, when zoning is enabled (nodes, interfaces, subnets, access points, clusters, traffic observers, and observation points)

netxms.power.domains, netxms.power.domain_ids, netxms.power.feeds, netxms.power.domain_types

Names, GUIDs, feed tags, and types of the nearest power domains, as arrays in the same order

geo.location.lat, geo.location.lon

Geographic coordinates of the object, or else of its rack, room, or facility

geo.country.iso_code, geo.region.iso_code, geo.locality.name, geo.postal_code

Postal address of the object, or else of its rack, room, or facility

Delivery and Data Loss

Data points and log records are queued in memory and sent in the background, one request per send. A send starts when the queue reaches the batch size or the flush interval has passed.

  • A failed request (an HTTP error status or a connection error) is repeated twice, immediately. If the third attempt fails, the batch is discarded and a warning is written to the server log.

  • When the queue is full, new data is discarded. The queue is kept in memory only: at a normal shutdown the server tries to send it once more, and data not sent then is lost.

  • Data that was not sent is not sent later: there is no backfill after the receiver is reachable again.

  • Data points that the receiver reports as rejected are logged but not sent again.

The metric export queue can be watched with the internal metrics Server.PDS.DriverStat(OTLP,queueSize) and Server.PDS.DriverStat(OTLP,messageDrops) on the management server node. An event forwarder also has its own queue, limited by EventForwarders.MaxQueueSize (see Forward Event).

There are no settings for a custom CA certificate or a client certificate.

Troubleshooting

Set these debug tags (see Debug Console):

  • otlp.metrics — metric export; level 6 logs delivered batches, level 7 every request and the DCIs that were skipped

  • otlp.export — event forwarding

  • pdsdrv — registration of the OTLP driver

  • event.forwarder — the event forwarder queue