This is the documentation for the development version. There are no official binaries for it; build it from the source code in git. Download links on these pages work only after the release. For the current release, see version 7.0.

Traffic Observer Integration

NetXMS can connect to an external traffic analyzer and use its data: which hosts a monitored node talks to, per-application traffic, top talkers, and interface throughput as seen by the analyzer. The current connector supports ntopng.

Overview

Two object classes model the analyzer:

Traffic observer

One analyzer instance, for example one ntopng server. Holds the connector name, the connection settings, the zone used for host matching, the removal policy for observation points, and the synchronization settings. Its child objects are its observation points.

Observation point

One traffic source of the analyzer (for ntopng, one monitored interface). Created and updated by discovery, and identified by the analyzer’s own interface ID (external ID).

The server talks to the analyzer through a traffic connector, a server module. For ntopng this is the NTOPNG connector in the ntopng server module. When at least one connector initializes successfully, the server registers the TRAFFIC component; only then does the management client offer to create traffic observers.

Host matching links analyzer data to managed nodes. For every host the analyzer reports on an in-scope, active observation point, the server looks for a node that has the host’s IP address as its primary IP address or on one of its interfaces. A matched node gets the Traffic view, the Traffic.ObservationPoints internal table, and host-level metrics. Unmatched hosts do not create nodes.

Requirements

  • ntopng with the REST API v2 reachable from the NetXMS server.

  • An ntopng API token. A token of an administrator user is required for host alias synchronization (see Host Alias Synchronization).

  • The ntopng module loaded by the server. Add it to netxmsd.conf and restart the server:

    Module = ntopng

Creating a Traffic Observer

  1. Right-click Infrastructure Services (or a container or collector under it) and select Create > Traffic observer…​

  2. Enter the name, select connector NTOPNG, and fill in the connection form.

  3. If zoning is enabled, select the zone for host matching. All zones matches hosts in every zone.

  4. Click OK.

The connection form is built from the fields the connector declares. For ntopng:

Field JSON key Default Description

URL

url

 — 

Base URL of the ntopng instance, for example https://ntopng.example.com:3000 (required)

API token

token

 — 

ntopng REST API token (required)

Request timeout (seconds)

timeout

30

HTTP request timeout

Verify TLS certificate

verifyTls

true

Verify the ntopng server certificate

Cache TTL (seconds)

cacheTtl

30

Lifetime of cached active host lists, host details, and interface counters

Host list page size

pageSize

1000

Number of hosts requested per page when reading the active host list

Maximum active hosts per observation point

maxHosts

100000

Limit at which reading of the active host list stops; hosts beyond it are not matched

The server stores these values as a JSON document under the listed keys. The REST API uses the same document as the credentials property of the object, and GET /v1/traffic-connectors lists the connectors with their field descriptors. The API token is not returned to clients: the object properties and the REST API omit it. To keep the stored token when you change other fields, leave the token empty.

If the observer’s connector module is not loaded on the server, the server cannot tell which fields are secret. It then returns no connection settings (connectorLoaded is false and credentials is null), ignores changes to them, and keeps the stored values. The Communication property page shows a warning instead of the form; the other settings on the page can still be changed.

Communication Settings

The Communication page of the observer properties also contains:

Zone for host matching

Zone in which observed hosts are matched to nodes, or All zones (zone UIN -1 in NXSL and the REST API). Shown only when zoning is enabled.

Node representing the analyzer itself (optional)

Link to the node that runs the analyzer.

Action when observation point disappears from discovery

Mark as inactive (default), Delete after grace period, Delete immediately, or Ignore. See Discovery and Observation Points.

Grace period (days)

For Delete after grace period: the number of days since the point was last discovered before it is deleted. The days count from the last successful discovery, so after a long discovery outage a missing point can be deleted at the next successful discovery. Default: 30.

Discovery and Observation Points

Discovery runs as part of the observer’s configuration poll. To run it immediately, right-click the observer and select Poll > Configuration. Each configuration poll:

  1. Tests the connection and reads the backend product, version, edition, and capabilities. The ntopng connector checks which optional endpoints are available to the configured token. Observation point application breakdown, top talkers, and DSCP statistics depend on the endpoints provided by the installed ntopng version.

  2. Reads the analyzer’s interfaces and reconciles observation points by external ID. New points are created; for existing points the type, state, sampling rate, and local networks are updated. The name is set only when the point is created, so a renamed point keeps its name.

  3. After a successful discovery, applies the removal policy to points that the analyzer no longer reports. If ntopng successfully reports an empty interface list, the policy applies to every existing point.

  4. Runs host matching for observation points that are in scope and active.

The status poll (Poll > Status) only tests the connection. It sets the connection state (Connected, Unreachable, or Authentication failure) and generates SYS_TRAFFIC_OBSERVER_UNREACHABLE or SYS_TRAFFIC_OBSERVER_RECOVERED.

Observation Point Scope

Discovered observation points are out of scope by default. Host matching and host-level data collection use only points in scope; point-level metrics are collected regardless of scope.

To include points:

  1. Open the observer and go to the Observation Points view.

  2. Select the points, right-click, and select Include in scope.

  3. Run Poll > Configuration on the observer, or wait for the next configuration poll. Including a point does not start matching by itself.

The Scope page in the observation point properties sets the same flag. When zoning is enabled, the page can also override the observer’s zone for this point (Inherit from observer by default, zone UIN -1 in NXSL and the REST API).

Excluding a point from scope deletes its host match records at once, so the matched nodes lose this point’s coverage.

Host Matching

Host matching compares the IP address of each active host with the IP addresses of nodes in the matching zone: the point’s own zone if set, otherwise the observer’s zone. A node matches when the address is its primary IP address or the address of one of its interfaces. Multicast, broadcast, and other non-unicast addresses are skipped.

With All zones, the address is looked up in every zone. If it resolves to different nodes in more than one zone, the host is left unmatched.

Match records are kept in memory and in the database. A host that is not reported in a matching pass keeps its record until the record ages out: records not refreshed within TrafficObserver.HostRetentionTime days (default 7) are deleted by housekeeping. A value of 0 disables aging. Marking a point inactive does not delete its records.

Views

Observation Points (on a traffic observer)

Connection state, backend, capabilities, last discovery, and host alias sync status, followed by the list of observation points with type, state, sampling rate, scope, and last discovery time.

Traffic (on an observation point)

Live throughput, active hosts, active flows, drops, and TCP retransmits; the active host list with matched nodes; application breakdown; top talkers.

Traffic (on a node)

Shown when at least one observation point has a match record for the node. Lists the point and host records for the node; select a record to see its statistics, application breakdown, and top peers.

The views query the analyzer directly and need no DCIs.

Data Collection

DCIs use the Traffic Observer origin. For this origin, the metric selection dialog in the DCI properties lists the metrics available on the selected object. Metrics and tables that need a capability the backend does not report are not listed. Values come from the connector cache, which is refreshed after cacheTtl seconds. If the analyzer is unreachable, cached values are served for up to twice that time.

Observer metrics

Version, Uptime, and the host alias synchronization metrics Sync.HostAliases.LastRun, Sync.HostAliases.RecordsSynced, Sync.HostAliases.Errors.

Observation point metrics

ThroughputIn, ThroughputOut (bits per second), PacketsIn, PacketsOut, BytesIn, BytesOut, ActiveHosts, ActiveFlows, Drops, TcpRetransmits. Direction is relative to the monitored network: In is download, Out is upload. Tables: L7Breakdown, TopTalkers, DSCPBreakdown.

Host metrics (on nodes)

Host.BytesIn(instance), Host.BytesOut(instance), Host.PacketsIn(instance), Host.PacketsOut(instance), Host.ActiveFlows(instance), Host.TcpRetransmits(instance), Host.Alerts(instance), Host.Present(instance). Tables: Host.L7Breakdown(instance), Host.Peers(instance). The instance is <point object ID>:<host key>; for ntopng the host key is <ip>@<vlan>, for example 200:10.5.5.100@0. Host.Present returns 1 while the host is in the analyzer’s active host list and 0 otherwise; the other host metrics return an error for a host that is not active.

The internal table Traffic.ObservationPoints (origin Internal) on a node has one row per match record, with columns INSTANCE, OBSERVER, POINT, POINT_ID, HOST_KEY, IP, VLAN, SAMPLING_RATE. INSTANCE holds the instance value for host metrics.

Example: Per-Host Template

To collect host metrics on every observed node:

  1. Create a template, open its Automatic Apply Rules property page, select Apply this template automatically to objects selected by filter, and enter the filter (see Auto-Apply):

    return $node->isObserved;
  2. In the template, create a DCI with origin Traffic Observer and metric Host.BytesIn({instance}).

  3. On its Instance Discovery page, select method Internal Table and enter Traffic.ObservationPoints.

  4. Repeat for the other host metrics you need.

Each node then gets one DCI instance per observation point and host record. For instance discovery details, see Instance Discovery Methods.

Host Alias Synchronization

NetXMS can send the names of matched nodes to the analyzer as host aliases, so the analyzer UI shows node names.

To enable it, open the observer properties, go to the Synchronization page, select Push names of matched nodes to the analyzer as host aliases, and set the synchronization interval (default 3600 seconds, 60 to 86400).

  • The system scheduled task Traffic.ConfigSync runs every minute and synchronizes each observer whose interval has elapsed. Observers that are not connected are skipped. The task exists only when the TRAFFIC component is registered.

  • To synchronize one observer on your own schedule, create a scheduled task of type Traffic.ConfigSync for that observer. It ignores the interval and the connection state, but synchronization must still be enabled on the observer. The task owner needs Control access to the observer.

  • For matched hosts, NetXMS is authoritative: an alias set manually in the analyzer is overwritten by the next run. Aliases are not removed when a match disappears.

  • The observer’s Observation Points view shows the sync status; failures generate SYS_TRAFFIC_SYNC_FAILED.

The ntopng connector advertises host alias synchronization only when the configured token can access the host alias endpoint. Synchronization is skipped if this capability is absent.

Events

Event Generated when

SYS_TRAFFIC_OBSERVER_UNREACHABLE

The analyzer API becomes unreachable or authentication fails

SYS_TRAFFIC_OBSERVER_RECOVERED

The connection is restored after a failure

SYS_OBSERVATION_POINT_STATE_CHANGED

Discovery changes the state of an existing observation point

SYS_TRAFFIC_SYNC_FAILED

A synchronization run fails

For event parameters, see Event Types.

Troubleshooting

Debug tags: traffic.connector (connector registration and calls), traffic.poll (polls, discovery, host matching), traffic.sync (synchronization runs), ntopng (HTTP requests to ntopng). See Enabling Debug Logging.

Discovery finds points but no nodes get a Traffic view

New points are out of scope. Include the points in scope and run a configuration poll on the observer.

Authentication failure right after setup

ntopng answers both an invalid token and an unknown API endpoint with a redirect to its login page, which NetXMS reports as an authentication failure. Check the token and that the ntopng version provides the REST API v2. Other redirects are reported as API errors; check the URL scheme and port.

Host not matched

Check that the host’s IP address belongs to a node in the matching zone, that the point is in scope and active, and, with All zones, that the address is not used by nodes in more than one zone.

Host alias synchronization does not run

Check that the observer is connected, that synchronization is enabled, and that the Traffic.ConfigSync system task exists. If host alias synchronization is missing from the capabilities in the Observation Points view, check that the token belongs to an ntopng administrator and run a configuration poll.