Traffic Observer Integration
NetXMS can connect to an external traffic analyzer and use its data: which hosts a monitored node talks to, per-application traffic, top talkers, and interface throughput as seen by the analyzer. The current connector supports ntopng.
Overview
Two object classes model the analyzer:
- Traffic observer
-
One analyzer instance, for example one ntopng server. Holds the connector name, the connection settings, the zone used for host matching, the removal policy for observation points, and the synchronization settings. Its child objects are its observation points.
- Observation point
-
One traffic source of the analyzer (for ntopng, one monitored interface). Created and updated by discovery, and identified by the analyzer’s own interface ID (external ID).
The server talks to the analyzer through a traffic connector, a server module.
For ntopng this is the NTOPNG connector in the ntopng server module.
When at least one connector initializes successfully, the server registers the TRAFFIC component; only then does the management client offer to create traffic observers.
Host matching links analyzer data to managed nodes.
For every host the analyzer reports on an in-scope, active observation point, the server looks for a node that has the host’s IP address as its primary IP address or on one of its interfaces.
A matched node gets the Traffic view, the Traffic.ObservationPoints internal table, and host-level metrics.
Unmatched hosts do not create nodes.
Requirements
-
ntopng with the REST API v2 reachable from the NetXMS server.
-
An ntopng API token. A token of an administrator user is required for host alias synchronization (see Host Alias Synchronization).
-
The
ntopngmodule loaded by the server. Add it tonetxmsd.confand restart the server:Module = ntopng
Creating a Traffic Observer
-
Right-click Infrastructure Services (or a container or collector under it) and select Create > Traffic observer…
-
Enter the name, select connector
NTOPNG, and fill in the connection form. -
If zoning is enabled, select the zone for host matching. All zones matches hosts in every zone.
-
Click OK.
The connection form is built from the fields the connector declares. For ntopng:
| Field | JSON key | Default | Description |
|---|---|---|---|
URL |
|
— |
Base URL of the ntopng instance, for example |
API token |
|
— |
ntopng REST API token (required) |
Request timeout (seconds) |
|
30 |
HTTP request timeout |
Verify TLS certificate |
|
true |
Verify the ntopng server certificate |
Cache TTL (seconds) |
|
30 |
Lifetime of cached active host lists, host details, and interface counters |
Host list page size |
|
1000 |
Number of hosts requested per page when reading the active host list |
Maximum active hosts per observation point |
|
100000 |
Limit at which reading of the active host list stops; hosts beyond it are not matched |
The server stores these values as a JSON document under the listed keys.
The REST API uses the same document as the credentials property of the object, and GET /v1/traffic-connectors lists the connectors with their field descriptors.
The API token is not returned to clients: the object properties and the REST API omit it.
To keep the stored token when you change other fields, leave the token empty.
If the observer’s connector module is not loaded on the server, the server cannot tell which fields are secret.
It then returns no connection settings (connectorLoaded is false and credentials is null), ignores changes to them, and keeps the stored values.
The Communication property page shows a warning instead of the form; the other settings on the page can still be changed.
Communication Settings
The Communication page of the observer properties also contains:
- Zone for host matching
-
Zone in which observed hosts are matched to nodes, or All zones (zone UIN
-1in NXSL and the REST API). Shown only when zoning is enabled. - Node representing the analyzer itself (optional)
-
Link to the node that runs the analyzer.
- Action when observation point disappears from discovery
-
Mark as inactive (default), Delete after grace period, Delete immediately, or Ignore. See Discovery and Observation Points.
- Grace period (days)
-
For Delete after grace period: the number of days since the point was last discovered before it is deleted. The days count from the last successful discovery, so after a long discovery outage a missing point can be deleted at the next successful discovery. Default: 30.
Discovery and Observation Points
Discovery runs as part of the observer’s configuration poll. To run it immediately, right-click the observer and select Poll > Configuration. Each configuration poll:
-
Tests the connection and reads the backend product, version, edition, and capabilities. The ntopng connector checks which optional endpoints are available to the configured token. Observation point application breakdown, top talkers, and DSCP statistics depend on the endpoints provided by the installed ntopng version.
-
Reads the analyzer’s interfaces and reconciles observation points by external ID. New points are created; for existing points the type, state, sampling rate, and local networks are updated. The name is set only when the point is created, so a renamed point keeps its name.
-
After a successful discovery, applies the removal policy to points that the analyzer no longer reports. If ntopng successfully reports an empty interface list, the policy applies to every existing point.
-
Runs host matching for observation points that are in scope and active.
The status poll (Poll > Status) only tests the connection.
It sets the connection state (Connected, Unreachable, or Authentication failure) and generates SYS_TRAFFIC_OBSERVER_UNREACHABLE or SYS_TRAFFIC_OBSERVER_RECOVERED.
Observation Point Scope
Discovered observation points are out of scope by default. Host matching and host-level data collection use only points in scope; point-level metrics are collected regardless of scope.
To include points:
-
Open the observer and go to the Observation Points view.
-
Select the points, right-click, and select Include in scope.
-
Run Poll > Configuration on the observer, or wait for the next configuration poll. Including a point does not start matching by itself.
The Scope page in the observation point properties sets the same flag.
When zoning is enabled, the page can also override the observer’s zone for this point (Inherit from observer by default, zone UIN -1 in NXSL and the REST API).
Excluding a point from scope deletes its host match records at once, so the matched nodes lose this point’s coverage.
Host Matching
Host matching compares the IP address of each active host with the IP addresses of nodes in the matching zone: the point’s own zone if set, otherwise the observer’s zone. A node matches when the address is its primary IP address or the address of one of its interfaces. Multicast, broadcast, and other non-unicast addresses are skipped.
With All zones, the address is looked up in every zone. If it resolves to different nodes in more than one zone, the host is left unmatched.
Match records are kept in memory and in the database.
A host that is not reported in a matching pass keeps its record until the record ages out: records not refreshed within TrafficObserver.HostRetentionTime days (default 7) are deleted by housekeeping.
A value of 0 disables aging.
Marking a point inactive does not delete its records.
Views
- Observation Points (on a traffic observer)
-
Connection state, backend, capabilities, last discovery, and host alias sync status, followed by the list of observation points with type, state, sampling rate, scope, and last discovery time.
- Traffic (on an observation point)
-
Live throughput, active hosts, active flows, drops, and TCP retransmits; the active host list with matched nodes; application breakdown; top talkers.
- Traffic (on a node)
-
Shown when at least one observation point has a match record for the node. Lists the point and host records for the node; select a record to see its statistics, application breakdown, and top peers.
The views query the analyzer directly and need no DCIs.
Data Collection
DCIs use the Traffic Observer origin.
For this origin, the metric selection dialog in the DCI properties lists the metrics available on the selected object.
Metrics and tables that need a capability the backend does not report are not listed.
Values come from the connector cache, which is refreshed after cacheTtl seconds.
If the analyzer is unreachable, cached values are served for up to twice that time.
- Observer metrics
-
Version,Uptime, and the host alias synchronization metricsSync.HostAliases.LastRun,Sync.HostAliases.RecordsSynced,Sync.HostAliases.Errors. - Observation point metrics
-
ThroughputIn,ThroughputOut(bits per second),PacketsIn,PacketsOut,BytesIn,BytesOut,ActiveHosts,ActiveFlows,Drops,TcpRetransmits. Direction is relative to the monitored network: In is download, Out is upload. Tables:L7Breakdown,TopTalkers,DSCPBreakdown. - Host metrics (on nodes)
-
Host.BytesIn(instance),Host.BytesOut(instance),Host.PacketsIn(instance),Host.PacketsOut(instance),Host.ActiveFlows(instance),Host.TcpRetransmits(instance),Host.Alerts(instance),Host.Present(instance). Tables:Host.L7Breakdown(instance),Host.Peers(instance). The instance is<point object ID>:<host key>; for ntopng the host key is<ip>@<vlan>, for example200:10.5.5.100@0.Host.Presentreturns 1 while the host is in the analyzer’s active host list and 0 otherwise; the other host metrics return an error for a host that is not active.
The internal table Traffic.ObservationPoints (origin Internal) on a node has one row per match record, with columns INSTANCE, OBSERVER, POINT, POINT_ID, HOST_KEY, IP, VLAN, SAMPLING_RATE.
INSTANCE holds the instance value for host metrics.
Example: Per-Host Template
To collect host metrics on every observed node:
-
Create a template, open its Automatic Apply Rules property page, select Apply this template automatically to objects selected by filter, and enter the filter (see Auto-Apply):
return $node->isObserved; -
In the template, create a DCI with origin Traffic Observer and metric
Host.BytesIn({instance}). -
On its Instance Discovery page, select method Internal Table and enter
Traffic.ObservationPoints. -
Repeat for the other host metrics you need.
Each node then gets one DCI instance per observation point and host record. For instance discovery details, see Instance Discovery Methods.
Host Alias Synchronization
NetXMS can send the names of matched nodes to the analyzer as host aliases, so the analyzer UI shows node names.
To enable it, open the observer properties, go to the Synchronization page, select Push names of matched nodes to the analyzer as host aliases, and set the synchronization interval (default 3600 seconds, 60 to 86400).
-
The system scheduled task
Traffic.ConfigSyncruns every minute and synchronizes each observer whose interval has elapsed. Observers that are not connected are skipped. The task exists only when theTRAFFICcomponent is registered. -
To synchronize one observer on your own schedule, create a scheduled task of type
Traffic.ConfigSyncfor that observer. It ignores the interval and the connection state, but synchronization must still be enabled on the observer. The task owner needs Control access to the observer. -
For matched hosts, NetXMS is authoritative: an alias set manually in the analyzer is overwritten by the next run. Aliases are not removed when a match disappears.
-
The observer’s Observation Points view shows the sync status; failures generate
SYS_TRAFFIC_SYNC_FAILED.
The ntopng connector advertises host alias synchronization only when the configured token can access the host alias endpoint. Synchronization is skipped if this capability is absent.
Events
| Event | Generated when |
|---|---|
|
The analyzer API becomes unreachable or authentication fails |
|
The connection is restored after a failure |
|
Discovery changes the state of an existing observation point |
|
A synchronization run fails |
For event parameters, see Event Types.
Troubleshooting
Debug tags: traffic.connector (connector registration and calls), traffic.poll (polls, discovery, host matching), traffic.sync (synchronization runs), ntopng (HTTP requests to ntopng).
See Enabling Debug Logging.
- Discovery finds points but no nodes get a Traffic view
-
New points are out of scope. Include the points in scope and run a configuration poll on the observer.
- Authentication failure right after setup
-
ntopng answers both an invalid token and an unknown API endpoint with a redirect to its login page, which NetXMS reports as an authentication failure. Check the token and that the ntopng version provides the REST API v2. Other redirects are reported as API errors; check the URL scheme and port.
- Host not matched
-
Check that the host’s IP address belongs to a node in the matching zone, that the point is in scope and active, and, with All zones, that the address is not used by nodes in more than one zone.
- Host alias synchronization does not run
-
Check that the observer is connected, that synchronization is enabled, and that the
Traffic.ConfigSyncsystem task exists. If host alias synchronization is missing from the capabilities in the Observation Points view, check that the token belongs to an ntopng administrator and run a configuration poll.
Related Pages
-
TrafficObserver and ObservationPoint NXSL classes