Application Monitoring

NetXMS provides several approaches for monitoring custom applications, including external metrics, custom subagents, and process-level monitoring.

Overview

Application monitoring in NetXMS covers:

  • Custom application metrics through external metrics and lists

  • Application process monitoring (running state, resource consumption)

  • Application log monitoring

  • Application-specific health checks via web services or custom scripts

External Metrics

External metrics allow you to extend the agent with custom metrics by executing scripts or commands. Results of the script execution become agent metrics.

Configuration

Define external metrics in the agent configuration file:

ExternalMetric = MyApp.QueueLength:cat /var/myapp/queue_count.txt
ExternalMetric = MyApp.ActiveWorkers:/opt/myapp/bin/count_workers.sh
ExternalMetric = MyApp.Version:rpm -q myapp --qf '%{VERSION}'

On Windows:

ExternalMetric = MyApp.QueueLength:powershell -File C:\MyApp\get_queue.ps1
ExternalMetric = MyApp.ServiceStatus:sc query MyAppService | findstr STATE

The command’s stdout becomes the metric value. The command must complete within the execution timeout (default: 5 seconds; can be changed with ExternalMetricTimeout, in milliseconds). Each external metric also registers a companion <name>.ExitCode metric that returns the exit code of the last command execution.

Commands are executed through the shell by default. To execute a program directly, without shell interpretation, enclose the command line in square brackets:

ExternalMetric = MyApp.Status:[/opt/myapp/bin/status --brief]
ExternalParameter, ExternalParameterShellExec, and ExternalMetricShellExec are deprecated aliases of ExternalMetric.

External Metrics with Arguments

Metrics can accept arguments from DCIs:

ExternalMetric = MyApp.ComponentStatus(*):curl -s http://localhost:8080/health/$1 | jq -r '.status'

DCI metric: MyApp.ComponentStatus(database) would execute curl …​ /health/database.

When using arguments in external metrics, ensure proper input sanitization to prevent command injection.

External Lists

External lists return multi-line output where each line is a list item:

ExternalList = MyApp.Instances:/opt/myapp/list_instances.sh

External Tables

External tables return structured tabular data:

ExternalTable = MyApp.Workers:instanceColumns=NAME,separator=\t:/opt/myapp/worker_status.sh

The middle field is a comma-separated list of named options:

Option Description

instanceColumns

Comma-separated list of instance (key) column names

separator

Column separator (default: comma; use \t for tab)

description

Table description

defaultColumnDataType

Data type assigned to table columns (default: int32)

mergeSeparators

Treat consecutive separators as one (default: false)

backgroundPolling

Collect the table in the background at a fixed interval instead of on request (default: false)

pollingInterval

Background polling interval in seconds (default: 60)

timeout

Command execution timeout in milliseconds for background polling (0 = use the ExternalMetricProviderTimeout agent setting, default 30000)

The script output must use the configured separator and include a header row.

Process Monitoring

Monitor application processes using the agent’s built-in process metrics:

Check if Application is Running

Process.Count(myapp)

Set a threshold to alert when the count drops to 0 (application not running) or exceeds a maximum.

Monitor Resource Usage

Process.CPUTime(myapp)
Process.MemoryUsage(myapp)
Process.Threads(myapp)
Process.Handles(myapp)
Process.MemoryUsage returns the process memory usage as a percentage of physical memory. For absolute values in bytes, use Process.VMSize (virtual memory) or Process.WkSet (working set).

Extended Process Matching

Use Process.CountEx for precise matching when the process name alone is ambiguous:

Process.CountEx(java,.*MyApplication\.jar.*)

This counts Java processes whose command line contains MyApplication.jar. The arguments are: name, command line regex, user regex, window title (Windows only). For full argument details, see Agent Metrics Reference.

Application Log Monitoring

Monitor application log files for errors, warnings, and specific events using the Log Watch subagent. See Log Monitoring for details.

Example log parser for an application:

<parser>
  <file>/var/log/myapp/application.log</file>
  <rules>
    <rule>
      <match>FATAL|CRITICAL</match>
      <event>100300</event>
      <description>Critical application error: %0</description>
    </rule>
    <rule>
      <match>ERROR.*database.*timeout</match>
      <event>100301</event>
      <description>Database timeout: %0</description>
    </rule>
    <rule>
      <match>Deployment completed.*version (\S+)</match>
      <event>100302</event>
      <description>Application deployed: version %1</description>
    </rule>
  </rules>
</parser>

Health Check Endpoints

Modern applications often expose health check endpoints. Use web service monitoring or service monitoring to check these:

  • HTTP health endpoints (/health, /status, /ready)

  • Custom API endpoints returning application metrics

  • Prometheus-format metrics endpoints

Background Task Monitoring

For applications with background tasks, batch jobs, or scheduled processes:

  • Use push DCIs to receive completion status from batch scripts

  • Monitor output files with File.Time.Modify to detect stale processing

  • Check queue depths and processing rates through database queries or external metrics

See Data Collection Concepts for push DCI configuration.

Application Performance Monitoring

While NetXMS is not a full APM solution, it can track key application performance indicators:

  • Response time via web service monitoring

  • Throughput via custom metrics (external metrics or database queries)

  • Error rate via log monitoring

  • Resource utilization via process and OS monitoring

  • Queue depths and processing lag via custom metrics

Combine these metrics with thresholds and dashboards for comprehensive application visibility.