Log Monitoring

NetXMS can monitor text log files on managed nodes using the Log Watch subagent (logwatch.nsm). The subagent reads log files, matches lines against configured patterns, and generates events on the NetXMS server.

For a complete XML element and attribute reference, see Log Parser XML Reference.

Architecture

Log monitoring in NetXMS works through a combination of components:

  1. The Log Watch subagent (logwatch.nsm) on the agent reads log files and applies parser rules

  2. Log parser definitions describe which files to monitor and what patterns to match

  3. When a pattern matches, the agent sends an event to the NetXMS server

  4. The server processes the event through the Event Processing Policy (EPP)

Log parser definitions can be configured directly in the agent configuration file or deployed centrally through agent policies.

Subagent Configuration

Load the Log Watch subagent in the agent configuration file:

SubAgent = logwatch.nsm

Then define log parser entries:

[LogWatch]
Parser = /etc/nxagentd.d/syslog-parser.xml
Parser = /etc/nxagentd.d/app-parser.xml

Log Parser Definition

Log parsers are defined in XML format. A parser definition specifies the log file to monitor, rules for matching lines, and events to generate.

Basic Structure

The <parser> element accepts optional attributes such as name, processAll, and checkInterval. See Parser Element Reference for the full attribute list.

<parser>
  <file>/var/log/syslog</file>
  <rules>
    <rule>
      <match>ERROR.*database connection failed</match>
      <event>100001</event>
      <description>Database connection error detected</description>
    </rule>
    <rule>
      <match>WARNING.*disk space low on (.*)</match>
      <event>100002</event>
      <description>Disk space warning</description>
    </rule>
  </rules>
</parser>

Capture groups from the <match> pattern are passed as parameters of the generated event — in the second rule, the captured file system name can be referenced as %1 in the message template of event 100002. The <description> element is only a rule label shown in trace output; it is never macro-expanded and has no effect on the generated event.

File Element Options

The <file> element supports attributes for controlling encoding, file handle behavior, symlink following, and more. See File Element Reference for the full attribute list.

<file encoding="utf-8" keepOpen="true" followSymlinks="true">/var/log/app.log</file>

Wildcard File Matching

Monitor all files matching a pattern:

<file>/var/log/app/*.log</file>

This monitors all .log files in the directory. New files matching the pattern are automatically detected.

Rule Configuration

Pattern Matching

The <match> element uses PCRE (Perl-Compatible Regular Expressions):

<rule>
  <match>(?i)error|critical|fatal</match>
  <event>100010</event>
  <description>Generic error match</description>
</rule>

Capture groups in the pattern are passed as parameters of the generated event, referenced as %1, %2, etc. in the event template’s message. There is no parameter containing the entire matched line.

Match Attributes

The <match> element supports attributes for case sensitivity, inversion, and repeat detection. See Match Element Reference for the full attribute list.

Inverted Matching

Use the invert attribute to match lines that do NOT match the pattern:

<rule>
  <match invert="true">^OK:</match>
  <event>100011</event>
  <description>Non-OK line</description>
</rule>

Severity Filter

The <severity> element is a match condition, not the severity of the generated event: it restricts the rule to log records whose severity level matches the given bitmask, and applies only to sources that provide level information (syslog, Windows Event Log). The severity of the generated event is defined by the event template on the server. See Severity Element Reference for details.

Rule Processing Order

By default, processing of a line stops at the first matching rule. To evaluate all rules for every line, set the processAll attribute on the parser element:

<parser processAll="true">
  ...
</parser>

When processAll is enabled, an individual rule can still stop further processing of a matched line by setting the break attribute:

<rule break="true">
  <match>session opened for user (.*)</match>
  <event>100020</event>
  <description>User login</description>
</rule>

Multi-line Matching (Context)

Some log messages span multiple lines. Context rules allow matching across line boundaries:

<rules>
  <rule context="exception">
    <match>^Caused by: (.*)</match>
    <event>100030</event>
    <description>Root cause</description>
  </rule>
  <rule>
    <match>Exception in thread "(.*)"</match>
    <context action="set" reset="manual">exception</context>
    <event>100031</event>
    <description>Java exception</description>
  </rule>
</rules>

The context name is given as the element text. The action attribute is set (default) or clear; for set, the reset attribute selects auto (context deactivates automatically after the next line matched in it) or manual (context stays active until cleared by a rule with action="clear"). See Context Element Reference for details.

Macros

Define reusable macros for common regex patterns:

<parser>
  <macros>
    <macro name="IP">(?:\d{1,3}\.){3}\d{1,3}</macro>
    <macro name="TIMESTAMP">\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}</macro>
  </macros>
  <file>/var/log/auth.log</file>
  <rules>
    <rule>
      <match>@{TIMESTAMP} .* Failed password for .* from @{IP}</match>
      <event>100040</event>
      <description>Failed login attempt</description>
    </rule>
  </rules>
</parser>

Macros are referenced with @{MACRO_NAME} syntax (e.g., @{IP}, @{TIMESTAMP}). Use \@ to include a literal @ character in patterns.

Exclusion Schedules

Log parsing can be suspended during specific time periods using exclusion schedules. This is useful to suppress known noise during maintenance windows or batch processing.

<parser>
  <file>/var/log/app.log</file>
  <exclusionSchedules>
    <schedule>* 2-4 * * *</schedule>
  </exclusionSchedules>
  <rules>
    <!-- rules here -->
  </rules>
</parser>

The <schedule> element uses cron format. During matching periods, the parser stops processing new log lines.

Event Parameters

Additional event parameters can be passed to the NetXMS server:

<rule>
  <match>User (?&lt;userName&gt;\S+) logged in from (?&lt;sourceIP&gt;\S+)</match>
  <event>100050</event>
  <description>User login watch</description>
</rule>

Named PCRE capture groups ((?<name>…​)) become named event parameters, referenced as %<userName> in event templates and EPP rules; plain capture groups are passed positionally as %1, %2, and so on, under generated parameter names group_1, group_2, …​ (used for %<name> lookups and EPP parameter matching).

Metric Extraction

Log parser rules can extract values from log lines and expose them as agent metrics or push values:

<rule>
  <match>temperature=(\d+)</match>
  <metrics>
    <metric group="1">sensor.temperature</metric>
  </metrics>
</rule>

See Metric Element Reference for attribute details.

Agent Actions

A rule can trigger an agent action when matched:

<rule>
  <match>critical failure in (.*)</match>
  <agentAction action="restart_service">httpd</agentAction>
</rule>

The action attribute specifies the agent action name to invoke. The element text provides arguments, passed verbatim — capture group references such as %1 are not substituted. The agent expands the arguments in the action’s command line as $1 .. $9; in this example, an action defined as Action = restart_service:service $1 restart runs service httpd restart.

Log Name Filter

The <logName> element is never evaluated when parsing log files — record attribute filters run only during server-side parsing of syslog messages and synchronized Windows events. To match Windows event log (channel) names, use the Log name condition of the server-side Windows event parser.

Deploying Parsers via Policy

Log parser definitions can be deployed centrally using agent policies instead of local configuration. This is the recommended approach for managing log monitoring across many nodes.

  1. In the management client, create a new Log Parser Policy on a template

  2. Paste or edit the log parser XML

  3. Apply the template to target nodes

The policy is automatically deployed to agents and takes effect without agent restart. See Agent Policies and Policy Templates for details.

Log Parser Metrics

The Log Watch subagent provides metrics about monitored log files. See Log Parser Metrics Reference for the full list.

Troubleshooting

Log File Not Being Monitored

  1. Verify logwatch.nsm is loaded: check agent log for loading messages, or query remotely with nxget -I <agent_address> and look for LogWatch parameters

  2. Check that the file path in the parser definition is correct and accessible by the agent

  3. Verify file permissions — the agent process must have read access

  4. Check agent log for parser loading errors

Patterns Not Matching

  1. Test regex patterns against sample log lines using a tool like regex101.com

  2. Check encoding — ensure the parser encoding matches the log file encoding

  3. Verify the parser is actually loaded by checking LogWatch.Parser.ProcessedRecords(<parserName>)

  4. Enable agent debug logging for detailed matching information

High CPU Usage

If log monitoring causes high CPU usage:

  • Reduce the number of rules per parser

  • Use more specific patterns (avoid overly broad regexes like .*)

  • Increase the parser’s file check interval (checkInterval attribute on the <parser> element, in milliseconds, default 10000; the graphical policy editor limits the value to 1000-60000) if real-time matching is not required