AI Operators

An AI operator is a persistent AI assistant instance that watches part of the monitored environment on its own. It runs in iterations: each iteration is a non-interactive AI assistant chat that reviews the current state with the AI assistant’s tools, records notable findings as observations, and decides when to run next. Between iterations, standing checks (NXSL scripts run by the server without the LLM) watch for conditions that need attention and can wake the operator early.

AI operators are managed in the Configuration perspective under AI Operators. Managing them requires the Manage AI operators system access right. The whole subsystem can be switched off with the AIOperator.Enabled server configuration variable. While it is off, no new iterations or standing check runs start; work already running is not cancelled.

How an Operator Works

Each operator runs with the access rights of its owner account. By default this is the built-in ai-operator account. The owner must exist, be enabled, and have the Use AI assistant system access right; this is checked when the owner is set and before every iteration. The owner’s rights are the operator’s only access limit: the operator can see and change only what its owner can.

Iterations

The server checks every 30 seconds which operators are due. At the end of each iteration the model chooses the delay to the next iteration, within the operator’s minimum and maximum interval. A iteration of the same operator never overlaps with another one.

Every iteration starts a new chat; the previous conversation is not reused. Continuity comes from the state the operator keeps between iterations:

Persona instructions

Written by an administrator in the operator properties: what the operator is for and how it should work.

Standing instructions

Written by the operator itself (see Standing Instructions).

Current focus, watch list, and memento

Short working notes that the model updates at the end of a iteration and receives at the start of the next one.

Each iteration also receives the iteration number, the interval limits, the scope text, a summary of the standing checks, and the results of the checks that woke it.

Token Budget

A non-zero daily token budget limits the input and output tokens an operator uses per day (UTC). The budget is checked before each iteration and the usage is added after the iteration, so the last iteration of the day can go over the budget. When the budget is used up, iterations are skipped until the next UTC day; standing checks keep running, because they do not use the LLM.

Failures

A iteration fails when the owner is not valid, the AI assistant gives no response, or the response cannot be parsed. The next attempt is made after the minimum interval. After AIOperator.MaxConsecutiveFailures consecutive failures (default 3) the operator is disabled and the SYS_AI_OPERATOR_FAILURE event is generated. A successful iteration resets the count; enabling a disabled operator resets it too and makes its next iteration due at once.

Creating an Operator

  1. In the Configuration perspective, open AI Operators and click New…​

  2. On the General page, set the properties below.

  3. Click OK.

By default a new operator is enabled, and its first iteration is due at the next scheduler pass.

Property Description

Name

Operator name (required)

Model slot

AI model slot to use; empty uses the default slot (see AI Reference)

Description

Free text

Owner

Account the operator runs as; empty uses the default ai-operator account. You can select your own account or the default account; selecting another account requires the Manage users system access right.

Scope filter

Object names or IDs the operator should concentrate on; empty means all objects the owner can access. This text is given to the model as guidance and does not limit access.

Persona instructions

What the operator is for and how it should work

Minimum interval, Maximum interval

Range for the delay between iterations (60 seconds to 7 days; default 5 minutes and 1 hour)

Daily token budget

Maximum tokens per UTC day; 0 means unlimited

Observation retention

Days to keep this operator’s observations; 0 uses AIOperatorObservations.RetentionTime

Observation record limit

Maximum number of observations kept for this operator; 0 uses AIOperatorObservations.MaxRecordsPerInstance

Enabled

Whether the operator runs

The operator list shows the state, owner, token usage today, iteration number, last and next execution, current focus, and the explanation of the last iteration. The view also offers Edit…​, Delete, Enable, Disable, and Reset accumulated state. Deleting an operator deletes its observations, standing checks, and standing instruction history; its execution log records stay until they expire.

Standing Instructions

Standing instructions are notes the operator writes for itself: what it has learned about the environment and how it wants to work in later iterations. They are added to the prompt after the persona instructions, which take precedence. The model can replace or clear them at the end of any iteration.

The Standing Instructions page of the operator properties shows the current text and lets an administrator edit it:

  • Locked makes the operator ignore its own updates; an administrator can still edit the text.

  • The history lists previous versions, most recent first, with the time and iteration of each change. View…​ shows a version; Restore copies it into the editor, and it becomes current when you apply the change. The server keeps the last AIOperator.Instructions.HistoryDepth versions (default 20; 0 keeps all).

  • Text longer than AIOperator.Instructions.MaxSize bytes (default 8192; 0 for no limit) is truncated, whether the operator or an administrator wrote it.

Reset accumulated state in the operator list clears the standing instructions (also when locked), the current focus, watch list, memento, and iteration counter. The cleared instructions go to the history. Standing checks, the lock flag, observations, and the configuration stay.

Standing Checks

A standing check is an NXSL script that the server runs at a fixed interval, without the LLM. When the script reports a problem, the check fires and either wakes the operator or records an observation. Checks let an operator watch many conditions cheaply and spend LLM iterations only when something happens.

The operator creates, changes, and deletes its own checks during iterations; administrators manage them on the Standing Checks page of the operator properties. The page lists each check with its action, interval, bound object, last verdict (Quiet, Fired, Error, or Never run), last fire time, and error count, and whether it was created by the operator or a user. Changes on this page take effect at once.

Check Properties

Property Description

Name

Check name (required)

Description

What the check watches and why

Object bound as $object

Optional object the script runs against; the owner must have read access to it

Action when fired

Wake operator (default) or Record observation

Run interval

Time between runs (30 seconds to 7 days; default 5 minutes)

Cooldown

Minimum time between two actions of this check; 0 means none

Renotify interval

While the check stays fired, repeat the action at this interval; 0 means act only when the check starts firing

Enabled

Whether the check runs

Locked

The operator cannot change or delete the check; administrators still can

Script

NXSL script (required); it must compile before the check is saved

For an existing check, the dialog also shows its run count, last run and fire times, consecutive errors, last result, and compilation error.

The number of checks per operator, enabled or not, is limited by AIOperator.Checks.MaxPerInstance (default 32; 0 for no limit).

Check Script

The script runs with the owner’s access rights, in a fresh VM each time. A check bound to an object gets the usual variables for that object ($object, $node for a node); an unbound check gets $object set to null.

Return value Result

null, false, 0, or an empty string

Quiet

A non-empty string

Fired, with the string as title and severity Warning

A hash map

Fired: title (default: the check name), severity (normal, warning, minor, major, critical, or 0 to 4; default warning), and optional details

Anything else, including true or a non-zero number

Error

For example, a check bound to a node that fires while the node’s CPU usage is high:

cpu = $node.findDCIByName("System.CPU.Usage")?.cachedValue;
if (cpu == null or cpu < 90)
   return null;
return { "title": "High CPU usage on " .. $node.name, "severity": "major", "details": "CPU usage is " .. cpu .. "%" };

A script that does not compile, fails at run time, returns an invalid value, or runs longer than AIOperator.Checks.ExecutionTimeLimit seconds (default 10; 0 for no limit) is an error. After AIOperator.Checks.MaxConsecutiveErrors consecutive errors (default 5; 0 never disables) the check is disabled and a Warning observation is recorded. A successful run, a script change, or enabling the check again resets the error count. For the full script context, see AI Operator Standing Check Script.

When a Check Acts

A check acts when it starts firing, not on every run that fires:

  • After a quiet or error result, a fired result performs the action, unless the cooldown since the last action has not passed; the action then waits for a later run.

  • While the check keeps firing, the action is repeated only if a renotify interval is set, and then only after both the renotify interval and the cooldown have passed.

  • A quiet result ends the firing state; it does not change observations already recorded.

The actions:

Record observation

Records an observation with the returned title, severity, and details, linked to the bound object and the check. The LLM is not involved.

Wake operator

Queues the check result for the operator, which then runs an extra iteration that receives it. The extra iteration starts no sooner than AIOperator.InterruptCooldown seconds (default 60) after the start of the previous iteration, regardless of the minimum interval. Results from several checks are delivered in one iteration. A queued result is dropped if that iteration cannot run (budget used up or owner not valid).

Standing checks run only while their operator is enabled.

Observations

An observation is a finding the operator records: an anomaly, degradation, recovery, or trend, with a severity, title, details, an optional object, and references to supporting data. Observations come from the operator’s iterations, from standing checks with the Record observation action, and from checks disabled after errors. An observation is not an alarm; recording one generates the SYS_AI_OPERATOR_OBSERVATION event (on the related object, or on the management server when there is none), which EPP rules can turn into alarms or notifications. The observation severity is a parameter of that event, not the event severity.

Each observation has a review state: New, Acknowledged, or Dismissed. Observations are shown in the Observations view of the AI Assistant perspective and in the AI Operator Observations log of the Logs perspective; right-click an observation to Acknowledge, Dismiss, or Mark as new. Housekeeping deletes observations older than the retention time or beyond the record limit of their operator, whatever their state.

Access to Observations

Reading observations requires the View event log system access right. It does not require Manage AI operators or ownership of the operator.

In the REST API (GET /v1/ai/observations), an observation linked to an object is returned only if the object still exists and the caller has read access to it; such observations are otherwise left out of the result. Observations without an object are returned to every caller with the View event log right. When the request names an object with the object parameter, a missing object gives status 404 and an object without read access gives 403. In the management client logs, access to observations follows the general log access rules: they are filtered by object access only when Server.Security.ExtendedLogQueryAccessControl is enabled.

Changing the state of an observation requires the Update alarms access right on its object, or Manage AI operators for an observation without an object.

Execution Log

The AI Operator Executions log (Logs perspective, View event log right) has one record per iteration, per check action, and per failed check run, with the operator, status, iteration number, duration, token usage, and explanation. The status is one of:

Status Meaning

Completed

The iteration finished

Failed

The iteration failed

Skipped

The iteration did not run because the daily token budget is used up

Check fired

A standing check performed its action

Check error

A standing check run failed; the explanation also says when the check was disabled after consecutive errors

Quiet check runs are not logged. Records are kept for AIOperatorExecutionLog.RetentionTime days (default 90).

REST API

All endpoints are under /v1/ai/. Operator and check endpoints require the Manage AI operators system access right.

Endpoint Purpose

GET, POST operators

List or create operators

GET, PATCH, DELETE operators/{id}

Read, change, or delete an operator (PATCH enabled to enable or disable)

GET, POST operators/{id}/checks

List or create standing checks

GET, PATCH, DELETE operators/{id}/checks/{check-id}

Read, change, or delete a standing check

GET operators/{id}/instructions-history

Standing instruction history

POST operators/{id}/reset-memento

Reset accumulated state

GET observations

List observations; filters instance, object, state (new, acknowledged, dismissed), since, limit (default 100, 0 for all)

PUT observations/{observation-id}/state

Change the review state: {"state": "acknowledged"}

NXSL

GetAIOperator() and GetAIOperators() return AIOperator objects with the read-only state of operators; they require the Manage AI operators system access right.

Troubleshooting

The operator does not run

Check that AIOperator.Enabled is on and the operator is enabled. If it was disabled after failures, see the Failed records in the execution log, fix the cause (often the owner account or the model slot), and enable it again.

Iterations are skipped

The daily token budget is used up; iterations resume on the next UTC day.

A standing check is disabled

It reached AIOperator.Checks.MaxConsecutiveErrors; open it on the Standing Checks page to see the last error, fix the script, and enable it.

For detailed messages, set the ai.operator debug tag to level 5 or higher (see Enabling Debug Logging).