AI Operators
An AI operator is a persistent AI assistant instance that watches part of the monitored environment on its own. It runs in cycles: each cycle is a non-interactive AI assistant chat that reviews the current state with the AI assistant’s tools, records notable findings as observations, and decides when to run next. Between cycles, standing checks (NXSL scripts run by the server without the LLM) watch for conditions that need attention and can wake the operator early.
AI operators are managed in the Configuration perspective under AI Operators.
Managing them requires the Manage AI operators system access right.
The whole subsystem can be switched off with the AIOperator.Enabled server configuration variable.
While it is off, no new cycles or standing check runs start; work already running is not cancelled.
How an Operator Works
Each operator runs with the access rights of its owner account.
By default this is the built-in ai-operator account.
The owner must exist, be enabled, and have the Use AI assistant system access right; this is checked when the owner is set and before every cycle.
The owner’s rights are the operator’s only access limit: the operator can see and change only what its owner can.
Cycles
The server checks every 30 seconds which operators are due. At the end of each cycle the model chooses the delay to the next cycle, within the operator’s minimum and maximum interval. A cycle of the same operator never overlaps with another one.
Every cycle starts a new chat; the previous conversation is not reused. Continuity comes from the state the operator keeps between cycles:
- Persona instructions
-
Written by an administrator in the operator properties: what the operator is for and how it should work.
- Standing instructions
-
Written by the operator itself (see Standing Instructions).
- Current focus, watch list, and memento
-
Short working notes that the model updates at the end of a cycle and receives at the start of the next one.
Each cycle also receives the cycle number, the interval limits, the scope text, a summary of the standing checks, and the results of the checks that woke it.
Token Budget
A non-zero daily token budget limits the input and output tokens an operator uses per day (UTC). The budget is checked before each cycle and the usage is added after the cycle, so the last cycle of the day can go over the budget. When the budget is used up, cycles are skipped until the next UTC day; standing checks keep running, because they do not use the LLM.
Failures
A cycle fails when the owner is not valid, the AI assistant gives no response, or the response cannot be parsed.
The next attempt is made after the minimum interval.
After AIOperator.MaxConsecutiveFailures consecutive failures (default 3) the operator is disabled and the SYS_AI_OPERATOR_FAILURE event is generated.
A successful cycle resets the count; enabling a disabled operator resets it too and makes its next cycle due at once.
Creating an Operator
-
In the Configuration perspective, open AI Operators and click New…
-
On the General page, set the properties below.
-
Click OK.
By default a new operator is enabled, and its first cycle is due at the next scheduler pass.
| Property | Description |
|---|---|
Name |
Operator name (required) |
Model slot |
AI model slot to use; empty uses the default slot (see AI Reference) |
Description |
Free text |
Owner |
Account the operator runs as; empty uses the default |
Scope filter |
Object names or IDs the operator should concentrate on; empty means all objects the owner can access. This text is given to the model as guidance and does not limit access. |
Persona instructions |
What the operator is for and how it should work |
Minimum interval, Maximum interval |
Range for the delay between cycles (60 seconds to 7 days; default 5 minutes and 1 hour) |
Daily token budget |
Maximum tokens per UTC day; 0 means unlimited |
Observation retention |
Days to keep this operator’s observations; 0 uses |
Observation record limit |
Maximum number of observations kept for this operator; 0 uses |
Enabled |
Whether the operator runs |
The operator list shows the state, owner, token usage today, cycle number, last and next execution, current focus, and the explanation of the last cycle. The view also offers Edit…, Delete, Enable, Disable, and Reset accumulated state. Deleting an operator deletes its observations, standing checks, and standing instruction history; its execution log records stay until they expire.
Standing Instructions
Standing instructions are notes the operator writes for itself: what it has learned about the environment and how it wants to work in later cycles. They are added to the prompt after the persona instructions, which take precedence. The model can replace or clear them at the end of any cycle.
The Standing Instructions page of the operator properties shows the current text and lets an administrator edit it:
-
Locked makes the operator ignore its own updates; an administrator can still edit the text.
-
The history lists previous versions, most recent first, with the time and cycle of each change. View… shows a version; Restore copies it into the editor, and it becomes current when you apply the change. The server keeps the last
AIOperator.Instructions.HistoryDepthversions (default 20; 0 keeps all). -
Text longer than
AIOperator.Instructions.MaxSizebytes (default 8192; 0 for no limit) is truncated, whether the operator or an administrator wrote it.
Reset accumulated state in the operator list clears the standing instructions (also when locked), the current focus, watch list, memento, and cycle counter. The cleared instructions go to the history. Standing checks, the lock flag, observations, and the configuration stay.
Standing Checks
A standing check is an NXSL script that the server runs at a fixed interval, without the LLM. When the script reports a problem, the check fires and either wakes the operator or records an observation. Checks let an operator watch many conditions cheaply and spend LLM cycles only when something happens.
The operator creates, changes, and deletes its own checks during cycles; administrators manage them on the Standing Checks page of the operator properties. The page lists each check with its action, interval, bound object, last verdict (Quiet, Fired, Error, or Never run), last fire time, and error count, and whether it was created by the operator or a user. Changes on this page take effect at once.
Check Properties
| Property | Description |
|---|---|
Name |
Check name (required) |
Description |
What the check watches and why |
Object bound as |
Optional object the script runs against; the owner must have read access to it |
Action when fired |
Wake operator (default) or Record observation |
Run interval |
Time between runs (30 seconds to 7 days; default 5 minutes) |
Cooldown |
Minimum time between two actions of this check; 0 means none |
Renotify interval |
While the check stays fired, repeat the action at this interval; 0 means act only when the check starts firing |
Enabled |
Whether the check runs |
Locked |
The operator cannot change or delete the check; administrators still can |
Script |
NXSL script (required); it must compile before the check is saved |
For an existing check, the dialog also shows its run count, last run and fire times, consecutive errors, last result, and compilation error.
The number of checks per operator, enabled or not, is limited by AIOperator.Checks.MaxPerInstance (default 32; 0 for no limit).
Check Script
The script runs with the owner’s access rights, in a fresh VM each time.
A check bound to an object gets the usual variables for that object ($object, $node for a node); an unbound check gets $object set to null.
| Return value | Result |
|---|---|
|
Quiet |
A non-empty string |
Fired, with the string as title and severity Warning |
A hash map |
Fired: |
Anything else, including |
Error |
For example, a check bound to a node that fires while the node’s CPU usage is high:
cpu = $node.findDCIByName("System.CPU.Usage")?.cachedValue;
if (cpu == null or cpu < 90)
return null;
return { "title": "High CPU usage on " .. $node.name, "severity": "major", "details": "CPU usage is " .. cpu .. "%" };
A script that does not compile, fails at run time, returns an invalid value, or runs longer than AIOperator.Checks.ExecutionTimeLimit seconds (default 10; 0 for no limit) is an error.
After AIOperator.Checks.MaxConsecutiveErrors consecutive errors (default 5; 0 never disables) the check is disabled and a Warning observation is recorded.
A successful run, a script change, or enabling the check again resets the error count.
For the full script context, see AI Operator Standing Check Script.
When a Check Acts
A check acts when it starts firing, not on every run that fires:
-
After a quiet or error result, a fired result performs the action, unless the cooldown since the last action has not passed; the action then waits for a later run.
-
While the check keeps firing, the action is repeated only if a renotify interval is set, and then only after both the renotify interval and the cooldown have passed.
-
A quiet result ends the firing state; it does not change observations already recorded.
The actions:
- Record observation
-
Records an observation with the returned title, severity, and details, linked to the bound object and the check. The LLM is not involved.
- Wake operator
-
Queues the check result for the operator, which then runs an extra cycle that receives it. The extra cycle starts no sooner than
AIOperator.InterruptCooldownseconds (default 60) after the start of the previous cycle, regardless of the minimum interval. Results from several checks are delivered in one cycle. A queued result is dropped if that cycle cannot run (budget used up or owner not valid).
Standing checks run only while their operator is enabled.
Observations
An observation is a finding the operator records: an anomaly, degradation, recovery, or trend, with a severity, title, details, an optional object, and references to supporting data.
Observations come from the operator’s cycles, from standing checks with the Record observation action, and from checks disabled after errors.
An observation is not an alarm; recording one generates the SYS_AI_OPERATOR_OBSERVATION event (on the related object, or on the management server when there is none), which EPP rules can turn into alarms or notifications.
The observation severity is a parameter of that event, not the event severity.
Each observation has a review state: New, Acknowledged, or Dismissed. Observations are shown in the Observations view of the AI Assistant perspective and in the AI Operator Observations log of the Logs perspective; right-click an observation to Acknowledge, Dismiss, or Mark as new. Housekeeping deletes observations older than the retention time or beyond the record limit of their operator, whatever their state.
Access to Observations
Reading observations requires the View event log system access right. It does not require Manage AI operators or ownership of the operator.
In the REST API (GET /v1/ai/observations), an observation linked to an object is returned only if the object still exists and the caller has read access to it; such observations are otherwise left out of the result.
Observations without an object are returned to every caller with the View event log right.
When the request names an object with the object parameter, a missing object gives status 404 and an object without read access gives 403.
In the management client logs, access to observations follows the general log access rules: they are filtered by object access only when Server.Security.ExtendedLogQueryAccessControl is enabled.
Changing the state of an observation requires the Update alarms access right on its object, or Manage AI operators for an observation without an object.
Execution Log
The AI Operator Executions log (Logs perspective, View event log right) has one record per cycle, per check action, and per failed check run, with the operator, status, cycle number, duration, token usage, and explanation. The status is one of:
| Status | Meaning |
|---|---|
Completed |
The cycle finished |
Failed |
The cycle failed |
Skipped |
The cycle did not run because the daily token budget is used up |
Check fired |
A standing check performed its action |
Check error |
A standing check run failed; the explanation also says when the check was disabled after consecutive errors |
Quiet check runs are not logged.
Records are kept for AIOperatorExecutionLog.RetentionTime days (default 90).
REST API
All endpoints are under /v1/ai/.
Operator and check endpoints require the Manage AI operators system access right.
| Endpoint | Purpose |
|---|---|
|
List or create operators |
|
Read, change, or delete an operator ( |
|
List or create standing checks |
|
Read, change, or delete a standing check |
|
Standing instruction history |
|
Reset accumulated state |
|
List observations; filters |
|
Change the review state: |
NXSL
GetAIOperator() and GetAIOperators() return AIOperator objects with the read-only state of operators; they require the Manage AI operators system access right.
Troubleshooting
- The operator does not run
-
Check that
AIOperator.Enabledis on and the operator is enabled. If it was disabled after failures, see the Failed records in the execution log, fix the cause (often the owner account or the model slot), and enable it again. - Cycles are skipped
-
The daily token budget is used up; cycles resume on the next UTC day.
- A standing check is disabled
-
It reached
AIOperator.Checks.MaxConsecutiveErrors; open it on the Standing Checks page to see the last error, fix the script, and enable it.
For detailed messages, set the ai.operator debug tag to level 5 or higher (see Enabling Debug Logging).