Monitoring Agent Activity and Evaluation

cortx provides two essential utilities to help you track and optimize your agents post-deployment:

• Agent Activity – For tracking real-time agent usage across workflows.
• Agent Evaluation – For analyzing the quality and reliability of agent outputs over time.

• Where to Find These Tools
  • Agent Activity:
    • Left Nav > Agents > Agent Catalog > Agent Activity
  • Agent Evaluation:
    • Left Nav > Agents > Agent Catalog > Select Agent > Performance Tab > Evaluate

Agent Activity

The Agent Activity tab shows a detailed log of each agent’s interactions, including tasks triggered, who triggered them, and their current status.

How to Monitor Agent Activity

• Go to the Agents section from the left nav.
  • Click Agent Catalog.

• Navigate to the Agent Activity Tab.
• Review the table that displays:
  • Agent ID & Name
  • Activity Description
  • Specific Action/Command performed
  • Timestamp (Modified On)
  • Initiator (Modified By)
  • Execution Status (Completed, Pending, Failed)

• Use Cases
  • Ensure agents are actively being used and triggered correctly.
  • Investigate failed or pending tasks.
  • Monitor frequency of use by different users.
  • Audit behaviour for debugging or reporting.

---

Agent Evaluation

Use Agent Evaluation to validate the effectiveness and consistency of an agent over time through dynamic performance metrics.

• How to Evaluate an Agent
  • Go to Agents > Agent Catalog.
  • Click on the Agent you wish to evaluate.

• In the agent detail screen, switch to the Performance tab.
• Expand the Self Evaluation section and click the Evaluate button.

• View the visualized charts for the following metrics:
  1. Relevance Over Time
  2. Completeness Over Time
  3. Approval Rate
  4. Hallucination Rate
  5. Tool Accuracy Rate

How to Use This Data

• Analyze spikes or drops in performance.
• Identify root causes for poor ratings (e.g., hallucinations or low completeness).
• Refine prompts, tool configurations, or logic rules based on trend insights.
• Use results to iterate and improve your agents continuously.

Best Practices

• Monitor Agent Activity weekly for task-level diagnostics.
• Run Evaluations after major updates or monthly to maintain output quality.
• Use the evaluation data to inform training loops, refine agent scope, or disable underperforming agents.