How We Evaluate & Test AI Models in the FlowAI Agent Harness

To make that assessment concrete, we compared five model configurations running the same DNS A record agent through FlowAI. One recorded run produced a usable notification while omitting a required verification step. Here is what we tested, what we observed, and how to apply those findings to model selection.

Start With the Workflow

A DNS A record change is a useful test because the expected actions are concrete. Look up the record. Validate the request. Obtain approval. Create the record. Check the resulting state. Handle the provision-results decision and any required rollback. Report the outcome.

The study evaluated the same agent flow and task across model providers. It focused on tool-call sequence adherence and the quality of the resulting email, rather than execution time. Token usage was captured for all five configurations.

The recorded sequences included a provision-results decision followed by record deletion. This comparison therefore includes the handling of the change after creation. It should not be read as evidence covering every approval, rejection, or failure branch.

What the Recorded Runs Showed

The comparison below separates workflow adherence, notification quality, and captured token usage for the recorded runs.

Source: internal DNS A Record Agent Model Comparison. Results describe the recorded runs. Notification assessments are qualitative. Token usage is not a cost comparison. Execution speed was not evaluated. Nemotron Ultra-3 used 37,260 input tokens and 2,995 output tokens, for a total of 40,255.

The important difference was the second GETARECORDS call. Four runs performed it after CREATEARECORD. The GPT-OSS run moved on without that lookup.

A creation response tells you what the tool returned. A subsequent lookup checks the resulting state. In this workflow, that check was an expected action. Its absence is a process-adherence issue, even when the final email is concise and usable.

This does not establish that GPT-OSS will always omit verification, or that the other models will always include it. It identifies a behavior to investigate and a requirement to test explicitly.

Read the Notification Against the Evidence

The email assessments also differed. Claude Sonnet received the highest presentation score in the internal review. Gemma produced a clear, balanced summary. The Nemotron summaries emphasized traceability, with more detail than the review considered necessary for a standard notification. GPT-OSS was concise, with less formatting and context.

Those differences matter to the operator reading the result. But presentation should have its own assessment. A well-organized email cannot establish that an omitted tool call occurred.

For a broader evaluation, compare the final message with the recorded actions. Does it distinguish creation from verification? Does it reflect the human decision? Does it describe rollback accurately? Does the reported outcome agree with the evidence? That checks factual accuracy as well as readability.

That combination makes model testing useful. You can evaluate how a model handles the actions and decisions your use case requires, then use the findings to refine the agent, strengthen execution controls, or assess another model configuration.

Our DNS comparison shows why that matters. A usable notification accompanied a skipped verification step in one recorded run. Inspecting the tool sequence exposed a difference that the final response alone would not establish. It also identified a concrete requirement to investigate and enforce.

Start with the infrastructure task you want the agent to perform. Define the required actions, approvals, verification, and recovery behavior. Then test the model and agent configuration against those requirements across repeated runs.

FlowAI brings model flexibility, infrastructure integration, and execution visibility into that evaluation. The goal is a model choice backed by evidence and an agent configured around the operational outcomes your team needs.

Bring your infrastructure use case to the Itential team. Let’s define what successful execution requires and put it to the test in FlowAI.