The Guide to Evaluating AI Models Within the FlowAI Agent Harness

The Model Selection Question

Choose a Model for the Work It Must Perform

Infrastructure teams do not select a model in isolation. They select a model configuration for a task performed through a specific set of tools, instructions, permissions, and infrastructure integrations. A configuration that works well for incident investigation may require different evidence before it is authorized to provision a service or perform rollback.

The useful question is therefore specific: can this model configuration perform this task, through this harness, while meeting the operational requirements? That question produces a testable contract. “Best model” without a task, environment, and acceptance criteria does not.

Distinguish the Model From the Harness

Changing the model within a shared harness is a practical comparison for a team deciding what to deploy. It is not a measurement of isolated model capability. Provider behavior, parameter support, tool interfaces, and context handling can influence the result. Record those differences rather than treating them as invisible implementation details.

Keep the Comparison Aligned With the Decision

Two experiments answer different questions. A shared-configuration experiment asks how models behave under the same task instructions and tool interfaces. A deployment-optimized experiment asks how well each model performs after documented configuration improvements. Both are useful. Report them separately so model-specific tuning is not mistaken for a controlled comparison.

The DNS comparison uses the same agent flow and task across providers. Its recorded configurations provide a starting point for investigation and retesting. The protocol later in this guide explains how to document settings, control initial state, and establish a reproducible comparison.

A model comparison within FlowAI and a comparison between harnesses answer different questions. Comparing harnesses requires equivalent task contracts, tools, control boundaries, and measurement methods.

Inside the FlowAI Harness

Connect Model Choice to Infrastructure Execution

FlowAI is the agent harness within the Itential Platform. Its architecture brings together tools, skills, context, and governance. Those components give model reasoning an operational setting: capabilities to invoke, task guidance to follow, state to inspect, and boundaries on permitted actions.

Model Flexibility With an Operational Foundation

FlowAI supports compatible models through supported provider integrations. The Model Registry configures provider connections and enabled models; profiles are environment-specific. The registry documentation also describes model-specific variables for on-prem profiles. These capabilities make provider connection and parameter choices part of the reproducibility record.

For evaluation, this separation matters. A team can investigate a different model choice while retaining the operational task and existing automation capabilities. The comparison remains meaningful only when the versions and behavior of those capabilities are documented and controlled.

Tools & Skills Serve Different Purposes

A tool provides a callable capability. A skill provides guidance about how to approach the task. A DNS lookup tool might return record state; task guidance might explain when to perform that lookup and what evidence to inspect before continuing. The tool interface and the guidance both affect what the model sees and how it selects its next action.

This distinction also improves diagnosis. A missing verification call may require clearer instructions, a more explicit tool description, a revised workflow design, or a different model configuration. An execution trace identifies the omission; it does not, by itself, identify the cause.

Governance & Task Completion Need Separate Checks

FlowAI operates within the platform’s governance and execution architecture. Authorized access to a tool and completion of every required task step are different properties. A model may use only authorized tools and still omit a required lookup. The DNS example makes this distinction concrete.

When a step is mandatory, define its enforcement in the operational design. For example, a workflow can require verification evidence before reporting an accepted outcome. Evaluate that control alongside the model’s tool-selection behavior.

A skill or prompt can explain the required process. Authorization and approval controls determine which actions are permitted. Acceptance checks determine whether the operational result meets the task contract. Each responsibility should have identifiable evidence.

FlowAI brings model flexibility, infrastructure integration, reusable automation, and execution visibility into the same operational environment. Teams can compare model choices against existing capabilities, inspect the resulting actions, and use the findings to improve the configuration.

Operational Acceptance

Define Success Before Comparing Models

An evaluation needs to distinguish performing an action from establishing its outcome. A successful tool response can show that a request was accepted or processed. The resulting infrastructure state may require a separate observation. An email that describes success is another representation of the run, not a substitute for either form of evidence.

Inspect the Arguments & Outcomes Behind Every Call

The correct tool name can still be called with the wrong zone, record, or parameters. A lookup can return an error or data that does not support the requested conclusion. Inspect the target, arguments, response, and interpretation of each required call rather than accepting the presence of a tool name as proof of correctness.

For a DNS change, the contract should identify the target zone, record name and type, address, and other relevant attributes. Define whether verification means inspecting the provider’s record state or testing resolution from a particular observation point. Those are different checks. The DNS comparison documents a second GETARECORDS call; it does not establish propagation behavior across resolvers.

Apply Mandatory Criteria Before Preferences

Define mandatory pass/fail conditions before ranking presentation or efficiency. If a required verification step is absent, an attractive notification should not raise the run into the accepted category. Preserve separate scores so a reviewer can see the difference between a process failure and a preference about verbosity.

The DNS Comparison

Put Five Models Through the Same DNS Workflow

The source study compares five model configurations running the same DNS A record agent through FlowAI. The recorded branch includes provisioning followed by a provision-results decision and deletion of the record. The comparison focuses on tool-call sequence adherence, notification presentation, and reported token usage.

Expected Sequence for the Recorded Branch

This sequence describes the expected recorded branch, not every possible approval, rejection, or error branch. The source records six steps for GPT OSS and seven for the other configurations. The difference is the missing lookup between creation and the provision-results decision.

The sequence does not show an independent readback after deletion. A deletion response and independently verified absence of the record are different evidence. A broader protocol should define whether post-rollback verification is required and record it explicitly.

Model Configurations in the Study

The Recorded Results

Compare Execution, Communication, & Token Usage

Execution Adherence & Notification Assessment

The email scores reflect the source study’s qualitative presentation assessment. Keep presentation separate from factual accuracy and execution adherence when interpreting the results.

Reported Token Usage

Reported usage ranges from 21,102 to 48,950 tokens. Each total is the sum of input and output tokens. Actual cost and elapsed time require separate measurement.

Read the Results in Their Operational Context

These results describe the documented runs. Attempt counts and variation across repeated tests are not supplied, so the four sequences containing verification should not be interpreted as a reliability rate.

Choose a configuration against the workload’s requirements. Assess mandatory adherence, communication quality, cost, elapsed time, and consistency separately before making the deployment decision.

Interpreting The Findings

Turn Observed Differences Into Better Model Decisions

The Missing Lookup Is a Concrete Test Target

The GPT OSS trace proceeds from creation to the provision-results decision without the second GETARECORDS call. This establishes an omission in the documented sequence. It does not establish that the model always omits verification, that the record was created incorrectly, or that the final email contained false statements.

The next investigation should examine the instructions, the verification tool description, the creation response, and the model’s subsequent tool selection. Retest the original configuration before changing it. Then change one relevant factor at a time and record the effect. Otherwise, improvements cannot be attributed to a particular configuration change.

Presentation & Factual Accuracy Need Separate Assessment

The study’s notification assessments capture readability, structure, and verbosity. A broader evaluation should also inspect whether the message identifies the actual human decision, distinguishes creation from verification, describes rollback accurately, and states any unresolved uncertainty.

A useful notification can report an incomplete run honestly. Conversely, a well-formatted message can overstate what the evidence supports. Grade those cases differently. Presentation should help the operator understand the evidence; it should not obscure missing evidence.

Lower Usage Does Not Establish Better Efficiency

The lowest-usage configuration also omitted verification. The highest-usage configuration received the strongest presentation assessment. Neither association establishes causality. The available record cannot show how much usage came from the verification step, reasoning, tool output, conversation history, or other provider behavior.

Cross-model tokens are also not a common unit of identical work. Tokenization and billing conventions can differ. Treat the counts as provider-reported consumption and reconcile them with pricing and measurement scope before drawing cost conclusions. The decision-relevant metric is the resource required to achieve accepted operational results, including unsuccessful attempts.

Model Selection & Execution Design Improve Together

If a required check can be skipped, model choice is only one response. A team can also make the check explicit in task guidance, improve the relevant tool contract, or place it within an execution path that requires evidence before the task is accepted. Those changes must be tested rather than assumed to resolve the issue.

This is why the harness belongs in the evaluation. It provides an operational setting in which a team can observe behavior and refine the configuration. The study demonstrates differences within FlowAI; it does not measure FlowAI’s relative effectiveness against another harness.

The Testing Protocol

Build a Repeatable Model Evaluation

Use the following protocol to extend the DNS example into a repeatable comparison for your infrastructure workloads.

Establish the Task Contract & Initial State

Specify the requested resource, permitted actions, required evidence, branch decisions, stopping conditions, and final reporting requirements. Define the acceptable terminal state for each branch. For provision-and-retain, the record should match the request. For rollback, the expected state may be absence of the newly created record while preserving any pre-existing records.

Use a controlled environment and restore the intended initial state between attempts. Record pre-existing resources and generated identifiers. A cleanup action must target the resource created by that attempt, rather than assume every matching name is safe to delete.

Record the Configuration

Do not assume the same numeric setting has identical meaning across providers. Unsupported settings, provider defaults, and request transformations belong in the record. A model comparison is reproducible only to the extent that these dependencies are visible.

Collect Comparable Execution Telemetry

OpenTelemetry (OTel) is an open-source, vendor-neutral observability framework and a Cloud Native Computing Foundation (CNCF) project. Formed through the merger of OpenTracing and OpenCensus, it provides tools, APIs, and SDKs for generating, collecting, and exporting traces, metrics, and logs.

Where instrumentation is available, teams can use OpenTelemetry to correlate model requests, tool execution, errors, and timing across an evaluation run. Associate telemetry with the run identifier and model configuration so reviewers can connect resource measurements to the actions performed.

Telemetry supports the evaluation record. Acceptance criteria and infrastructure state checks still determine whether the intended operational outcome was achieved.

Exercise Distinct Branches

Begin with provisioning and retention, rejection before provisioning, provisioning followed by rollback, creation failure, and verification mismatch or failure. Add malformed inputs, pre-existing records, ambiguous tool responses, and interrupted approvals where relevant. Each scenario needs its own expected actions and state contract.

For example, rejection before provisioning should not lead to creation. A failed verification should not be represented as a verified success. Define recovery behavior in advance so the assessor is not inventing the expected response after seeing the model’s output.