Bring Your Own Models: Govern Network Changes With Jev & FlowAI

John Capobianco • 00:01

Hey everybody, John Capo Bianco here from Itential, the head of AI and DevRel, and I’m going to be showing you how we can introduce the new decision models, System One thinking models, in the form of JEV today. Microsoft has introduced their own. There are no costs for output tokens. Let’s start there with these models.

John Capobianco • 00:27

There are no chat interfaces, meaning the slash chat slash completions API is not present. We’ve explored a model like this in Antares. Antares only took in a CWE number, and in my case, a link to the code, the GitHub repository, for looking up weaknesses. Jev takes an input, a standard input like a JSON payload and a question, a prompt, but it’s not a chat prompt.

John Capobianco • 00:54

And the it will only pay a minimal, minimal amount for input tokens. I was given a five dollar starter fee account with Jev. I’m still milking that weeks later, and I’ve done some amazing things with it. I thought

John Capobianco • 01:12

And it wasn’t even if I know that this is going to work and I’m really excited. And I’m actually recording my first, I usually do a dry run, but I wanted to do this one live to get my genuine reaction and response and see how this works out. But I also will show you how this was built. So we’re gonna start with what I built, we’re gonna do a demo or two, and then we’re gonna explore how I built it, which I think is just as important because the how I build it is much different.

John Capobianco • 01:41

Than how I build other solutions now that I’m working with agentic solutions. So building it is agenc. The outcome is also agentic. This is a solution factory. So let’s take a look at some architecture images. Let me make sure I have them up on the screen first. preview. And let me start sharing my screen.

John Capobianco • 02:13

Okay everyone, this is the CML that we’re working with with the two routers, two switches, and a management plane. And let’s take a look at the AI images. So that on that image that I showed you, or that CML, all it has is management. There’s no BGP, there’s no OSPF, no routes, no static routes, other than the management default gateway.

John Capobianco • 02:40

no VLANs, no subinterfaces, nothing. Zero. Nothing. Okay. And I’m gonna build this agentically using this sort of multi-agent agentic workflow where Opus is gonna coordinate, Sonnet’s gonna reason, and Jev is gonna be the decision maker. Right? So I don’t need to go over all this stuff, but the coordinator is gonna call the planner, the investigator.

John Capobianco • 03:08

All of the sub agents have access via the gateway to the MCP tools. And those MCPs like Py ETS can reach the network devices. The JEV TypeSafe MCP that I wrote can reach the GEV API. JEV is going to give us proceed, halt, approve, deny, accept, deny, right? And Gemma is going to be used for communications for Slack and email because we don’t need to spend the tokens.

John Capobianco • 03:36

It’s just a simple communication and summary, and Gemmafor is perfectly fine to handle that. A little deeper dive into the human in the loop and where Jev is going to make the decisions for us as part of analysis and planning, as part of authorization and deployment, and a part of validating and communicating. We’re going to use some meat tokens here today with the human in the loop and go into the work center and apply our senior network expertise.

John Capobianco • 04:06

Review what Jev has already approved, by the way. This is just one more extra human approval. Four agents, three Jeb decisions, one human in the loop. Then I’m gonna reset the lab and we’re gonna do the exact same setup, but we’re gonna take the humans out of the loop and see if we can get the Jev System One thinking to be the only approver for the execution that we need and not be the Homer Simpson you know, tappy bird.

John Capobianco • 04:35

That just presses yes all day to agents recommendations. So let’s hop into the projects. Let’s take a look at the coordinator real quick. It has a big prompt here with lots of great stuff. I’m not gonna read it, but I will show you later how I built it. I did not type a single line of this. I built this with the co codex builder skills. Okay. We just have a dummy variable to take in a string when we kick it off, and it

John Capobianco • 05:05

Its tools include other agents. Isn’t that cool? So it’s going to call these other agents. We have the five tools here. All right. Including the Jev Evaluate.

John Capobianco • 05:21

If we hop onto the and by the way, the coordinator is using the Opus 4.8 model. When we move into investigator and planner, it only has two show commands. Well, John, what if the coordinator or the investigator tries to configure something? How? It can’t. It doesn’t have the tools. At build time, we really limit the scope and only give the agent what it needs to achieve its goals. So, yes, the executor is going to have.

John Capobianco • 05:50

PyTS configure with diff, but that is the only agent with any configuration capabilities. And then finally we’ve got our communications agent using Gemma 4 with an email option and Slack notification.

John Capobianco • 06:23

Okay, let’s fire up this agent. Right? Let it rip. Rip. Run now. All right, so we can see that this was manually triggered by me at this timestamp. We’ve got the session ID and we can split this out to its own window. It knows the first step is to call the investigator to collect the baseline. So let’s open that. And it should be running a bunch of show run commands.

John Capobianco • 06:50

and including running config and a couple of others in parallel for the show commands, not show run, but the other commands will all be p-calls. And it’s gonna gather up as much state that it needs from the four devices to make its initial investigation. All right. So you can see now it’s moving into some parallel calls here.

John Capobianco • 07:13

Where the MCP is able to take all four devices and run whatever this was, show IP interface brief, I think. Okay. So it’s only gonna take a couple minutes now that we’re using parallel calls to gather everything. And then there’s gonna be a a reasoning step that’s gonna take a little bit of time, a couple of cycles of reasoning before this hands it back off to the you know, the other agent.

John Capobianco • 07:40

the the controller agent the the what’s it called the coordinator thank you all right so couple more parallel commands here these ones are just on the switches interface trunks some of them might be just on the routers I can filter down and see just the tool calls or just the reasoning

John Capobianco • 08:03

And I imagine it’s probably now done. And we’re gonna have a little bit of longer reasoning taking place here.

John Capobianco • 08:19

And then it’s gonna hand it back to that coordinator agent, which is going to invoke Jev. And Jev is gonna make a decision on whether or not based on the investigation and based on the out the plans, whether or not we should proceed into the planning phase.

John Capobianco • 08:38

So we just gotta bear with us and be a little bit of patience here. probably doesn’t take as long as it feels because I’m sitting here waiting. but it’s under a minute. We’re gonna, we should get the reasoning back pretty soon here.

John Capobianco • 08:54

And this is again using Sonnet 5.5, this investigator planner agent, whereas the coordinator is using the Opus classification of models.

John Capobianco • 09:21

And maybe through the magic of editing we might be able to cut this down a little bit so it’s we can jump right to the next point where the agents handed this back off to the coordinator. Excuse me, still a little bit sick.

John Capobianco • 09:44

And all this talking doesn’t help.

John Capobianco • 09:53

So we can see here any second we’re gonna get the the handoff back here. You see, it’s only been running for about three minutes, which is pretty remarkable to gather all that state and context from four different devices in parallel. Here we go. So I’m gonna close this one off. We can see that the reasoning here has the evidences ready.

John Capobianco • 10:16

And it’s gonna hand it back off to Jeff to make the decision. Hand off to the purpose phase with Jev’s proceed. All right. So.

John Capobianco • 10:31

We have some reasoning here.

John Capobianco • 10:35

Now perform the direct JEV evidence reviews, starting with the four device scope reviews. I’ll make them sequentially, one call per turn. So excuse me. It’s gonna check the evidence for each of the devices. Here we go. So that one was successful. Sufficient evidence, no blocker. Now the next router.

John Capobianco • 11:31

Excuse me, I needed some water. So we have no blockers.

John Capobianco • 11:38

And we have sufficient scores.

John Capobianco • 11:47

Okay, so we have the scores back from Jev, our decision maker.

John Capobianco • 11:58

Right?

John Capobianco • 12:03

We can see here the confidence, the halt has a very, very low probability, but a very high probability of being sufficient. A little bit of failed and conflicting evidence, a little bit of missing evidence, but no blockers. Evidence scope, the reviewed commands. Pretty wicked, How it made its decision. So now proceed.

John Capobianco • 12:35

With a confidence of one, finding no blockers, planning is authorized. So now.

John Capobianco • 12:47

Hang on, what’s going on?

John Capobianco • 12:54

Need to re-invoke purpose with full evidence ready payload to collect. Okay, so it’s it’s doing some more reasoning here. And it should invoke that planner agent now. So let’s just see what happens here. We didn’t crash, but it didn’t like something here.

John Capobianco • 13:17

So we’ll just let it cook and see what it comes back with.

John Capobianco • 13:39

It’s still reasoning. Let’s just give it some time. I’m just curious what this happened, what happened here. Missing baseline evidence.

John Capobianco • 13:49

So

John Capobianco • 13:53

Reinvoke the propose. All right, let’s see what it’s doing here. Investigator correctly stop. The proposed phase requires full baseline evidence, not just the JEV receipts. I’ll reinvoke propose with the evidence ready payload included. Okay, good. Self-healing agents. Okay, so now we’re into the planner. And we have the fixed intent here.

John Capobianco • 14:21

And it should say here’s the JEV final evidence. Right. So it’s including JEV’s you know approval and then the final proposal here. So the proposal is ready, and here’s all the configs that it’s proposed for the four devices, and then a verification plan to verify. All right, pretty cool. So we head back to our coordinator.

John Capobianco • 14:51

It’s synthesizing the plan. And this now should invoke them the right proposal ready. So step four, direct JEV, the proposal reviews. All right, starting with R1. And now it’s going to start to evaluate these proposals with JEV. And we’ll get that confidence score reinforcing to the human.

John Capobianco • 15:19

So this one had a pretty not a low score, it’s high enough to approve. But we got a 0.74 there. The this the decision choice is to approve. That is what controls. Now R2. And maybe if we take a look at so R2 was pretty, you know, 6.9. Now it’s moving on to the switch. And we can take a look at some of the numbers here the confidence, the probabilities.

John Capobianco • 15:49

failed or conflicting, missing, but we did get you know, the approval to move ahead. The decision was made.

John Capobianco • 16:06

So now we’re waiting on the last device here, switch two, and then I believe it will crunch a little bit more, a little bit more reasoning, and ultimately have the meat tokens have me, the human, with my senior network expertise, go into the work center and approve it. So here we go. We had an approval, the four different scores, no blocker, final bounded proposal aggregation call.

John Capobianco • 16:34

Covering the cross device, OSPF, IBGP, VLAN and verification consistency.

John Capobianco • 16:42

And we had one last evaluation it looks like to move ahead.

John Capobianco • 16:54

And now the reasoning coordinator, I hope, should call me to do the human in the loop approval.

John Capobianco • 17:07

Check it out. Human approval for the NetClaw 4 device plan.

John Capobianco • 17:16

So check this out. We have automated review, proceed on the evidence, per device proposal, proposal final aggregation, confidence of 0.96 with no blocker. We have the plan summary. We have the exact commands on each device. All right. Now look at all that.

John Capobianco • 17:45

PGP, OSPF, point to point links, sub interfaces.

John Capobianco • 17:56

All right, router two, switch one, switch two. Looks good to me. 14 checks, verification expectations, right? On what it’s gonna test and everything. So we’re gonna approve that, submit, and then head back to our agent.

John Capobianco • 18:17

Okay, editors, this is gonna be cut, but just so this should go after I click approve and go to Work Center. We’ll pick up from there. Thank you.

John Capobianco • 18:39

Okay, we’ve approved it in Work Center. And now we’re going to go back to our agent. That’s the whole reason this was human in the loop, H I L. So now we’re going to call the executor and validator. First for the executor part. And Jeff’s going to get involved here. Approval is verified, applying switch one first. So now we’re into PyETS configure with diff, the MCP tool.

John Capobianco • 19:09

That’s going to push all of this config to switch one, and then we’ll get switch two, router one, router two, or whatever order it does them in. Ideally, then we’re going to get a validation from this agent using PyETS testing, pinging from subinterfaces and other things like that. So now we’re running what device is this on?

John Capobianco • 19:38

This looks like it’s the other switch. It is the other switch. All right, and we can see that we’ve got the response from the job here.

John Capobianco • 19:55

Really cool.

John Capobianco • 20:00

And now we’re doing one of the routers, it looks like. So look at the router. A loop back, the point-to-point link, the sub interfaces, the OSPF, the BGP, all rolled out through this agentic workflow.

John Capobianco • 20:22

And now we’re going to do router two.

John Capobianco • 20:29

All of this is immutable and auditable and in our logs and part of the session ID. So now we’re grabbing fresh running configs. And it looks like it’s doing this in parallel. And it’s gonna do some reasoning.

John Capobianco • 20:58

Now we’re running show commands on mass in parallel for it to grab all of the information it needs after it let it let it settle a little bit. And now we’re running some pings from VLAN 1 on R1 to VLAN 30 on R2.

John Capobianco • 21:23

Some more pings. Let’s take a look at one of these pings, what it looks like.

John Capobianco • 21:39

We don’t have much of a success rate. That’s not great. Let’s just see what’s going on here. And by the way, we can also hop into here now and just to show you what what’s happened here in the background.

John Capobianco • 22:04

So we have LLDP, we’ve got a loop back zero. We’ve got, none of the CDP LLDP on management interface, our sub interfaces, our IPs, our OSPF config, our route, our BGP config, no static routes. Pretty cool. Pretty cool. And it maybe is trying the pings again here.

John Capobianco • 22:37

We go. We have a success rate of 100% now. So likely some just some convergence, right? Exclusion conclusion. Here’s the execution result. Three of the 14 checks failed. So let’s see what’s going to happen here. Handing back off to the coordinator. Submit this to Jeff for a result decision. Three checks failed.

John Capobianco • 23:07

If Jeff requests a rollback, call me with operation rollback. Right? Pretty cool. So let’s go back to our coordinator. Execution completed.

John Capobianco • 23:22

Its own assessment, potential issues on three checks. Let’s check with Jev.

John Capobianco • 23:35

Okay, so the approved scoped rollback. So it actually looks like it’s gonna roll this back. That’s interesting. I’m wondering why. Probably because of some of these failed tests. So I’m just gonna make note of some of this, by the way. Let’s go back to the executor. Rollback config. So it’s using the rollback config.

John Capobianco • 24:02

And let me just grab this and send it back to my build agent.

John Capobianco • 24:10

I want to report on this back to my build agent. so we got pretty close. We got pretty close. But this, I you know, I included this to show you the the reason we’re using the decision. Some of those tests failed. and Jev decided that this was a no-go and to roll back. So I’m gonna see what the reasoning says here. you know.

John Capobianco • 24:34

That’s fine. Operator attention needed. Rollback recovery. All right. So that is Jev rejecting the fix. I’ll go back to my build agent and feed this in and see if I can get it to pass. I don’t think this demo would have been as insightful had I let.

John Capobianco • 24:55

just had a happy path and and and and recorded only the success. So Jev actually decided to roll back because of some failed checks. So we’ve got an incomplete recovery. It’s going to go to the Gemma reporter and I’m going to go to my build agent. We’re going to give this another shot and see if we can get it to fully configure and get us through to the next step. But this is a failed attempt.

John Capobianco • 25:20

Hey everyone! John Capabianco here again. Thought I would break in. So what we’ve seen is the whole process, the architecture, and a failed rejected response. Jeff saw something it didn’t like and decided to roll back to the pre-change state. I thought that was worth showing. I didn’t plan any of this. I’m truly sharing Jev’s decision making, and it just decided not to do it.

John Capobianco • 25:48

Let’s take a look at a positive decision to proceed with the config and how Jev’s called again to confirm that we don’t need to roll back after it’s already approved the push. Right? That third last pillar. And then we’re going to look at the Gemma 4 communication stream that I received just to show you that an open source, open weight model can really give you some good stuff.

John Capobianco • 26:17

Alright, so I started this agent project. I click run agent, I put a prompt in here. All right, that kicked off this agent. You can see my prompt here. Alright. And this is a mix of the agent’s card and my string input. Start this human in the loop by calling the planner. Called the planner. I can click this to go see the planner’s logs.

John Capobianco • 26:47

Right? Plan ready. Agent reasons and sees that we have a ready plan. So let’s see if Jev thinks about the plan. So it’s a bigger than 50k, but Jev decided that we’re gonna approve this. All 30 name plan checks passed, and we had some Shaw matching. So we’re gonna have the human in the loop. Now I want to show you this really quickly, okay, everyone.

John Capobianco • 27:16

Because Jev has made my human in the loop decision quite easy to make. Right? Here’s the approved by Jev’s account. Jev approved this. How come I can’t seem to scroll? There we go. I’m sorry. So look at this: the Jev plan checks and all of their confidence scores for each level of the plan and a recovery.

John Capobianco • 27:46

Confidence. all the actual configs that the other model generated, a rollback plan, a validation plan, some required pings. So I said, yeah, that looks great. Thank you for all that amazing help in making the decision almost for me. So now we’re gonna have.

John Capobianco • 28:12

The executor and validator, and let’s actually look at this one. And it pushed to switch one, switch two, router one, router two. Bottom up approach. Nothing failed. All the stages. Any transient errors, the 14 checks it ran, the eight pings it ran, the tool receipts, the switch reads, the router reads, and the pings in order.

John Capobianco • 28:42

Coordinator should now run the name Jev ResultGate. The result gate. Right? So the result gate here.

John Capobianco • 28:54

Gives us a pass with a 98% confidence and a very, very small failure probability. Right?

John Capobianco • 29:11

then we’re gonna hand it off to the reporter. Here’s this conclusion. Everything was accepted, one fresh, human in the loop run executed end to end, no second run started, what happened, all the validations and findings. And if I hop into my email, I have this summary of outcome, technical verification details, key validation findings, session integrity and references.

John Capobianco • 29:42

From the Gemma 4 reporter agent. All right. Now, one more thing I’m going to do. Let me stop sharing my screen if I can.

John Capobianco • 29:56

The next one is the human out of the loop. So it should be the same thing, except it’s not gonna ask me. It’s gonna do full autonomous human out of the loop. So I’m gonna reset my environment. Gonna take a couple minutes. I’m gonna do that agentically, by the way, with an agent. And then I’m gonna kick off the hool human out of the loop and see what happens.

John Capobianco • 30:24

All right, stay tuned. Thanks for your patience, everyone.

John Capobianco • 30:28

Hey, all right. This is the exciting one because I hit run, I sent in a string, and I just went about my business. Human out of the loop with the decision-making capability handled by the JEV model. And it didn’t fail and it didn’t roll back. Could have, but it didn’t. All right, so let’s take a look at this workflow. And that’ll be it for this adventure in decision-making models. Let me clear this off. Here we go.

John Capobianco • 30:58

All right, so right here’s my agent prompt. Manually triggered five tool calls, 15 minutes, start time, end time, 350,000 tokens. We can see all the reasoning. Let’s focus on the tool calling. So the agent calls the planner, Jev evaluates the plans. it’s a big file with all the reasoning from the plans. So we can’t, we won’t look at that. It approves, moves on to the executor.

John Capobianco • 31:27

The executor then sends back its validation results, and we get this amazing pass score and a very small probability of failure with a very high confidence, right? And all these numbers and all this data about the validation that we did with Pi ETS, a testing framework, okay, not just the configuration, but the testing framework, and then we get the Gemma report.

John Capobianco • 31:57

And if we look at all, we can see that the reasoning, the reasoning is being guided by the decision maker. Right? It’s a reasoning agent. It’s not necessarily a decision-making agent. So I’m going to submit all the plans to JEV evaluation, 30 name checks. 30 name checks passed, right? We have an approval from JEV. It’s made the decision as an approve. All right. No human.

John Capobianco • 32:26

Approval task required because we launched the hole mode.

John Capobianco • 32:33

execution across all four devices is successful. I’ll submit the executed result for evaluation with 15 name result checks. Accept. All 15 names pass. No rollback required. Proceeding to the final report. Fifteen minutes. BGP OSPF spanning tree, sub interfaces, IP addresses. You name it. Completely tested too.

John Capobianco • 33:02

So let’s go back to me here. What we’ve seen here, and I showed you the images and everything. One thing I didn’t show you is how I built this. So, what I mean by how I built this, let me, if you just bear with me here. I sort of forgot that I committed to tell you how I built this. give me one second, and I want to show you a prompt result. I’ll be right back.

John Capobianco • 33:31

Okay, swear, this is the last thing. I made a couple of images because I wanted to show you how I built all this stuff, which I think is just as important as to what we’ve built here today. So this is my build ecosystem. And I know it’s noisy, but let’s focus on the blocks. We’ve got my local build workspace, my local Docker, and my Itentio platform in the cloud. All right. At the bottom, we see the connected services in the lab, my CML.

John Capobianco • 33:58

Which is running on my workstation, not this Mac. So there’s even more networking involved here. My System One Decision API that’s in the cloud, wrapped in an MCP. My Claude Model Access, Opus 4.8, Sonnet 5.5. That is in the platform, in the Itential Platforms model registry. Gemma 4 is locally in my Docker running Olama. All right. So this this is a

John Capobianco • 34:28

A really nice mesh of tools, both free and in the cloud and paid for. Now in the local build workspace, I’m using Codex as my harness connected to Astra High Extra High. I’ve installed the Itential Builder skills into Codex. I’ve logged into my IAG CTL as admin, and I have my Git and my Python and my VS Code. My local Docker here.

John Capobianco • 34:57

I’ve got the PyTS MCP as a Docker itself, but in the gateway, I’ve got the TypeSafe JEV MCP right in the gateway. The gateway is another container. All right. Then we reach over through the gateway connection into the cloud where gateway manager and the tools discover, decorate, and authorize. I’ve got my model registry to register Olama and Claude and Opus and Sonnet.

John Capobianco • 35:27

And I have my agent projects up there, which is native agent to agent, human in the loop, and human out of the loop, both of them. So from a prompt to a multi-agent system, from that build environment, we built two different operating modes of agent to agent. So the build brief condensed. Here are some of the things I prompted it. Build native ADA agents on an attention for the CML network.

John Capobianco • 35:57

Use Opus to coordinate, Sonnet to plan and execute, Py ETS for evidence and changes, JEV for plan and result decisions, Gemma4 for the final email. Create human in the loop with work center approval, and HOOL without a human approval step. Find the native gateway tools, preserve management access, freeze the approved commands, and verify results. Publish both modes.

John Capobianco • 36:26

Read back their definitions and retain the test evidence. That through Codex and the Itential Builder, we had the specify, build, publish, verify steps, Itential’s own spec-driven development approach for infrastructure, lending heavily from the software spec kit from GitHub. And at the bottom, we see that build system.

John Capobianco • 36:54

We’ve got our coordinator talking to our planner, talking to our executor, talking to our reporter, our planner and executor. the coordinator, excuse me, has Jev built in. Jev planned, checked the plans, and it checked the results. 30 plans and 15 results, which led to one very well-informed human-in-the-loop decision and one that didn’t involve humans whatsoever.

John Capobianco • 37:25

So let’s go back in here, stop sharing the screen, and wrap things up. Jeff is less than a month old. I like to really incorporate and embrace and test and verify and get my hands dirty. It’s the only way to learn this stuff. Okay. And this took me very long time. And it wasn’t because of any failures and itential failures and MCPs.

John Capobianco • 37:55

It was because Jeff didn’t like what it was seeing. And the workflow takes some time to reset the lab and this and that and a rollback. So it did take some time. And there is some edits in this video. I know I set out to have this completely be edit-free, end-to-end, but I did show you the failure. I did show you what I’ve gone through to make this work. Now at the end of that, I have a human in the loop, human out of the loop, 15-minute runtime, taking a

John Capobianco • 38:25

Four node topology from nothing but management access to fully configured BGP overlay, OSPF underlay, sub-interfaces, router on a stick, four VLANs, all the spanning tree guards at layer two, trunk, the whole deal. All right, so I hope this was exciting and inspirational, and maybe it gives you some ideas. One final architectural thought.

John Capobianco • 38:51

There is likely going to be a way for us to make an agent that is the decision-making agent. This is very early, and I’ve adapted it through MCP for the API call to use it as a tool. So think of this as two ways: an agent with decision-making capabilities, but decision-making is not its primary function. A validator, a planner, an executor, right? And it’s using the decision making to help that agent’s own function.

John Capobianco • 39:22

Or a dedicated decision-making agent, and we have the full mesh graph where we send it down, send it back to the coordinator, send it to the decision maker, back up and down. That probably scales a little bit better. It’s probably a cleaner orchestration flow where we just insert the decision-making agent as opposed to burdening an agent with both decision making and testing, executing, planning. So

John Capobianco • 39:51

Much more to come. Thank you for tuning in, and we’ll talk to you soon with some more vibes.