Building Production-Ready Agents for Network Infrastructure

Holly Holcomb • 00:07

Hello, hello. Welcome, everyone. I know that we probably have a few folks still joining. So maybe we’ll give it just a few minutes and then we can go ahead and kick things off. Karan, how are you doing?

Karan Munalingal • 00:25

I’m doing really well, Holly. How are you? Very excited to be on the AI office hours. So thanks for having me.

Holly Holcomb • 00:32

Yeah, I know. I am super excited. This is a topic that I feel like we have been discussing a lot with a ton of people, especially among our customer base. And so I think it’ll be a really awesome session. I’m looking forward to it. So I think just in the essence of time management, maybe we can go ahead and kick things off. You know, like I said before, I really appreciate everybody making time to join.

Holly Holcomb • 00:58

Join for our office hours today. I know it’s in the middle of the week. It’s in the middle of your workday. And for some of you over in the European time zone, it’s late in the afternoon. So thank you. Thank you for joining. I am really excited for us to talk today about building agents that actually work and getting some of the lessons learned out of our AI innovation program from Karin.

Holly Holcomb • 01:22

So before we get started, for those of you who do not know me, my name is Holly Holcomb. I work with our customer success team. And I know a lot of familiar names that I’m seeing on the list here, but I’m. Working closely with customers to talk about what their goals are. And traditionally, that’s been really focused around orchestration and automation. And now, lately, we have been shifting more and more towards talking about agentic operations and what that looks like. And specifically, joining us today to talk about some explicit lessons learned from customers from our AI innovation program is Karin Unelingl.

Holly Holcomb • 02:00

He’s our SVP of AI Strategy and Innovation. And I’ve had the pleasure of working with Karin for many years now. Karin, I think you’re celebrating an itential virus. Is that right?

Karan Munalingal • 02:11

That’s right. I hit number 11: 11 years at Itential.

Holly Holcomb • 02:15

Oh my gosh, 11 years. That’s awesome.

Karan Munalingal • 02:17

It’s been fun. Yes.

Holly Holcomb • 02:19

Well, super grateful to have you and very excited to talk about this topic today. Before we dig in, I do want to say we do hope that you’ll ask questions to put notes inside of the chat. If you’ve got any questions, things that you want to know more about, we can either cover it offline, all of that. So please, we love engagement. We love hearing customer questions. Please feel free to send them our way. All right.

Holly Holcomb • 02:48

So I’m going to take a moment and get grounded on what we’re going to cover during today’s session. And the purpose and goal of today’s session is just to make sure that you guys walk away with a good understanding of some of the very specific lessons learned that came out of the AI Innovation Program. We’re actually going to do a really brief overview of what that program was, what that private beta was. Karin’s going to walk us through what it looked like. We’re going to talk about the AI journey that we see across the industry right now and how that maps directly with customers like those of you here. And then we’re going to dig into the meat of this webinar, which is the lessons learned. We’re going to talk about the things that Karin saw coming out of that program, that private beta, and how it influenced not only the way that we look at agentic operations, but how it influenced the product, Flow AI.

Holly Holcomb • 03:40

And then we’ll talk about what’s next. Before we dig into an overview of the AI Innovation Program, I want to get grounded on what we hope to accomplish with this particular session. Our goal is to make sure that you guys walk away with a good understanding of some of those lessons learned from the AI Innovation Program specific to building agents. So, there are things that came out of the AI Innovation Program that were really specific to the customers that were actively trying to solve problems with agents inside of their infrastructure using what they have today. And that’s inclusive of the platform, that’s inclusive of their adapters and integrations, all the things that they’ve already set up. I know that the cohort of customers. Oh, I think we might have somebody who might be muted.

Holly Holcomb • 04:22

Maybe, Brett. So, we had a few different verticals that were involved, and in terms of company profile, the size of companies was also pretty diverse as well. So, I’m really looking forward, Karin, to you sharing a little bit more about what the AI Innovation Program was and giving us kind of the who, what, where, when, and how of the program and letting us know some of the insights that you got.

Karan Munalingal • 04:47

So perfect. That sounds good. So hello, everyone. Nice to see some familiar names on this list that I worked with in the past. But, you know, this is the 1st time, you know, as it’s done this, right? We have actually gone out and worked with a cohort of our customers in order to actually work with them to understand how the agentic AI will fit into the infrastructure land, right? So hence the innovation program, where you will notice here on the left-hand side, our goal mutually with the participating customers was to give them early access to the capabilities that we’re actually building, which is now Flow AI, it’s GA.

Karan Munalingal • 05:30

We’re all super excited for all of our customers to take advantage of that. But, you know, getting to the GA, the thought process was if we had a cohort of our customers participating with us with early access to the capability and be able to improvise that in their environment. And it was a learning experience for both the customers as well as us, you know, someone who’s providing the technology to them. As part of the entire program, you know, our goal was to truly understand, just like we did for the automations and the orchestration, what use cases and what needs are primed for taking advantage of reasoning, right? And I’m going to talk about this a little bit later on, but when you bring agents into the picture, naturally the agents interact with LLM. So now it’s time to understand that: hey, is every use case good for an agent tech AI use case? No, that’s what we learned.

Karan Munalingal • 06:30

Right. Number two, the thought process of having a solution design mentality before you go and start building agents, right? Naturally, all of our customers here that have joined the calls, but anyone else who is actually using Idential. They go through a familiar process of, you know, the SDLC. You know, you gather requirements, you have your backlog, someone bills it, someone tests it. The same thought process, like, do we now have to apply that to agents as well, where it’s a combination of setting context, right? Which is prompts and then associating tooling.

Karan Munalingal • 07:03

So we wanted to understand and learn on how do we work with a set of these customers and create a path for everybody else, right? So you’ll notice the prime thing as part of this program was to work with our customers where they were actually building. Brand new agents that do brand new things that address the backlog. But each one of these customers also wanted to know: hey, I’ve already built this very complex orchestration using Itential. What would that look like as an agent? Right? That was a big part because everyone wants to compare what I’m doing today.

Karan Munalingal • 07:41

How better is this? Right. So we 100% actually went through that exercise for every one of our customers. And the final piece, which was very important to a lot of our customers, is co-developing an AI strategy for infrastructure. One of the things going into the program, what we wanted to do is create something that is a left behind for participating customers so they can now take that to leadership and say, hey, we have tried this technology in our environment, just like we build workflows. Now we’re building agents securely, but this is a path. On how we can include more people to actually build that with us, right?

Karan Munalingal • 08:19

So it’s not just, hey, the technology work, there’s always a people in the process part of it. So that’s something that we 100% wanted to incorporate as part of this program. And as Holly kind of called out, right, it’s as part of this cohort, you’ll notice like tier one service provider, you know, financial services organization, MSPs, you know, we had utility organization that was part of it. So from our perspective, I think we hit every possible vertical we could because every one of these verticals have their own set of requirements on the compliance part, especially when introducing agentic capability within their environment. So, you know, that was that was the whole scope of the program. And then, you know, Holly and I will discuss some of the lessons learned in the upcoming part of our session.

Holly Holcomb • 09:10

Awesome. Yeah, I think one thing that you just mentioned, Karin, that is like really key to point out is that there was a lot of diversity in the types of customers that were engaged. There’s a lot of diversity in what they were trying to accomplish. And I would guess that there was probably some diversity in what they had so far, like where they started with when you 1st began the program with them and where they ended up kind of thing. I know that one thing that I have really appreciated hearing you speak to historically is kind of the overall journey that we see across the industry with customers and what that looks like, especially with the influx of Products and noise, and updates, and news about agentic operations. Like, what is the through line?

Holly Holcomb • 09:53

Like, what do we actually need to focus on? And so, I know that I’ve really enjoyed hearing you talk about the journey. And I’d love for you to give us kind of your take on what the journey is and how it mapped to what you saw in the program.

Karan Munalingal • 10:08

100%. So, you know, before we actually started the innovation program, this journey, right, internally, this was our vision, very similar to how for the last 12 years, all of our customers have joined with us to take the path towards automation and orchestration. We said, like, let’s sit down and figure out how do they do that with agentic AI for infrastructure. So last November at the customer advisory board, we actually presented that to a cohort of our customers again, right? It’s we said, hey, this is how we envision all of our customers going through the maturity curve, as well as a roadmap that they can improvise to safely introduce AI for infrastructure, right? So one of the things that we have learned, and our goal was to actually validate this as part of the innovation program is. The cohort of customers that joined, they’re all large customers, right?

Karan Munalingal • 11:05

So I just, you know, in my head, I just assume like everyone has a like a strategy, you know, and a roadmap on how they’re actually going to do that. And, you know, it was a lesson learned that not everyone does. But what we actually learned is going through this roadmap here, a lot of folks had already started experimenting with ChatGPT, Gemini, Claude, et cetera, right? Like they’re at the stage where they’re using generative AI to ask questions, get responses back, and then go do their work, right? Everybody has already gone through that phase when we chatted about it. What I will say is 50% of the customers that participated in the innovation program, they were further away from the experimentation. They were actually leveraging MCP integration.

Karan Munalingal • 11:56

And that too with Atential, because Atential released that last year in May. A lot of our customers had actually started improvising MCP on top of their platform, the Atential platform. But some of them actually internally had an effort going on to build their own agentic stack to be able to build agents and run an agent. So it was very interesting to see and hear from them before, you know, as they’re joining the program, because they’re all interested and excited to test out Flow AI, because they already went through some of the tribes and tribulation of standing up their own stack to experiment with. You know, so some of the things that came up was: hey, when I stood up my stack, it was a must for me to have MCP servers in order to integrate with some of these technologies, right? And then when I connect to multiple MCP servers, now I’m having to write classification rules so my agent doesn’t pick the wrong tooling. So these were some of the lessons learned, specifically on building an agent and running an agent, right?

Karan Munalingal • 12:58

When we started asking, hey, so how do you think about securing and governing what you’re building? That was not, that was not even what they had planned to work on in the next few months, right? That’s like, hey, we’ll figure that out later. So you can see that, you know, there’s a lot of opportunity out there for folks to stand up their own stack and get excited to build some of these agents. But when it’s time to actually put things into production, your Infosec team, your AI governance team will start asking questions on how secure is it? Like, have you built a moat around this to make sure nothing goes wrong? Right?

Karan Munalingal • 13:38

Because agents think for themselves and they’ll do stuff if there is no context control, there is no security with respect to what it’s able to get access to from a data perspective. So these are some of the questions that also came up. As we’re working through the innovation program with the customers that are in different portions of this journey. And it was very interesting. Everyone that participated agreed to the fact that at some point we do want autonomous operations where they envision Agents coming from their other investments, like whether it’s, you know, monitoring or AI ops, they’re also providing agents. They envision, hey, with Itential being the center of their reference architecture that actually makes changes into a network, it would be triggered by another agent that does the initial piece, right?

Karan Munalingal • 14:31

So it was great to get that validation that we thought about this a few months back and also understanding that customers are now starting to compare, hey, do I just have this as part of my platform or do I need to go and build the entire stack of capabilities again by myself and manage and maintain it? So Holly, that’s all I have with respect to the journey that we also see the next set of our customers taking is it’s very similar to what they did for automation. But this time around, because it’s an agent that can think for itself, there’s a lot of scrutiny on, hey, human in the loop, like who’s going to do what? Can the agent do this? Can the agent not do this? So a lot of that will be covered in the lessons learned as well.

Holly Holcomb • 15:22

Awesome. And I know one question that has been kind of like tossed around internally is like, is this like, is there one path that goes across every single one of the stages that you’ve outlined here? Is it one kind of like linear path? Do you have to hit every single one of these stages in order to be effective?

Karan Munalingal • 15:41

No, we believe the reason why we actually created like the inspiration behind Flow AI is to actually help a lot of customers leapfrog from experimentation straight into being able to build agents, right? I think naturally it’s a trusted platform for a lot of folks that have joined today. From our perspective, what we built and why we built it is because it’s now inside the platform. So you already have secure integration with all your technologies because you guys are all building workflows and associating scripts, et cetera. So think about leveraging what you have already integrated to and can integrate to securely and now having an agent leverage those things as tools. So the opportunity that we have, and I know we’re going to talk about it later, Holly, is we have an opportunity to help accelerate the backlog that is sitting there where a lot of folks are now planning out, hey, I have to build a workflow for this. I have to build a script or a playbook.

Karan Munalingal • 16:45

Can we now leapfrog and build agents to do those same things? Right. And that’s what we’re starting to see is customers are coming in and saying, hey, I can actually. bring more people to the table because they don’t have to learn how to build a workflow necessarily, right? I think this is an opportunity for more folks to actually participate that have the knowledge about the network and infrastructure.

Holly Holcomb • 17:10

Yeah, I think that you said two things there that I really appreciate. A few minutes back, you mentioned that we had, we thought that a lot of folks would have very explicit plans and strategy in place for how they’re going to embed AI into their work today. And then the point very specifically of like, use what you’ve already got. You’ve already created a layer that has audit trail, that has integration to the different systems, it has a mechanism for reaching out to devices. Like use what you’ve got and leapfrog across these stages. So I know that it can be really tough, especially when there’s lots of governance and oversight, people asking questions around what your strategy is and how it’s going to work. I love the idea of us using what you’ve got to move faster.

Karan Munalingal • 17:54

Yep, because it’s already certified and it’s working, right? And Fosec signed off on that, and we’re already touching the infrastructure to drive the value. Now we can get accelerated value, bring in reasoning into the picture.

Holly Holcomb • 18:09

Absolutely. Okay, it is at the moment we’ve all been waiting for. It’s time to talk about some lessons learned. And no webinar would be complete without an amazing demo as a part of it. So, Karin, I’m super excited for us to dig into the lessons learned and also get a sneak peek, a demo of Flow AI as a part of it. So, one of the things that I know we have had a ton of conversation about internally and also with some of our customers is specifically the time to build. How much faster you are able to create and build with agentic reasoning.

Holly Holcomb • 18:46

So, I’m hoping that maybe you can share with us a little bit about what you saw during the program with our customers and what that looked like in terms of speed of build.

Karan Munalingal • 18:55

Yes, it was very interesting. And everyone that has joined today and everyone that will be watching who is an existing intentional customer, you know, from an orchestration standpoint, our drag and drop studio capability is a thing, right? Like it already makes your life easier because now you can take advantage of APIs and your existing automations and drag and drop and create the end-to-end orchestration. So thus far, every one of our customers have been fully focused on building deterministic execution capability, right? Whether it’s interacting with the playbook, whether it’s an end-to-end workflow with APIs and command templates, et cetera, like it’s all static. You know what you’re building because you have the instructions and the requirements, right? But what’s interesting is the network does not remain static.

Karan Munalingal • 19:47

The network constantly changes. Everyone here, including myself, when I was doing implementation part of the PS team, because a requirement change, because of vendor change, we had to go back and modify the workflow, right? It’s not the 1st time build, it’s the managing and the maintenance that also is part of the entire life cycle of a deterministic execution. So, from a faster time to build, you can imagine if any one of these customers that have joined today, if you go back and look at one of your workflows that actually solves a problem, you’ll very quickly realize that throughout that entire workflow, whether it’s one workflow or a lot of child jobs, et cetera, 60 to 70% of that is business logic as well as data transformation that’s happening across scripts and between APIs, right? And when it comes to reasoning, This is why we have introduced reasoning as a leverage within the platform, because when an agent is able to reason with an LLM, the general LLMs are really good at solving problems around business logic, data transformation, right? They’re really good at that.

Karan Munalingal • 21:04

So you can imagine someone saying, hey, I’m going to, as part of the innovation program, right? When they, one of the tests was, I’ve already built this complicated workflow that already works. It’s in production. I wanted to see what it looks like as an agent. So what they did is they basically said, I have a mop based on what I actually built this workflow. I’m going to turn that into a prompt. So when they turned that into a prompt and associated tooling, they very quickly saw that the agent did exactly the same, actually more than what their current workflow does, right?

Karan Munalingal • 21:38

That is a light bulb moment for a lot of our participants because they actually saw what they built. They went through the tribes and tribulation to build that workflow that works today. And then they came and built an agent, right? It’s a self-validating thing to say, was it faster or was it slower? On average, every agent that the customers built as part of this program took somewhere from 60 minutes to 120 minutes, right? Max 2 h . Some of it were built within 15 minutes because we already had pre-baked templates and stuff for them.

Karan Munalingal • 22:13

So, you know, you can imagine how the reasoning helps quite a bit here because you’re not having to figure out how to pull data from one API into a next. The reasoning takes care of it because it understands the schemas between scripts and APIs, and it’s able to conform and create those payloads based on the intent that you define within the prompting, right? So this is where, you know, when we talk about. Time to value. We have an opportunity to work with all of our customers to drive this faster. If you have a backlog of things, agent 1st is a mentality that we’re driving because it does give you an edge versus having to write every step down. And what if the steps change?

Karan Munalingal • 23:01

Now you’re going back and modifying flows. So, you know, that’s where the time to value comes in, Holly, is the introduction of agentic reasoning paired with existing. Deterministic capability like APIs and scripts and existing workflows, right? Those still remain. It’s just customers have the ability to just go faster because they’re like, whoa, instead of me having to drag and drop and do data transformation, I can just write what I want and it does it.

Holly Holcomb • 23:29

Absolutely. I think of all the times when I have seen a customer struggle to get like the steps or the mop or the functional requirements, whatever you want to call it, written down from maybe their end users to be able to put into an automation and how much frustration and lag and delay that causes and the overall time it takes to get something out the door. And I also think about the things that have like historically been really hard to automate, like triage-related use cases. Did you see much of that as a part of the program? People trying to focus on things that have been historically really hard to automate?

Karan Munalingal • 24:05

Yeah, it was very interesting because, you know, going into the program, very few customers went after the Provisioning use cases, majority of the customers, because they’re just getting their hands dirty, right? So they wanted to do a read-only agent. And one of the challenges today, as a human, is when you have to go across so many different systems and gather that information and create a report or even understand what’s happening based on an issue that someone raised. That was a perfect time for them to prompt it out and have an agent go do that because the agent can actually correlate and make sense if there’s an association of issues or not, right? So very 1st thing that they did is folks created the network health check agent. They created a circuit health check agent.

Karan Munalingal • 24:57

We had some customers creating compliance agent. Like they were all read-only, right? That agent is not making a change, but we did have customers that took a step further and said, hey, I’m happy with the read-only agent. Now it’s time to actually go change something based on the context that I’m providing. Right. So you have to remember, I think there was a level of assurance for some of these customers participating in the program is because they saw how Flow AI still leverages a platform capability to make changes. It doesn’t just directly go to the network.

Karan Munalingal • 25:33

So they at least validated that to say, hey, just because I tell an agent go do something, is it going to directly do it? Or it always flows through the governance and the access control and the secure integration. So that was definitely part of a test that we went through for it with every one of our customers.

Holly Holcomb • 25:52

Love it. I know it’s something that really hits home with a large swath of our users as well. Triage is like, it’s everywhere. Everybody that’s managing a good deal of infrastructure has to deal with triage and day two related activities. And it’s an excellent use of reasoning.

Karan Munalingal • 26:11

Yes, I think we’ll talk about it when we get into a demo around the CVE, right? It’s impossible for someone who has to go and assess how much of their infrastructure is impacted by a brand new CVE that Cisco rolled out or Panorama rolled out. Like, even in a workflow, it’s difficult. Even in a script is difficult. But when we started transitioning towards an agent doing it, it was fairly straightforward. So that was a great lesson learned that there are certain things it’s just going to be hard to write a script for or even create a workflow for. And it probably doesn’t make sense to do that.

Holly Holcomb • 26:49

Yeah.

Karan Munalingal • 26:50

Love it.

Holly Holcomb • 26:52

Okay, our next lesson, we’ve got a couple on here, and then we’re going to transition. You’re going to show us a really cool demo to kind of reflect on the last lesson learned as well as these next two. Million dollar question for you, Karin. How do you build confidence and trust with the people around you? Because I think we’ve all got, there’s a lot of fear that goes into introducing reasoning or an agent into your infrastructure. So I would love to get your feedback on how you guys work to build confidence and how that reflected in the product.

Karan Munalingal • 27:27

That’s great. Before some of these customers actually joined the program, we had to sit through a lot of questions to be answered around just the platform and what Full AI is, how it connects, et cetera. But once that was kind of stowed away, we installed the software, we got it connected into their environment. I think the 1st thing that someone asked is, how do I control what someone can build? as an you know from an agent perspective because you have to remember like if you saw the journey folks are coming from the world where they were using chat gpt it’s an enterprise uh tooling right but now they’re coming to hey like this agent actually touches my network so i need i need you to tell me how much control and security i have before someone builds anything and when it executes i want to make sure that it securely executes only what i told it to do number three i want a full detailed log of what it actually did right three things you want control at a build level you want control at a execution level and then you want visibility to exactly what it did so uh from our standpoint i think As we got into the program, as we heard some of the feedback and fear, we improvised a lot of this within the platform, right? Naturally, I mentioned Holly and to the rest of the folks that joined, right?

Karan Munalingal • 28:57

As you’re embarking on having to build your own stack just to kind of experiment or try it out, there’s a lot that goes into making sure that the agent doesn’t hallucinate and picks a wrong tool. It actually, from a platform perspective, because we’re already integrated securely into all the technologies that you guys are using, building workflows with. Imagine those same APIs, those same scripts, those same playbooks. Those automatically federate up to become tooling for our agents, right? So we’re not changing anything on how you’re communicating with your network, with your third-party system. All of that remains the same. The only thing that we have changed with respect to the capability within Flow AI that was tested is explicit tool association during build time.

Karan Munalingal • 29:47

So, e.g. , Holly, like you might come in today and you say, Hey, I want to build an agent, but I truly don’t know all the APIs, right? So today, if you were to hook into an MCP server, it automatically gets access to those APIs, and then you end up writing classification rules. What Itential has done as part of Flow AI is explicitly allowing. SMEs to pick which tool the agent has access to. That’s your 1st level of measure that the agent can only do what you ask it to do with these three tools. It’s like asking a, it’s asking like a carpenter, but not giving a carpenter a drill, right?

Karan Munalingal • 30:32

It’s like, hey, you can do your job, but you can only do it with a saw, right? You’re literally restricting what the carpenter could do. The same thing with agents. So from our perspective, I think what we have done really well to support this level of governance is access control to who can build agents, access control to what they can build agents with, access control around who can execute the agent, right? All three of that become very important with respect to building, but also executing at scale against very critical infrastructure. The 2nd piece I do want to talk about is You know, existing secrets management, existing secure integration into your technology, like none of that was invented just for Flow AI.

Karan Munalingal • 31:25

It already existed. So it was very easy from an itential standpoint to actually have Flow AI being part of the platform. So we’re leveraging all the platform-level capabilities that we’re already providing to folks who are building workflows and onboarding scripts, et cetera, right? These are the enterprise governance capability that already existed. So from our standpoint, we’re enabling our customers to use what they already feel secure with and now just layer the reasoning and the agentic part on top. So that was the 1st part. I can dive into the 2nd one unless you have a question about the 1st one.

Holly Holcomb • 32:02

No, I think that the piece that I’ve heard you use this analogy in the past, but like if you similar to your carpenter tool analogy, like if you give a human a hundred page document, it’s going to be hard for them to figure out how they extract what it is that they need in order to be effective at their job versus if you give them a one-page summary of exactly what they need in order to do their job. So I think that it’s it makes a lot of sense when you start putting it into context around like the tools that you’re giving the agent in order for them to be effective and how explicit tool association and selection is critical for that.

Karan Munalingal • 32:37

Yep. And we that was definitely one of the lessons learned because as we begun the program with some of the customers, like obviously there is excitement. So they see, oh, I have access to this tool and that tool and I’m adding all that here. And then you run the agent and you’re like, hey, why did it use that tool versus this one? Right. Then you start controlling the narrative for the agent to say, hey, I don’t want you to ever even think about that. So I’m going to explicitly only allow you to do this.

Karan Munalingal • 33:07

Right. So you immediately start securing what the agent can ever do in your network or anywhere else.

Holly Holcomb • 33:14

Absolutely. All right. For us to dig into our 2nd lesson learned on the slide, I know that you and I have both been at some point in time part of our PS organization where we’re actively working on implementing with our customers. And the age-old problem that we run into is we’ve got an end system, another vendor that we’re integrating with. And we have to spend a ton of time trying to figure out what the bespoke payload is for that particular system because the way that the API is structured is not super helpful. What does that look like with agents?

Karan Munalingal • 33:50

Like, what does that look like with agents? I’ll tell you what it looks like with agents: the lesson that we learned is: as a human, when you give me an API and the input says body, it’s a single input, it says body, it’s an empty object. As a human, I have the ability to go read documentation and specifically figure out, hey, this is what I want. When you give the same paradigm, the same API as a tool to an agent, what we learned is the agent has a goal that you gave it. So it will keep trying and burn your tokens to figure out what is the right payload to do that job. So you can imagine, as bad as it is for a human to figure out every payload that we have to put in, it is worse for an agent and it’s burning your tokens. So, you know, with that in mind, this was one of the things that we actually implemented within the platform because we realized this soon enough to say, hey, the agent is slightly hallucinating because it doesn’t know what to provide as an input to that API because the vendor did not actually provide a proper schema.

Karan Munalingal • 35:01

Like, this is all the stuff that I need. It just says empty body, right? And we see a lot of that. Like, this is very common across a lot of vendor APIs. So with that thought in mind, we have what we introduced, right? This was a lesson learned, which then turned into a capability that we provided as part of the platform flow AI, is to be able to decorate some of these things. And I’ll show a demo of what that looks like because it’s not just about.

Karan Munalingal • 35:31

You know, vendors that have API with a single input. It’s also about vendors that have an API that has 60 inputs. It’s the same, right? Because your agent is still trying to understand out of these 60 parameters, like which one should I be sending based on the prompt. So either you convolute a prompt and make that super detail, or you can now define the boundaries within which the tool has to be called. Right. And as part of the lesson learned, I think customers appreciated that they had full control of how they define a schema so the agent doesn’t hallucinate, but also it limits the agent from configuring other properties on that object.

Karan Munalingal • 36:16

Imagine Panorama pre-rule, right? If you gave it access to all attributes, it can go haywire and do what it wants to. So being able to limit that also now became an important part from a control, context control standpoint.

Holly Holcomb • 36:32

Love it. Well, I know I’m really excited to see this demo. I’m excited for us to get a sneak peek of what that looks like and what those product features look like. So, Karin, do you think you could take us through and show us what it looks like in the product?

Karan Munalingal • 36:47

Yes, I will give everyone a sneak peek into what the experience was as part of the innovation program, just to kind of double down on some of the lessons learned, right? I think the 1st one I do want to talk about is the speed to build, right? The time to value. How do customers see that actually happening? So, what you’re currently looking at is the Attention platforms. Everyone who has joined naturally knows what this looks like. One of the new things that you’ll see here on the left is Asian projects.

Karan Munalingal • 37:19

Right. When I get into the governance part of our discussion, I’ll talk about this as a way for you to create a curated agent development experience. So it’s not just global, like everyone can see, everyone can do. So this is almost like SDLC, but think about this as ADLC, agent development lifecycle. Holly comes in. She has an SME in a particular domain and she wants to build a brand new agent. She builds it in her own project.

Karan Munalingal • 37:48

And until that agent is ready and confirmed, she doesn’t have to share it with anybody. Right. So this is how we’re actually approaching the overall governance. But I digress. Before I go into that, this is agent projects. This is where you have the ability to build different types of agents, leveraging prompting, tooling, et cetera. What I will quickly talk about is one of the things that was mentioned within the innovation program is customers wanted to see what an agent would look like for their existing workflow.

Karan Munalingal • 38:20

I’ll just show you an example. This is a project, and I would say it’s a simple enough use case for DNS A record provisioning, right? I had to create this workflow. You’ll notice as part of this workflow, there’s a lot of details around queries and merges and evaluation. All of that actually resorts to business logic as well as data transformation coming from one API making decisions, going to the next API, et cetera. So, as I mentioned, 60 to 75% of this entire workflow is data manipulation and making business logic decisions, right? This same workflow.

Karan Munalingal • 39:01

So, right now, what this workflow does is it gets an A record, and this is a pre-check to see if it exists or not, right? If it doesn’t exist, it makes a decision, it’ll create it. Once it creates it, it actually does a post-check to make sure it actually did it, and then it will present that to an end user using a manual task. And if the user actually says, I’m going to deny it, it will delete that A record from the system, right? And it’ll notify somebody. Like, you’ll notice a mail with options, so it sends an email. So, that’s what the workflow does.

Karan Munalingal • 39:34

Those are the five steps. And this was a workflow that I had to build to achieve those five steps. Now, the customer said, Hey, what would this look like? And your flow agent and flow AI. So we bounce back to this. So within the agent project, what we have done is we have actually created an agent that does the exact same thing. The difference is instead of me dragging and dropping or writing Python scripts to support data transformation, I was clearly able to come in and provide instructions.

Karan Munalingal • 40:07

These are my prompts for this agent to say, perform a pre-check, then execute it if the pre-checks pass, then do a post-check, which is a validation step. And then if all works, present it to the end user. And if they confirm it, notify to everybody. If they deny it, delete the A record. Everything is written down here, right? So this is natural language where an SME who today actually defines requirements for somebody else to build the automation.

Holly Holcomb • 40:53

I think we may have lost Karin. Just give him a moment, see if his internet comes back. So, I guess while we’re waiting for Karin to come back, Manny, I saw that you had a question that you put out there. Does Flow AI use tokens only when generating the agent, or does it also use tokens each time the agent runs? This is a great question. I know that we did a previous session on our AI office hours that was all around spec-driven development and using some of our clawed skills in order to build all different types of assets inside of our portfolio, whether it’s a workflow or in some cases, it could be building an agent. So, I’ll give you the very simple answer.

Holly Holcomb • 41:41

Flow AI, when you build an agent, there is no token consumption associated with that build process. So, what Karin is showing right now with his DNSA record agent, it’s actually you can see at the very top, it says run agent. Once you hit run agent, then at that point, it’s going to start consuming tokens. The actual build process and like associating tools and what you’re going to allow that agent to do, that process is something that you configure inside of the agent project, and it does not require any consumption of tokens. So, hopefully, that’s helpful. Karen, I see you’re back. So, sorry to finish your state there.

Karan Munalingal • 42:18

My internet drops. But, yeah, I think I was talking about the prompt as the 1st part of context control that we did, right? We basically provided instruction. The same thing that I use to build that workflow. The 2nd piece you’ll notice comes to tooling. So, in this case, you know, if I were to edit this agent, I just quickly wanted to show all the tooling that, you know, me as an individual, as an SME, have access to. So, you’ll notice different types of tooling here.

Karan Munalingal • 42:47

So, I have access to. all the methods from different adapters that have been onboarded. If you guys are using integration models, if you have LCM customers, you have actions, transformation, existing workflows, services. So, if you have playbooks, scripts, all of those types of tooling are now available for me to associate with this particular agent. What I specifically did, though, you’ll notice I only gave this agent three InfoBlox APIs, right? Because I’m creating an ARECord InfoBlocks. So, I said, Hey, agent, go do this job.

Karan Munalingal • 43:20

And to do this job, I’m giving you access to create a record API, delete a record API, and get ARECords API. In addition to that, I gave it access to view data because I wanted human in the loop. So, as part of the overall governance, we’re also providing ability to insert human in the loop so someone can confirm before agent actually makes a change through one of these deterministic tooling. And finally, posting a chat message in Slack, right, from a notify perspective. So, that’s about it with respect to what. Folks went through when they created these sort of agents within the innovation program. They, it was very interesting because some of the folks said, Hey, this is very detailed.

Karan Munalingal • 44:04

What I’m going to do is I might just go to co-pilot and ask, hey, help me define a prompt for this problem that I’m trying to solve. And they copy-pasted that and we associated tools with it and then ran the agent, right? I think that was it. Like that was the exercise that we did in literally 160 minutes. So you can imagine going from this, where if an additional requirement came through to say, Hey, you know what? I know we’re only sending an email. Can you also now send a Slack message?

Karan Munalingal • 44:34

Or if the end user denies a request, also create an incident in ServiceNow because I want to log that, right? Imagine having to do that enhancement in a workflow that’s already in production. Now, as part of the development lifecycle, you would drag and drop ServiceNow APIs, you would drag and drop Slack APIs, et cetera. On the contrary, From a time to value standpoint, I can just come in here and type what I want the agent to do and then give it that one API that it’s going to do it with. Right. So that’s the big piece that customers saw is they knew what they’re doing today using workflows and everything else.

Karan Munalingal • 45:16

And you pivot back and they’re like, so you’re telling me that when this agent is run, I don’t have to worry about JSTs. I don’t have to worry about, you know, like business logic and decisions. And the answer is yes, right? That’s why it’s faster because the agent actually reasons through, understands schemas between your APIs and scripts and other things, and it will formulate that payload. And just to show you an example, I can go to agent session. And we ran one, right? So you’ll notice here, in this case, like this is an existing agent that was executed, right?

Karan Munalingal • 45:54

These are the core instruction, like we provided. It was provided a set of input on what A record to create. But you’ll notice here, it is starting to now figure out which tool to use, right? Once it figures out which tool to use, you’ll notice it automatically creates the payload for those tools as well. Right, this is where it starts getting very interesting. Where it says, Hey, this is a business logic statement, right? I just put that in words:

Karan Munalingal • 46:23

Hey, if it doesn’t exist, that’s one of your criteria. Please go ahead and proceed and create it. So I recognize it’s an empty array, which means it doesn’t exist. So I’m going to go ahead and create it. So when it actually called the create a record API, you’ll notice it formulated this payload here. I didn’t have to type it up, I didn’t have to do anything, right? This is where our customers started to see speed on how they can leverage reasoning to actually create a valuable automation and orchestration.

Karan Munalingal • 46:54

Right? That was the very 1st part of it, right? That’s them testing. Can I? If I built a workflow, can I convert that into an agent? And can I maintain it going forward? Right.

Karan Munalingal • 47:07

Now it’s an agent. Now they’re managing the agent based on works, not drag and drop. So that was the 1st part. And as Holly, you were mentioning, right? Like every time you run an agent, the tokens is kind of called out here. So you understand every time you run that agent that you just created. How many tokens is it burning?

Karan Munalingal • 47:25

Then you associate that with the value, right? So these were some of the things that we actually talked about during the innovation program. Because at some point, people will get excited because it’s so easy. They’ll build all these agents and someone will ask, hey, you’re running that agent a thousand times a month and it has accumulated millions of tokens and it’s doing the same thing again and again. Like, can we do something about it? Right. And I know Holly will talk about it later on during the presentation, the last two, but this was the 1st part of the demonstration.

Karan Munalingal • 47:56

The 2nd part I quickly want to highlight is around customers wanting to build a brand new use case by themselves, right? The 1st example is going from an existing workflow to an agent. The 2nd example, you’ll notice here under DNS management, as a DDI expert, they wanted to do a malware blocking agent. So what they did is they basically just went to Claude or Gemini and said, hey, can you write me the prompts to go do this? And they copy-pasted that here. And in order to provide the tooling, you’ll notice it only gave access to two tools. That’s it.

Karan Munalingal • 48:34

So, at this point, they now have an agent that can parse through information coming from an input. It will do the pre-check to say if it’s already blocked. Is it a critical website that should be blocked? It does way more than your initial mop or the intent that you had, right? So, this is how easy it was for them to start building agents. On average, in five sessions, and each session was close to 16 minutes or so, each one of our customers at least built five agents. With us, and in the background, they build three agents by themselves.

Karan Munalingal • 49:08

So it was, it was very compelling during readouts on how excited they were that they were able to contribute versus having to just write requirements and hand it off, right? So that’s the 1st two part of the demonstration. And the focus was how quickly can someone build an agent, whether to mirror your existing orchestration or crush your backlog, right? If you have a backlog, you know what you want to do, just come in here and type it in words and then iterate over and over again, right? It’s way faster, as you saw, just me showing you as an example, but now you guys can actually experience it too. The 2nd part of the demonstration is going to be focused on the overall governance, because that was a big deal. Like who can build?

Karan Munalingal • 49:52

What do they have access to? So as I’m showing you here, if I were to edit, I’m an admin on this particular platform. So you’ll notice from a source, each one of the adapters, every integration model, every gateway service, I have access to, right? So every one of the API I can use as a tool because I’m an admin. What we are enabling customers to do is control on the build side. So you’ll notice here I have the ability to actually create a group. So I created a DDI team group and I gave the pre-sales.

Karan Munalingal • 50:28

I actually have a user here, Attorney’s Pre-Sales, and I only gave it access to certain things. So when I was doing the demo, you guys are able to see every application, every agent project I have access to, right? But in this case, I’m literally limiting this user or this group access to Infoblox and ServiceNow. That’s it. That’s all they have access to. So when they come in to build an agent, they can only build agents based off of ServiceNow APIs and Infoblox APIs and nothing else, right? And what I will do just to kind of validate this, because every customer asked me to actually do that with them, is let me quickly share my screen again.

Karan Munalingal • 51:16

So, if everyone can see my screen, I’m going to quickly log in as the pre-sales user here. And I’ll just show you what that experience looks like because it is very important for a lot of our customers when something like an agentic AI capability comes into picture. They want to make sure that there’s governance and there’s a lot of control on what people are able to do in the network, right? So, in this case, I’m just going to go ahead and access this. If it lets me, so once we’re here, you’ll notice on the left-hand side immediately, I don’t have access to all applications, right? I only have access to Agent Projects Studio, right? Because I’m part of the DDI member.

Karan Munalingal • 52:01

My team was nice enough to share the workflows that they’re created as well as some of the Asian projects. So, when I go into Asian projects, you’ll notice just like you do today with studio projects, you will be able to share your agents with your team members outside of your team in a very controlled fashion, right? In this case, well, I do want to call out a specific thing. I created my own project, right? As I mentioned, Holly can come in, create her own project, and start creating agents within the project. When I get in here, when I click on a create new agent, you’ll notice this is what that experience looks like, right? You can bring your prompts, you can bring your description, but essentially, you also have to give this agent tooling so it can actually do the job in the infrastructure.

Karan Munalingal • 52:49

So, if I open this thing up, you’ll notice it’s very restricted. Me, as part of the DDI team, I can only do stuff with these five applications and two adapters and integration models, right? ServiceNow and InfoBlocks. Those are the only APIs that I can clearly work with. Right. So, this is where all the governance comes into picture on the build time, but also on the execution time when you expose this through something like operations manager, right? Holly can build it, but Holly cannot run that agent in production.

Karan Munalingal • 53:22

Only I can run it. So, this is why we are, you know, we’re very excited that Flow AI is essentially part of the platform because it allows us to take advantage of all the access control, all the integration capability that’s already there that customers are already using. So, Holly, that was the end of my demonstration, just to kind of hit on some of the lessons learned and what we actually improvised within the platform. Sorry, I think one thing that we missed on is the whole concept around decoration before we exit. One of the things that you’ll see here, and I mentioned, right, like I’ll look up create incident. As a tool. So let’s say I wanted to build an agent that involves creating an incident when something goes wrong.

Karan Munalingal • 54:10

When I add this and I go to decorate, what we now have the ability to do is, as a user, let’s say I create a brand new decorator, you will notice this is the native schema for that API. It just says body. This is what I was saying. If I naturally just gave this to the agent, the agent will start formulating a payload and say, hey, I tried to create an incident, but it didn’t work. Right. But when you enable customers to actually decorate it, in this example, we basically created a decoration with just the primary inputs: summary, short description, and the description. That’s it.

Karan Munalingal • 54:49

So now the agent knows, hey, when I call this tool, I must only provide these three. Or if I provide extra, it’s not going to fail, right? But I must provide these. So this is where we start enabling our customers. Instead of saying, hey, I’m going to wait until my vendor fixes the API. I will say, good luck, because get in line. Instead, you can enable yourself the ability to actually decorate what you want the agent to focus on, and it will do that, right?

Karan Munalingal • 55:19

So this is our way of also enforcing the agent to a compliant stack of capabilities it has, but also how it communicates with these APIs and services. So that was the full-on demonstration, Holly, to cover all of our lessons learned. Obviously, I love to demo. So at some point, if anyone wants to hit me up, I’m happy to jump on a call.

Holly Holcomb • 55:44

Yes. Thanks, Karin. All right. We’ve got just a little bit of time left, and there are a couple more lessons that we want to share with the audience. So we’ll touch really briefly on the role of subject matter experts before and after agentic operations. And I’d love to get your feedback. I know you’ve had a really great anecdote from the program of the type of roles that were engaged and who was able to build and contribute.

Karan Munalingal • 56:11

Awesome. Yeah, it was very interesting. Across the cohort, we had some DevOps folks that participated on the majority basis is the folks that are building workflows today. So network engineers, some operators. We had some executives that join because they wanted to try their hand at building agents. And what I’ll tell you, Holly, is what we learned as part of this is today, the way that organizations are operating is you have a set of folks that are in between business and the infrastructure that are translating the requirements and saying, hey, this is what I want you to go build, right? Hence the image on the left.

Karan Munalingal • 56:53

You have your SME that actually knows what needs to be done in the network, and they’re giving those requirements to a developer. They’re building it. It gets tested and you’re like, hey, that’s not exactly what I wanted. Go fix this piece. And the developer fixes it. And then it goes through the retest, push, and consume cycle. What we saw happening as part of the innovation program is these are the folks that were writing the requirements.

Karan Munalingal • 57:17

So they said, hey, if I’m writing the requirements, I’m just going to write that in a prompt format. And you tell me which tools to pick. Right. So that transition going from. Being able to write requirement and hand it off to somebody else. Now it’s the SMEs who are the developer of these agents and the manager of these agents, right? That’s the shift that we’re starting to see: these are the folks that know everything about the infrastructure, security, the network.

Karan Munalingal • 57:45

They can come and start building some of these agents with a lot of control and best practices in place. Right. So that’s a huge shift for us from a flow AI standpoint. I think we have an opportunity to work with a lot of our customers and enable them to bring their experts to the table and not be scared of, hey, well, I don’t have expertise in writing a script or even dragging and dropping and understanding APIs. I think the reasoning helps solve some of that problem. So we definitely saw that shift as part of that, Holly. Love it.

Holly Holcomb • 58:20

All right. Our last lesson learned that we’re going to cover is around, it’s kind of a combination of all the things. You touched on this a little bit during your demo, Karin. So I really want to hone in on it. We’ve talked about tool association and governance and decoration and the true power and value that comes with agentic operations and reasoning. I think that, you know, Manny kind of alluded to it earlier with his question about like, where is the token consumption happening and that type of thing? There’s a natural tension that comes with all the speed that reasoning provides.

Holly Holcomb • 58:57

And so I’d love to hear your feedback on what type of dialogue you guys had during the program around the balance between determinism and agentic reasoning.

Karan Munalingal • 59:07

Yeah, it was very interesting. What I would say is, coming, every one of these customers coming from a fully deterministic land, they see that at no cost, right? Other than humans actually building it. But then they saw how fast they could build an agent. So everyone got excited and started building a lot of agents very quickly. Naturally, when the agent has to think based on the data that it’s consuming, the systems that it’s interacting with, it will consume tokens from a reasoning standpoint, right? And we show that every time you execute an agent.

Karan Munalingal • 59:40

So, as quick as the executive built an agent, and you know, as part of the innovation program, she built an upgrade agent as well as a network hell check agent. And she did minimal prompting, minimal tooling, and said go. And she saw that token count go up to close to 150,000. So the question came up in the room to say, Hey, if I have to do a hell check and it costs me 150,000 tokens every single time, it’s static enough. What can I do within the platform? Right. And this is where the idea came into picture: is

Karan Munalingal • 01:00:18

Itential is the only platform that enables this balance between leveraging reasoning, but also full-on determinism. To us, APIs are deterministic, a script is deterministic, a workflow, which is an orchestrated path, is deterministic. But imagine being able to accelerate your backlog development using agents 1st . And you run those agents 100 times, thousand times, and you start to figure out that, hey, it’s consistently doing what I’m asking it to do with all the business logic and the error handling. But then you also take a look at, oh, every single time I run it, it’s taking 40,000 tokens. And then you do the math 40,000 times, 1,000 every month, right? So now with our agentic development, so spec-driven development capabilities, our Itential builder skills, these customers, some of these customers that went through the program actually converted their agent that they ran close to 50 times into a workflow.

Karan Munalingal • 01:01:23

So now you basically go from fully reasoned agent that did everything that you asked it to do. It also found out new things that you should do based on guidance and expertise. They were able to convert this agent using our builder skills into a workflow. And now they’re running those workflows. And it’s running faster in a sense because it’s not reasoning anymore. It’s already reasoned out. Now it’s following the same path every single time at no token cost, right?

Karan Munalingal • 01:01:53

And when customers saw that, like the jaw just dropped, Polly. I’m just going to say that is because they said, hold on a 2nd . So you’re telling me I can build an agent in 60 to 120 minutes, run that in a lab environment, validate it, and now I can convert that into a workflow and not have to worry about tokens. Like that is the shift towards the tail end of the program where people got really excited because they saw an opportunity to go faster, but not keep burning tokens forever. And that’s what you would do when you just go stand up your own stat because it’s all MCP driven, right? Like that’s, it’s all reasoned versus Itential enables you to go back and forth from, hey, I built a workflow. My environment changed.

Karan Munalingal • 01:02:40

Can I now build an agent to address that using reasoning and then go back to a workflow, right? This is where the Hyperloop comes into picture for customers to accelerate development against their infrastructure, but also take full advantage of our deterministic capability that they today do.

Holly Holcomb • 01:02:58

Love it. Yes. I love the fact that you used the hyperloop there. It’s exactly what I was thinking in my head as you were talking through it. Like speed of build time with agents, be able to test and validate quickly, and then be able to save on token costs afterwards because you’re able to turn it into a deterministic asset that saves you time and executes consistently, which some of our organizations really need. So All right, Karin, I know that I think we are just ever so slightly over on time.

Holly Holcomb • 01:03:24

I know we’re all surprised. And so I’m going to close it out for today for office hours. Please sign up for the next office hours. It’s August 13th. It’s at noon, same time, 1st Thursday of every month. So join us. We’re going to do a secondary session, and this one is going to be all about getting agents into production.

Holly Holcomb • 01:03:44

So thank you, everyone, for joining. And we will see you next month. Thanks, guys.

Karan Munalingal • 01:03:49

Thanks, Holly. Thanks, everybody. Take care. Thanks, Karan.