Holly Holcomb • 00:00
Agents. If you did miss it, it’s available on demand on our website. This session, we’re going to be talking about how you get those agents to production. So, Karan, welcome. I think that last time we celebrated your itentialversary, and this time we get to celebrate your birthday.
Karan Munalingal • 00:22
Oh, man. Happy birthday. Hey, Holly. Hey, everybody. I appreciate the wishes. Thanks, Holly.
Holly Holcomb • 00:29
And so I guess, you know, before we dig into the content and the agenda and what we’re going to cover today, as always, we love questions. We love engagement. If you have thoughts or questions or doubts, please, please, please feel free to put them in the chat. We love seeing them. We love knowing what you’re interested in. So please feel free to put all of those thoughts and questions inside of the chat. We’d love to see them.
Holly Holcomb • 00:56
So, Karan, we’ve got a lot to cover. during our AI office hours today. So I’m going to dig into the agenda. We’ll talk about purpose and goal. And of course, no AI office hours would be complete without an amazing demo at the end. So I’m super excited to see what you’ve got prepared for us today. As you guys know, last session was focused on building effective agents and some of the lessons learned that came out of the private beta that Karan helped lead with Itential.
Holly Holcomb • 01:28
So some of you out there may have recognized some of those lessons learned based on your own experiences. This session is going to be really focused on what it takes to get those agents to production. Because as many of you guys know, a cool prototype is amazing, but the kind of stringent requirements and compliance associated with getting something to production is a different story, different set of requirements. So we’re going to talk about where people begin. How they usually begin the process and how they iterate on what they build initially, and some of the pieces that Karan and team were able to see during that private beta, and how they fed into how we think about best practices with modularity and agent orchestration. We’re going to talk about tooling. I am very excited for us to talk about change management in governance.
Holly Holcomb • 02:20
That’s the 1st time that anybody said that in history, but I’m so excited for us to dig into what governance looks like and change management looks like in agentic operations. Then we’ll talk about production readiness, and last but definitely not least, the actual demo. Karan, you ready to dig in?
Karan Munalingal • 02:40
Yes, very much so. Let’s go.
Holly Holcomb • 02:42
All right. So as I know that we’ve talked about this a lot, building a prototype of an agent that is doing something interesting is just the 1st step. So Karan, I want to hear your feedback on all the things that go beyond the prototype. What are the pieces that you feel like you’ve heard customers say they need in order for it to get to production?
Karan Munalingal • 03:08
Excellent. So, yes, you know, going through the innovation program is very interesting. As I mentioned last time around, half of the customers had already started their journey around the agentic side of the world and on potentially doing some automation orchestration, but using agents. So they stood at their own stack and they were kind of walking through what they have done, you know, rather quickly using some of the available frameworks, et cetera. But a lot of focus initially when they’re showcasing and demonstrating were on, hey, look, like I have a framework. I created some prompts. It’s all NLP driven.
Karan Munalingal • 03:45
I have MCP servers. I have connectivity to LLM. And boom, I created an agent that does something. Like, you know, whether it’s a read-only agent or does something like, you know, something that actually does a interface change, you know, naturally they’re very proud of what it does. But then follows some of the questions on, hey, like, is your team and the executive team okay with rolling this thing out as is into production? And the immediate answer was no, right? And I asked, like, why is that?
Karan Munalingal • 04:15
It’s like, well, you know, naturally, because it’s agents and they can think for themselves, we now have to put way more guardrails than humans building the initial set of automations. Right. Like when the agent can think for itself, make tool selections the way that it wants to solve your problem. Because, you know, to think about it, when you were building a digital coworker and you give the coworker access to all the tooling in your house or elsewhere and say, hey, this is my goal, go build a house. They will try and build a house any way they know and can. But now you have potentially have a house that is super brittle without a door. So essentially, coming back to taking agentic AI for infrastructure, specifically into production, there was a huge, huge amount of focus on visibility, guardrails, access control, not only for the humans building the agent, but the agent itself taking the execution against the infrastructure.
Karan Munalingal • 05:24
How do we address if it fails? How do we address if it creates more than what I asked it to? Like a lot of that kind of boil down to, hey, we need some governance, we need some security, we need access control, we need a process in place. So when someone is building their, you know, Proactively building things that are compliant to that organization’s requirement, right? This is not a free-for-all, like, hey, you got access to the tool, go build the agent, and yay, it’s going to go into production. There’s a whole lot of scrutiny that goes around into what that agent is doing, what that agent has access to.
Karan Munalingal • 06:04
So, you know, things like even secret management, just like we do for automation today, an agent gets access to a system. And if it’s not rotated from a secret perspective, people get scared, right? So these are all the things that you have to be mindful of. And one of the biggest things that came out towards the tail end when all of the innovation program customers had kind of laid out is: hey, we believe that agents are great and we do want to proceed to help more people build the agents. But what we value more is a platform that has some of these things out of box. So we don’t have to go build all of that again in a different stack, right? I think one of their.
Karan Munalingal • 06:53
Main points is if we were to go invest into AI and have more people use AI for the purpose of building automation or solving problems within the infrastructure, the time to value have to be there. And consistency of what the agent does in production has to be there. Because if it works 90% of the time, it’s no good because they know an automation a bit, you know, if it’s static, it works 100% of the time. So, you know, people have to start thinking about beyond the surface layer where, you know, everyone is talking about context management and how you can do proper prompt engineering and what is the best LLM I’m going to choose. There’s a whole lot of stuff, the non-functional bits in the background that you have to consider and address before the governance body at your organization will allow you to run agents, especially for networking, security, compute, because it directly impacts how they operate today.
Holly Holcomb • 07:56
I love how you framed that. I know that we have, because there are so many new concepts that are coming into the market right now as it relates to agentic operations, I think a lot of our customers are asking similar questions to try and figure out the parallels and similarities of what we’ve done historically with automation orchestration and how that maps to agentic operations. And I love the fact that you called it out specifically. There’s the act of building the agent and making sure that it’s effective and it stays on track and accomplishes the goal and nothing more, nothing less. But it’s all of the other pieces that we’ve been talking about for years and years that our customers need to see in order to feel confident actually moving it to production. So I love the concept of it’s just the tip of the iceberg when we talk about what you need to see in production. There’s a lot more under the surface.
Karan Munalingal • 08:51
Yeah. And one and one thing that I missed on, Holly, is. Naturally, when folks start going down the agent route, it’s great because when it works in the lab, you’re like, hey, it worked. It’s time to roll it out and everyone to use it. As simple as a DNS change, it might take 30,000, 35,000 tokens. And then someone is sitting there in the background thinking, all right, we actually make DNS changes a thousand times a month. And I’m sitting here doing the calculation.
Karan Munalingal • 09:24
So now, you know, going into production, are you now just stuck with a fully agentic strategy? Or do you also have the ability to pull back and do deterministic work that doesn’t require reasoning? Right. So, this is where, you know, the customers that went through the innovation program, they started to realize that, oh, yeah, I didn’t even think about that because the stack that I built purely relies on, you know, agentic and LLM and reasoning and MCP to actually do the work. But because I do have my beloved workflows that I’ve already built within the platform, I can always retort back to learning from reasoning and actually creating a workflow without using any tokens because the purpose of reasoning is already served. It found out all the things that I never addressed. And now I’ve actually baked that into a deterministic orchestration, right?
Karan Munalingal • 10:16
So I think that this is where the whole concept of tokenomics is starting to come into picture. People are trying to look at where do I use the LLM where I don’t, because essentially it does drive cost at the end, right?
Holly Holcomb • 10:30
Absolutely. Yeah, I think it’s twofold. There’s a question of how do I make sure that I’m using most effectively the tools that I have today that I’ve invested in, that I know work, that are consistent in terms of execution every time. But also, how do I make the most of what I’ve got? The better, you know, the cohesive overarching strategy. But then, how am I making the decision around financial cost and when to use agentic operations and when to use determinism? I feel like that’s a really good point.
Holly Holcomb • 11:02
Well, I know that we’ve talked a little bit about all of the things underneath the surface, all of the other aspects that we have to keep in mind when we talk about getting something to production. I think the next question that we oftentimes hear customers talking about is when I 1st got started and I was trying to think about how I’m going to build an agent to do this task. I grabbed all of the steps in my mop. I grabbed all the data and all the tools and I shoved it into the agent and I said, go for it. And it sounded like based on our previous conversations that you guys had some pretty explicit lesson learned around best practices and how you structure these agents. And I’d love for us to chat a little bit about that.
Karan Munalingal • 11:48
Yes. You know, just like automation, I mean, just like scripting, when someone starts off doing some level of automation, they’re starting with a script that does one portion. And then they’re like, oh, well, I got a new set of requirements. I also needed to do this. And the script just grows constantly. Right. You come to a point where you’re like, well, now it’s the script has turned into an application.
Karan Munalingal • 12:14
That was not the intent. So the same concept going from script to someone even building your workflow, like our customers who are building, you know, solving complex problems, but I’ve seen where workflows are 70 TAS deep, 100 TAS deep in a single one. Right. And they started working with us on the best practices. And the thought process there is: if there’s, there are certain things that could be reusable, you chunk those, you create a modular version of that. And the same principle we learned as part of the actual implementation with some of our customers in the innovation program is it’s very easy for me to say, hey, like, I’m the expert in this field. I now have an opportunity to build an agent using Flow AI.
Karan Munalingal • 12:59
So I’m just going to flip on over to ChatGPT or Claude and ask, I’m trying to solve this problem. Give me the prompts, right? It’s very natural. So they basically just take that and bring the prompt in. And now you’re like, hey, well, I need to do explicit tool association because I don’t want the agent to have access to 100,000 tools or APIs or scripts. I want it to do certain things. So within the platform, they would end up choosing, you know, 28 tools to address the 15 steps that Claude gave them to solve an upgrade problem.
Karan Munalingal • 13:34
So in this particular case, you’ll notice like an upgrade agent. That we built for the very 1st time with a customer, where they just kind of clawed prompted, brought the thing in. It addressed all nine of these steps in that prompt. Like, hey, I want to recommend the OS identification. I want to figure out if the image exists, doesn’t exist, is there enough space? So every single step was well-prompted within a single prompt, right? And they gave it access to close to 28 tools and they ran it.
Karan Munalingal • 14:06
It did 70% of the work the 1st time. It improved and did 85% of the work. And then consistently is 90%, 92%. And in certain cases, it started going off the rails. So, you know, what we realized, and this was a known thing, just like a human, an agent can also hallucinate. And the reason why it could hallucinate, it has access to a lot of data and a lot of tooling, and you’re asking it to do a lot, right? So I understand people might say the agents work 24-7.
Karan Munalingal • 14:38
Yes, they do. But the effective ones do. There are certain agents that get tired because the instructions are too much and they have access to too many similar tools. So they’re trying to do their best to pick the right tool to solve your problem and your goal, right? So that is something that we learn not only for the upgrade agent. There are certain orchestration that we did in the data center cases around VLAN and stuff. When we started shoving a lot of prompts in there, a lot of tool, like in certain cases, based on the number of times we ran it, we saw a deviation.
Karan Munalingal • 15:11
So what we then did is we actually applied the principle of modularity at the agent level. So if you see on the right-hand side, Now we’re talking about a multi-agent system, right? It’s a modular set of specialized agents that do specific things trying to solve the same problem, which is software lifecycle in this case, right? So, you know, because of Mythos, everyone wants to understand what is the impact on their infrastructure, you know? So, as part of this next leg, we work with the customer to build specific agents that just do the CVE validation and suggestion to say, hey, based on the CVE and based on what I know about your devices, only these devices are impacted. So, just focus on that.
Karan Munalingal • 15:55
Now, you have done the device selection. The next thing is, hey, let’s go see if the device actually has space on it for me to move the available release. That’s another agent, right? So, you can start kind of going on the path where we ended up creating eight to nine different agents to serve its purpose. And half of those became reusable because now you can start using those in different parts of your orchestration strategy, right? Things like pre-check, post-check. Sometimes you have to do file transfer when you’re onboarding a new device because it has to be upgraded, right?
Karan Munalingal • 16:28
This is where you start getting scale on the value front is because you’re not rebuilding the same agents again and again with a minute little difference, right? So, this was one of the main lessons learned: there are going to be certain problems that customers will have to solve. And when they want to attempt to solve that using agents and orchestrations. We should think about like, is this activity happening in stages? Is it going to happen over more than two days? Right. It’s not a quick action.
Karan Munalingal • 16:59
It is a long-running action. So, how do I modularize these actions into specialized agents so that I can create almost like a system community, like an agent community to solve that one problem? Right? So, you know, that’s definitely one of the things that we learn. And that is very much in line with when someone goes from automation to orchestration. When you have bits and pieces like Lego blocks, just think about these as Lego agents that you can bring them together to actually solve a bigger problem and you’re not having to rewrite every single time.
Holly Holcomb • 17:37
I think my favorite analogy that you’ve used around this topic is around building a house. So the tooling that maybe a roofer is going to use is very different from the tooling that an electrician is going to use. You wouldn’t want to give the same, you don’t want to give a nail gun, e.g. , to an electrician to try and do their job. That’s a terrible decision.
Karan Munalingal • 17:59
That’s right.
Holly Holcomb • 18:00
And the idea of making sure that your agents have the right tooling so that they don’t make mistakes and use the wrong tool to try and accomplish the goal and have an adverse outcome. I think that has always landed. It’s landed with me. And I think that you’re touching on something here that is another trend that came out and the idea of having the kind of orchestration agent, or like if we were to take your house construction analogy, like maybe the project manager overseeing the house’s construction type thing. The idea of having an agent that’s going to go through and kind of make sure that all of the team is working together and effectively.
Karan Munalingal • 18:42
100%. That’s great. I forgot about that analogy. But yes, it’s, you know, you think about, you know, when you’re building a team, it can’t just have one A player, right? You want all A players that are very good at their craft. So the same thing applies to agents as well. When you build specialized agents that don’t hallucinate because they have very strict prompts and context and then only have access to the tools that they need to do the job, they will not get confused.
Karan Munalingal • 19:13
They will effectively do that every single time consistently. So now instead of having one agent that does something 90% of the time consistently, you have nine agents that do it 100% of the time, right? That’s where the beauty is, right? That’s how you scale because you can depend on that agent to always do the right thing consistently the way that you have asked it to.
Holly Holcomb • 19:38
Love it. Awesome. Well, I know that we’ve talked a little bit about modularity. I am very excited to see the demo that you’ve got for us and how this reflects on that. The next thing that I feel like we hear really frequently from customers is, especially when I’m getting started, what do I have to have? What are the baseline dependencies that I have to have in order to get this to production? And I’ve been getting a lot of questions from customers around MCP servers.
Holly Holcomb • 20:08
And if the expectation is that in order for me to have an agentic strategy, I’ve got to kind of retool all of what I’ve got deployed in my infrastructure today in order to account for this new tooling that exists. And there’s a lot of, I think there are a lot of thoughts and opinions about what is necessary to even get started. So what was one of the outcomes that you found as a part of the AI Innovation program in terms of what was necessary in order to get started?
Karan Munalingal • 20:40
Okay, you know, granted that we actually did it with a cohort of our existing customers who have already spent time integrating securely into their stacks, right? Whether that’s controllers, access to devices, secret management, SSO, right? So when Floy came in, we like there was no requirement from us to say, hey, go stand up something new so I can talk to that system, right? So this is where I think. For all of our existing customers and new ones, you know, they’re looking at Itential. I talked about it last time. People have done automation and orchestration for such a long time now.
Karan Munalingal • 21:21
And within their architecture review board, within their governance board, like APIs are a thing, right? It’s certified. People are using it for such a long time, and people have built so much on top of it. And our mentality is always, you know, let’s use what you already have and then build on top. So the same thing applies within the AI space where Flow AI naturally leverages existing integrations, existing workflows, existing scripts and playbooks that folks have already written and are using today as part of their orchestration strategy. Guess what? Each one of those things are now tools for an agent that you’ll build.
Karan Munalingal • 22:01
So getting started does not require you to bring additional infrastructure and additional protocols. But as mentioned here, right? The community and every vendor that is wanting to participate in the AI ecosystem will roll out MCP servers with very predefined tools that are really good, right? I’m not talking about, hey, I have an MCP server that’s a translation of my current API, right? So the questions that customer ask is like, hey, if this is a translation of their API one-to-one, why would I not just use what I already have in C? Because I don’t want to go stand up another MCP server, right? So this is where it kind of stems from: they just want a harness where they can start doing work a different way than having to do more work just to improvise AI in their environment.
Karan Munalingal • 22:54
So, and naturally, you know, in their head, when they kind of spoke about it, it’s like the more I add, the more I have to govern. Right. So now, if I, in addition, I already have Itential talking to this system using APIs. Like, do I now have to dupe and also put an MCP server and switch over? And, you know, these are the kind of questions that folks who have already put in the work will ask. And that’s why I’m saying it’s great as vendors roll out very effective MCP servers to the community that have abstracted tooling that does a lot more than just call a single API. That’s perfect because guess what?
Karan Munalingal • 23:35
Within the Itential framework in our ecosystem, they can also bring the MCP server along and we will federate those specialized MCP tools as part of our Flow AI harness. So now you get the best of both worlds. You get started immediately today, now, because you guys are already automating, but now you guys get to use Flow AI using the same tools, the same connectivity, the same security, the same governance, the same access control. And then you can start adding some of the newer things like MCP server because the vendors are providing a better version of it.
Holly Holcomb • 24:14
Love it. Just to ask the question, I know we talk about this a lot when it comes to APIs. Not all APIs are created equal. Some are better than others. Does the same notion apply to MCP servers?
Karan Munalingal • 24:30
Yes, 100%. As we were experimenting with certain vendors, working with customers and ourselves internally, we quickly realized that certain ones have done a lot more work into actually curating certain tools within their MCP server that are very useful. So I don’t have to sit some of the APIs together. It’s already there and it’s certified. But then, you know, some who are still working on their MCP and AI strategy are right now, their initial version of MCP server is a mere reflection of what their API does, right? So this is where the balance comes in. From our perspective, we’re not tied to either strategy.
Karan Munalingal • 25:10
So naturally, customers who already know APIs, they’re leveraging APIs right at the gate because that’s what they are currently doing as part of their automation orchestration strategy. And they’re now playing the waiting game to say, hey, I’ll just wait until my vendor rolls out a better MCP server later on in tooling. And I can also start using the combination of it.
Holly Holcomb • 25:32
Love it. Yeah, I know that that’s one that is one question that we’ve heard over and over again: in order for me to move forward, do I need to go ahead and deploy all these MCP servers for all the vendors that are releasing them right now? And I love the comment that you made that in some cases, the MCP service just merely wrapping the endpoints that already are exposed as a part of their APIs. And you’ve already got those in existence on your platform. Why spend time? Why deploy a new system that you now have to administer, that you have to manage rather than just getting started today with what you’ve got? So it’s a great point.
Holly Holcomb • 26:14
The next topic that we’ve got is one that is near and dear to my heart. It is change management alignment and governance. And I know that, you know, there’s, I think there are multiple pieces to this topic. Change management has been kind of the through line, I think, over the years, for like the past 10 to 12 years. Anytime we talk about automation and orchestration specifically, you can’t talk about it without talking about change management. You can’t talk about it without saying, in order for me to make this change on my infrastructure, I need to make sure a ticket is updated, that there’s a ticket that exists, that it knows what it is that I’m about to do on my infrastructure, that we capture pre- and post-checks. There isn’t a world in which we can do these types of things on our infrastructure without change management overseeing the entire process.
Holly Holcomb • 27:05
And I think that the secondary piece of it is around how we’re tracking the changes that happen outside of the ticketing system, the audit trail that exists. So I would love to hear any insights that you guys had that came out of the AI innovation program as it relates to the, you know, do you see people changing the requirements as it relates to change management for agentic operations? Is it the same? Is there any change in expectation for tracking activities on the infrastructure through ticket systems?
Karan Munalingal • 27:39
Yeah, what I’ll tell you is, I know you referenced that the journey that we’re on with improvising and introducing agentic operations for infrastructure takes us back seven, eight years when people actually started doing automation. And you know, some of our customers who started building their 1st use cases, set of use cases in orchestration, they had a lot of those manual task review boxes in here, right? To them, they’re basically just providing assurance to humans that the automation is not going to go crazy. So now, in the world of AI, everyone already knows that the agent can think for itself beyond what the human told it to, because it has access to knowledge base and tools, et cetera. So I think based on our interaction with a lot of leaders, you know, both on the infrastructure side as well as the security, like the CISO team, they are very much focused on creating a more robust change management strategy. So it’s not overbearing on the humans, but leveraging the visibility as well as auditing that agents and automations can do, you know, by themselves. So there’s a trail that is left to say, hey, that’s not what I expected, especially in production, right?
Karan Munalingal • 29:01
In labs, you’re working with a small set of networks and a controller that nobody, not a lot of people are using. So you can do whatever you want. But as soon as you go into production, every change has an impact, right? So I think there’s a lot of scrutiny around. Hey, how do we introduce human in the loop? Especially now that we’re starting with agents, we would rather have more human in the loop. And then, as we trust the agent or the multi-agent system, we can start removing the humans and actively reviewing every change that is proposing versus only when it falls out.
Karan Munalingal • 29:40
The same paradigm, the same dynamic that we had for automation when people got started with automation orchestration is coming back to agents, but a little bit more robust because people actually want to make sure. I mean, they’re very excited that agents will make their life easier. Now they can build a digital coworker, but it’s going to take some time to trust your digital coworker because you’re basically giving it, hey, can you go do that? I’m going to give you these tools to go do the work, but you haven’t done it this way. Pause, show me what you’re about to go do. You know, the same concept. So I would say it’s going to be more robust around change management.
Karan Munalingal • 30:22
They’re probably going to put a little bit more intelligence where one agent is proposing, another agent is going to validate if it’s the right change based on internal policies, et cetera. So those were the conversations that came in, which were great because I know now people are actively thinking about how to make this thing work instead of saying, nah, I don’t trust the agents. I’m going to walk away because it’s just too much risk. I didn’t hear that. I think our customers are looking forward to improvising agents as part of their operational strategy, which was great. And the 2nd thing here, Holly, was governance. It’s not governance on humans who are building, it’s governance on what the agent can and cannot do.
Karan Munalingal • 31:03
So it stems from what are you going to allow the human to build? What are you going to allow the agent to run? And how are you going to make sure that the agent is consistently doing what you’re telling it to do? So this is where a whole lot of focus on non-functional capability comes into picture because it’s not a feature. This is just must-have documentation and proof. Just so something goes wrong, you know exactly how to go address it.
Holly Holcomb • 31:33
I know that in our world, in order for you to get something to production, whether it’s an orchestration and automation, whether it’s an agent, the kind of process around the change is just as important as the change many times. So there’s no world in which you’re making these changes without a ticket being updated, without there being some visibility and audit trail, both in your ITSM as well as in the tool that’s making the changes. Those are hard requirements. And so when I think about the types of things that really are necessity for us to get these things to production, change management is at the core of it. Governance is at the core of it. It’s making sure that if something happens that you weren’t expecting, that you’ve got an audit trail around it. You understand what happened.
Holly Holcomb • 32:28
And also that, you know, when the changes are made, that they’re made not just within the tool, but they’re tracked in your ticketing system as well. Hard requirement for the security and compliance teams out there.
Karan Munalingal • 32:40
Very much so. Very much so.
Holly Holcomb • 32:43
Awesome. Well, we’ve talked a lot about audit trail, which kind of leads me to the next topic that we’ve got in terms of production readiness. There are so many things that go into it. You know, we’ve had a few customers that are also working on their agentic strategy where they’re building from the ground up agents and trying to kind of do the DIY approach. And it’s really fascinating to see some of the things that they’re running into along the way. So, you know, again, when we talk about prototyping an agent, like being able to build something and let it do its thing and watch it run and that type of thing is really incredible. And then the next step is like, how do I start exposing this to other people?
Holly Holcomb • 33:24
Like, how do other people start accessing? It sounds like very familiar and reminiscent of the conversations we’ve had over the years. Like, well, if I want to take this and offer it to somebody else, you know, how will they leverage it? How do I make sure it’s the right people leveraging it? All of those types of production readiness conversations that are like, You know, starting with a tool and then starting to expand who can access it, who can use it, how it works, where it works. So, Karan, when you think about some of the things that came out of that AI innovation program and the private beta and that kind of centered around how you got things to production, were there pieces that you felt like stuck out more so than others in terms of what was required and that type of thing?
Karan Munalingal • 34:10
Yeah, I mean, naturally, we talked about in the last office hours, a whole lot of focus was on the building part of this agent, right? Which is great because now you have harness, you have control, you’re building stuff. But it gets more interesting because you can build whatever you want. If it’s not being used, it’s useless, right? So, talking to some of the executives, you know, as part of the program and elsewhere, even during our advisory board, what came up is just like automation, the value is driven based on adoption and usage. So, let’s talk about what it is going to take for someone to trust and request. This and knowing that an agent is going to solve the problem and not a human or an automation, right?
Karan Munalingal • 35:00
And this, what this surfaces is, you know, the very 1st slide that you showed around the iceberg, right? The initial focus in prototyping, even in lab, it’s very much so on. I’m going to prove it. I just proved it. Let’s push it in production. And then someone says, halt. And they said, all right, before this thing goes into production, because the real impact is someone is going to invoke it a thousand times, 10,000 times, right?
Karan Munalingal • 35:27
So what is the impact there? So working backwards from there, the non-functional things about your architecture becomes a big deal, right? As they say, it’s. People continue to reference 95% of AI projects fail is because no one thought about it from an architecture standpoint and actually put a lot of focus on non-functional requirements that have to be met for the agents to continue succeeding and performing in production because that’s the thing. Production is a real thing, right? So you’ll notice like the things that they highlighted is I need visibility, number one, because I already don’t have high level of trust on agents, right? So I need every single thing that it touches, does how it does.
Karan Munalingal • 36:18
What does it have access to, right? Data integrations. Is it doing it securely? You know, is my data going to the LLM encrypted, not encrypted? How do I control when the agent stops? Because I see it doing bad stuff. I want it to be able to stop.
Karan Munalingal • 36:37
Like what kind of guardrails and controls do humans have? To make sure the agents will not do something bad, or they start to, and they can just yank the chain, right? So, all of these non-functional things around being able to leverage their existing secrets management strategy, access into the device that are restricted, all of that becomes a big deal. And finally, on the exposure front, again, let’s say we address all that stuff and now it’s time to expose. Well, I only want to expose this capability to certain team members, right? It’s not open for all. So, how do I do that in my stack?
Karan Munalingal • 37:16
How do I make sure when this request comes in from this team, I reject it versus the one coming from the other one, I honor and I provide notifications and guidance back, right? So, When we’re talking about production readiness, it goes all the way. It stems from not only building, being able to test, right? Being able to promote it across stages, being able to retest, rebuild. And then finally, once it’s in production, it has to work at scale, right? This is not a one-time thing because agents can do things more intelligently and faster.
Karan Munalingal • 37:53
No one is going to say, hey, I’m only going to run that agent 10 times. They’re going to want to run it thousands of times because it’s driving a lot of value. So when it gets to up to that point, like people have to start thinking about: do I have the architecture to support security scale with a lot of governance and visibility? Like that’s what production readiness means and requires, especially for every one of our customers and future prospects and customers looking to improvise. An agentic strategy for infrastructure.
Holly Holcomb • 38:27
Yeah, it’s the engine behind the scenes that’s just as important, especially when it comes to production readiness. Like, does it have the scale and ability to go through and do what you did on one device in a lab somewhere on production-grade scale, which is, you know, there’s a lot that goes into that security, like you mentioned. Some of it is just the power behind it, the ability to go through. And I know one thing that we see a lot with our customers is that, you know, they’ve already set up this infrastructure to go and do these changes across their network. They’ve already set up the platform and the gateways to go into the different pockets of their network based on how things are segmented. Why not leverage the same guardrails, the same infrastructure, and the same power that you’re using today with determinism to be able to support your agentic strategy? I think we talk about that a lot with our customers right now, because especially as, like I said before, there’s a lot of ambiguity with new concepts, new naming, that type of thing.
Holly Holcomb • 39:30
But at the end of the day, everybody’s looking for how they map things back to scale and how they map things back to the kinds of compliance requirements that they’ve got. I think one thing I really appreciate about what you just said, Karan, is you said, and it’s hard to say it out loud because we all love tinkering. I love tinkering myself, but like the act of building itself is not inherently producing value. It’s valuable for the individual building, don’t get me wrong. But the act of building is just step one. The act of consumption, the act of running it, the outcomes, that is the value. I was having a conversation with somebody recently and they said, you know, is the goal just to build agents though?
Holly Holcomb • 40:15
And we like paused the conversation. And somebody said, the goal is what it’s always been. The goal is to solve complex and comprehensive problems across your infrastructure. How you achieve it is, you know, we think that this is a really effective way to go about it, but the how, what you’re solving versus how are two very separate things. So I love the point you made about, you know, the consumption and usage is where you really see value being driven. So I think it’s a really valid point. To get tactical for a minute, there’s also the piece of how do I get it from dev to staging to production?
Holly Holcomb • 40:53
Like, what’s, you know, what does that even look like?
Karan Munalingal • 40:56
Yeah, it’s especially, you know, with a lot of our customers who are building so much within the platform and now have an established strategy around, you know, doing so for their automations and orchestrations within the attention stack. But that’s the main thing, right? Like when you build, now you need to make sure that you also have a productionized version of what you just built, right? So it does go through, you know, you have to understand what is your CI/CD strategy? How are you going to make sure that it scales? Because What we saw during the innovation program is because it was easy for SMEs to come in, they got very excited and started building a lot of agents in the matter of 60 minutes to 120 minutes.
Karan Munalingal • 41:43
So you can think about, you know, going from writing a script to like building a workflow that takes a little bit of time. And now someone just comes along and says, I’m just going to write what I actually do day to day in associate tools and I watch it run and I’m ready, right? Let’s say they are ready. How do you not waste time in getting something that’s functional, like a functioning agent that’s actually going to drive value into production? So this leads back to, you know, architecturally. Whatever, if it’s a DIY strategy, you’re using Itential, like, how are you using your existing pipelines? How are you taking advantage of the concepts around projects that we have introduced?
Karan Munalingal • 42:23
So now it’s almost like a container for doing your work, building your agents, your agent systems, and now migrating that along the way, right? Our goal always has been you build fast and you promote fast and you get value fast. So, with respect to customers, they are excited to build, but one of the things that they also understand is if I build and test fast, I also want to have a good strategy that it makes into production because it doesn’t just end the very 1st time you launch your 1st agent in production. That agent is going to have more requirements and more work, more enhancement. So, how do you close the loop on additional development of that agent, enhancement of the agent? And it should not take weeks, months for it to make it back into production from dev. So, that’s definitely one of the requirements from a time to value standpoint that every customer had brought up.
Karan Munalingal • 43:17
It’s like, hey, if I build it, how fast can I take it into production? How safe, how safely can I run it in production and at what scale? So, thanks for bringing that up, Polly. But, yes, it’s top of mind for a lot of our teams.
Holly Holcomb • 43:32
Awesome. Well, I want to make sure that we get to a demo. We’ve covered a lot in the last 40 minutes or so. We’ve talked about how it goes beyond just prototyping once you get something into production. There’s more that’s underneath the surface that we have to be mindful of. We’ve talked about context overloading, and I’m very excited to see the demo that you’re going to showcase today about the before and after with context overloading. We’ve talked about making sure that we’re really prescriptive on the tooling that we provide and that the tooling itself is not just MCP dependent.
Holly Holcomb • 44:12
You can use what you’ve got today to get started and keep moving. Change management, not just in terms of like ticketing, but also in auditability and traceability of what’s being executed across your infrastructure. And then Production readiness is not just a single feature. It’s all the things that we just talked about. It’s, you know, the process of deploying, the process of iterating. It’s all of the pieces around making sure that it aligns with your security posture as well.
Holly Holcomb • 44:41
So, Karan, I know that’s a lot. I’m so excited for you to show us a demo that shows some of these things like in practice.
Karan Munalingal • 44:49
Awesome. It’s a challenge. I’m going to try and do a demonstration in 10 minutes or less, but I’m going to make sure that I address all the things that we talked about today. So, where we’re going to start with respect to just the show and tell here, essentially, is, you know, the very 1st thing that Holly and I covered around just around best practices that we learn that matter going into production is essentially building an agent that does everything or a community of agents that solve the same problem more effectively, right? So, just to kind of show you an example, this is agent projects. I will talk a little bit about this around governance on how different teams can come within the platform like they are today and building orchestration. They will do the same thing with agent projects.
Karan Munalingal • 45:39
Right. The decision that we took here is now Holly, who might be an expert in Meraki, can come in and build agents just on top of her technology, which is Meraki. And that now becomes a reusable asset for the rest of the organization. So, similar to that, when we got started with the innovation program, we’re working with customers, and naturally they kind of pivoted to clot code and said, Hey, I’m trying to do a Cisco upgrade, build me an abstracted agent that will work on multiple vendors. So, let’s look at, I’ve just clicked into this particular agent and I called it all-in-one agent just to represent the context that was given to this agent. Like, think about this as my digital coworker, and I’m telling this coworker, Hey, I want you to help me with all these things. So, you’ll notice down below, there’s this workflow that has eight steps, right?
Holly Holcomb • 46:28
Step one, I’m seeing the little scroll bar over to the side here, it’s like very small compared to what we’re seeing.
Karan Munalingal • 46:35
So, you’ll notice, right? Like, hey, go do like the upgrade baseline and then upgrade readiness, stage the image, execute the upgrade, post-grade validation. You know, this is all written in words, which is great. But as it’s written in words, you’ll notice we’re also asking it to do some pre-checks and post-check and fallouts, right? So you can think about the more you ask this co-worker to do, you also have to tell it what not to do. And when you see something, stop. Like if the wire is hanging out, don’t continue touching the wire.
Karan Munalingal • 47:10
You know, you see what I’m saying? Like, so this is where you start, you know, people get excited and they’ll build this and it will work 90% of the time, right? Like when we ran it a few times, it continuously kept on improving because it started learning. It’s like, oh, yeah, this is a tool. This is a payload naturally. And then you’ll notice now you’re asking it to do all that work, like eight-step workflow, fairly complicated. And then you have given it access to 14 tools, right?
Karan Munalingal • 47:38
All this stuff. It’s like, hey, I’m going to give you access to all these 14 things, and then you go figure it out by yourself. Right. That’s the notion. And what we learn as part of this experiment is this is great, but now think about potentially breaking down what you want this agent to do. Like, let’s say those are eight steps or nine steps. Now, what we did is we actually created a different agent project, which is a multi-agent system.
Karan Munalingal • 48:08
And we broke down that agent’s work into eight different things. Like, you’re going to do pre-checks, you’re going to do post-checks, you’re going to do assessment, you’re going to register the CVE, you’re going to be responsible for staging the image, you’re going to upgrade the agent, right? So, one of the things like, Because of Mythos, a lot of our customers, and Holly, you’ve been part of some of these conversations, they don’t want to just say, oh, Mythos says, go upgrade. So I’m going to upgrade 10,000 devices. The 1st question they ask is, does this CV actually impact any part of my infrastructure? And the answer is yes.
Karan Munalingal • 48:44
Which part? So I can start organizing work, right? So instead of shoving this agent’s context, which is this, you’ll notice we’re just asking this agent to be a specialized agent in doing these three things, right? Go assess against the inventory, learn about the CV, and then tell me how does this impact me? And you’ll notice it only has access to three tools. Based on our experience, this agent is going to be very effective and has worked 100% of the time doing its job that I’m asking it to. And I can run it every single day instead of running that one agent, which is an upgrade agent with a lot of contacts every single day.
Karan Munalingal • 49:24
I don’t want to do that, right? If I’m not upgrading a device, but this I can run every single day just to start creating a backlog of devices I want to upgrade. Same thing, like doing pre-checks, like, hey, very simple, because I wanted to address multi-vendor. I’m just basically saying, go do this, and it has access to Ford tools, right? So you think about how our customers are now empowered, very similar to how they’re building their orchestrations today with child jobs and modular flows. They would now do the same thing in the agentic world where they have a big problem to solve. It’s okay.
Karan Munalingal • 50:03
You can get all the prompts, but now figure out: can I now create reusable agents within the platform that are specialized agents and have all of these work together to now upgrade my fleet? Right, because you need to do the CVE so it figures out which device to upgrade. Then the next stage is that this agent kicks off and says, Oh, this device doesn’t have enough space on it, so I’m going to take care of it and then put the new release on. Then you have the pre-check agent, then comes the upgrade execution agent. So if you go to the upgrade execution agent, basically just says run the upgrade commands. Very simple, straightforward. Right.
Karan Munalingal • 50:42
So you can see how we went from a single superstar to a team of superstars that are going to effectively work non-stop to make sure that your goal is met. And this was one of the things that I felt was great because it applies to their current operating model, where now they’re like, hey, I want to solve this agentically, but let me not just do build one agent like I used to build one workflow with 100 tasks. I’m actually going to create something that’s modular and it becomes reusable. So you can think about how the upgrade agent by itself is reusable during onboarding. You can think about the CVE thing that could be scheduled every single day. You know, you can do an agent that actually brings on the device and adds it to records. Like these are some of the things that customers are trying to actively solve today.
Karan Munalingal • 51:37
So that’s the 1st part of the demonstration. It’s just going to highlight how as A developer as a consumer in production, I can effectively have the agents work 100% of the time. That’s the goal. 90 is not acceptable for a lot of folks, right? If it’s an agent, it has to work 100% of the time. So, this is how we would solve that.
Karan Munalingal • 52:00
The 2nd piece I do want to outline, and what I’m going to actually do is go back to one of the agents that I love showcasing, which is a DNS agent. In this particular case, right, I have an agent, simple steps. It’s doing pre-check. This is very focused on creating a record and notifying the team, right? That’s all it’s doing. But essentially, you’ll notice there’s a concept of tooling that we talked about. So, what Holly and I mentioned is in our case, we’re connected into various different technologies in our own domains.
Karan Munalingal • 52:36
I have essentially access to, so if I were to edit this agent, the whole thought process that our existing customers, if they have adapters, integrations, compliance plans, gateway services, if they’re using LCM and they have actions, et cetera, all of those immediately are now available as tool types to associate, explicitly associate with your agents. Right. It’s about getting started with what you already have and know works. Immediately, you can transition from, hey, I’ve been building workflows to I can now build a combination of agents and workflows with our, what I already have. On the other side, if you have an MCP server, that gets associated through our gateway. And now we can also consume your MCP tools as you plan to bring them along when your vendor gives you good ones. So this is where, you know, not only for context control, because the more tools you give, the more the agent has to think.
Karan Munalingal • 53:39
Now you can say, do I need to build a new tool? Or my vendor already has an API that I can use to do that job. So now you’re starting to think about what is the most effective way the agent is going to solve my problem instead of, oh man, like for the agent to do something, now I have to go build stuff for the agent to use. So, Holly, like we have a lot of our customers who have invested a lot of money in various different vendors, and they’re providing API specs, MCP server. Imagine just being able to load that up and everything is available here. Right. And that’s the key part: let’s keep, let’s make sure that we’re able to use what’s already in place.
Karan Munalingal • 54:20
And then we’ll make it very easy for you to bring some of the new set of capabilities on.
Holly Holcomb • 54:26
Absolutely.
Karan Munalingal • 54:27
So that was the focus on the tooling piece. What I wanted to showcase, right? Going into production, this is the 1st thing people ask when you run the agent. So, what I’m actually going to do is go to operations manager because this is where the consumption parts come into picture: value, right? Value is associated with usage of what you just built. So, just like you have a lot of our customers that built automation orchestration and expose it via APIs or what have you, I think one of the things, even for DIY, like someone has to figure this thing out. How do I make sure I specifically expose certain capabilities?
Karan Munalingal • 55:04
Like for this, in this example, my agent via an API to a particular team and via form to another team and via ServiceNow request to another team, right? The ways that people consume in your organization are going to be different. So, we want to make sure that we’re making sure that we’re complying to that. So, what I’m just going to do is, I’m just going to quickly, this is going to be. That’s it, right? It’s a form because I want to fill it out this way. I could have done the same thing with an API.
Karan Munalingal • 55:39
I’m going to run it. The reason I want to run it so I can show with respect to visibility that we would provide to our customer. And this is a requirement, right, from the CISO organization. We want to know what the agent is thinking about, which tool it’s using to do certain work. So you’ll notice in this case, every single step is captured and outlined here within our so-called agent session application, right? Customers have access to this at all times. So now you can actually understand all the reasoning steps it’s taking, all the tool calls it’s making.
Karan Munalingal • 56:14
And more importantly, if you go to one of the decisions that it took around calling a tool, so for the get a record. We’re basically outlining what is the payload that the agent created to support that, right? Because it can also do data transformation, but you have visibility to what it’s sent to the downstream system through our platform. The agents do not directly talk to devices or the controller, it happens through our governed platform. So, Itential platform is essentially still talking via an adapter, integration model, a script. And you will see all the responses coming back, right? The 2nd thing is someone might say, Hey, I want because in my agent, I had a review step, which is a human in the loop, you’ll notice it’s paused.
Karan Munalingal • 57:01
It’s not going to move forward unless someone reviews. So, this is a governance and the change management holly. Naturally, I could have done this in ServiceNow with a few words in my prompt to say, Create a ticket, wait until someone approves in ServiceNow and move forward. But within Itential, we have Work Center now, right? So, now I can actually expose this particular review task to a human who says, Oh, this is what the agent is trying to do. And I’m either going to approve or deny, right? It’s all in place where now you can put in as many human in the loop tasks and steps that you want.
Karan Munalingal • 57:36
And at this point, I’m just going to deny. And when we go back, it’s actually going to listen to and look at my decision and it’s going to undo what it actually did in InfoBlocks, right? This is where when it goes into production, you’re basically giving the power back to humans so they can make decisions on, hey, do I approve this? Do I not approve it? And this is how we’re going to help build confidence so people can actually run these agents to get a lot of value. And I know we’re 2 min away. The final piece I want to call out, Holly, is the non-functional bits, right?
Karan Munalingal • 58:12
If you look at visibility, we just talked about it. Human in the loop, we just talked about it. Things around access control, right? All of that is actually baked in. Things like service accounts. How do you make sure that certain people can consume things, build things? All of that has to be catered for, because when this thing goes into production, it can’t be just like a tool and a laptop that someone is using, right?
Karan Munalingal • 58:38
It has to be production grade. And finally, analytics is a big thing. Every one of our customers that are not only building automations and orchestrations and agents now want to understand what value it’s driving. Is it saving me time? Is it making me more productive? Am I doing more with less? These are the answers that they can get from insights here, both on the automations that they’re running as well as the agents.
Karan Munalingal • 59:05
So just wanted to call this thing out. And that was, I know, it was a very short demonstration. I love doing more demos, but hopefully that covers what we just talked about today, Holly, around what it actually takes to go into production with agentic AI in general.
Holly Holcomb • 59:24
Yeah, that was awesome. And kudos, that was a very concise to the point demo. We covered a lot in this session. I’m very impressed you managed to cover it all in the demo itself. So, what’s next? If you are interested in talking more one-on-one, you want to have a more specific conversation around what your agentic strategy looks like, what you guys have tinkered with today, where you’d like to go, what your goals and strategy are for getting there, please, please, please reach out to us. We would love to talk more about this topic.
Holly Holcomb • 01:00:01
Please reach out to your customer success manager. Let them know that you want to have a deep dive on this and talk about maybe some challenges that you’ve encountered, whether it is on like trying to understand what the goal should be or where to begin. We would love to have those conversations. Karan, as always, thanks so much for all of the insight. It was wonderful getting to chat through some of the lessons learned that you guys got from the AI Innovation Program and the private beta. Thank you so much for joining.
Karan Munalingal • 01:00:30
Of course. Thanks for having me, Holly. Thanks. Appreciate it.
