(So you can skip ahead, if you want.)

00:00 Introduction and Series Overview
02:08 Beyond Cost Savings ROI
04:15 ROI Metrics Framework
06:37 Efficiency vs Enablement Metrics
11:08 North Star Value Drivers
15:32 Culture and Orchestration Impact
20:12 Baseline Data Capture
25:15 Infrastructure Complexity Assessment
27:09 Sustainable ROI Discipline
31:05 Wrap Up & Resources

William Collins • 00:10

Welcome everyone on this. Let’s get down to business. So in the 1st webinar in this series, we covered uh the the invisible issues that kind of kill automation programs before they kind of get off the ground. And then we did a follow-up. We did a 2nd one where we dove into um configuration consistency. And then in the last one we did, we talked about actually building a program um that enabled just diverse contributions across teams, you know, in these multi vendor environments that we see every day. And today we’re really uh we’re bringing it all together by talking about how you actually measure and demonstrate the value of all this work.

Holly Holcomb • 00:29

Also frigid. Uh we actually got snow this week in Atlanta, which like is very unusual for Atlanta this time of year. I’m used to 70 degree days, so it’s way too cold.

William Collins • 00:42

That is unusual. Yeah, we have like ice crystals floating down from the sky. It’s cold. I had to let the car warm up this morning, all the things that I uh yeah, didn’t miss doing during spring and summer. So yeah. So in kind of I guess continued funding and you know, just any sort of momentum for your program. So I think in the original blog,

William Collins • 01:48

And, you know, because at the end of the day, if you can’t show the business value, if you can’t demonstrate ROI, you’re going to have a very hard time getting continued support, uh Frigid, frigid Wednesday in November. I’m William, and I work as director of tech evangelism at Itential. With me is Holly Holcomb, Director of Customer Success. How are you doing on this fine Wednesday, Holly?

William Collins • 02:08

that you you authored that kind of kicked off this whole series, you know, that inspired the whole the whole thing. Uh you kind of argued that network automation ROI goes far beyond just cost savings. And I think that is such an important point. And that’s kind of what I wanted to kick off with today is, you know, why do you think so many organizations just kind of stop it, you know, hey, we saved X dollars and they’re never pushing anything any further. What does making ROI kind of a core practice really mean in in this automation program?

Holly Holcomb • 02:43

Yeah, I think you nailed it on the head. I I think a lot of um a lot of teams begin with cost savings as their initial metric when it comes to ROI for an orchestration program. Um but it is that’s just one right we’ve got a white paper on our website that’s got a ton of different um ideas for how you can measure uh the success of your team. Um and I think that when it comes to justifying budget and allocating a team’s time or resources for infrastructure, um, sometimes that cost savings metric um or productivity centric metric, which is like, you know, it took X amount of time for two people to do it previously and now it takes 5 min of somebody’s time to do it with orchestration, that that comes up because it’s um it’s a really common thought process when it comes to justifying automation. And if it makes sense for your organization, like it’s a it’s a great one to try and like bake into your process, especially when it comes to prioritizing the work that your team is going to choose to do. Um in other organizations, uh there’s, you know, metrics that make more sense, especially when the delivery of network services um and delivery time to give maybe an application developer the infrastructure they need to build their application. Um maybe that delivery time is a better metric um that should be used to help prioritize and make sure that um what you’re building aligns with the business’s needs.

Holly Holcomb • 04:15

I think whatever the definition is that you choose, the bottom line is that it needs to be written down and it has to be a mechanism that you use to understand what it is that you’re going to target 1st and how you’re going to measure the value of that long term for that body of work. Um so just as an example, maybe there’s some degree of innovation as like a core value of your program or your organization. And um your metric is all about how you’re going to embed maybe like AI in a meaningful way within your automation and orchestra orchestration program. Um having some degree of metric that aligns with that driver within your program is so critical.

William Collins • 05:01

Love that. Yeah, it’s also important too, like when you think about it, sometimes sometimes funding is just there for tools and for platforms, and then you end up with a lot of them, and that funding is there until it’s not. And then they start looking at that list of things that they’re going to cut funding for. And if you don’t have all your T’s crossed and your I’s dotted, hey, guess what? Um, your your funding for network automation or some platform that you’ve bought or some tools that you bought, you know, that might be the 1st thing to go. So it makes a lot of sense to, you know, not put the cart before the horse and think about these things, you know, as you start trying to offer meaningful value to the business.

Holly Holcomb • 05:42

And especially when it comes to like having a um a metric that is like very specifically a data point um that aligns with what the organization is trying to accomplish. Like um it’s to your point, like it it is very easy to decide you’re going to decommission a tool or remove it from your ecosystem for cost um cutting purposes. Um, but it’s uh much more compelling to point to data-driven metrics um that align with the business um to help justify the existence of a team or a program.

William Collins • 06:15

I love that. That’s great.

Holly Holcomb • 06:18

So um, William, I guess the next question that I’ve got for you is what have you seen that has worked in the industry? Like what what metrics are effective? What do you see as like um, you know, working with the people that you talk to on a day-to-day basis?

William Collins • 06:37

This is a this is a hard question. And I think the question um, like this question in particular, kind of gets lost in like smoke and mirrors a lot of the time. So one of the things I’ve seen, just looking at the the culmination of like a lot of years, is 1st , like this is finally starting to evolve in a meaningful way, which is really good. Um, but also is that metrics like have to change over time, like as your organization changes or your your organization matures. And I know saying that just seems like really obvious. Like obviously it, you know, but let me let me just expand a little. So like very early on in an automation journey, you’re

William Collins • 07:27

Yeah. You’re focused on I would say uh efficiency metrics. So things like how much uh how much time did we save here, or how many uh fewer tickets did we open for this, or how many VLANs did we configure, whatever. Insert whatever efficiency metric you want to. And those are totally valid. They’re they’re great metrics, especially when you’re starting out because you need to show the quick win, like some sort of immediate value as to why you need you know dollars for a thing. But as you mature, that doesn’t cut it forever.

William Collins • 08:06

So what I’ve seen is like organizations really shift um their their mentality, their mental model more towards enablement metrics. And this is really, really powerful because it’s not just about doing you know the same old story, okay. I did, you know, X, Y, and Z faster. It’s about doing things that you couldn’t do before, which is super awesome. So is an example. There’s a lot of examples, I guess you could insert here, but maybe uh before you had any sort of orchestration in place, like spinning up a new branch office location maybe took three weeks and you know, maybe required uh pretty extensive coordination across maybe six different teams. I don’t know.

William Collins • 08:53

Just hypothetical. And now with automation and orchestration, you can do it in like two days with you know minimal like manual intervention. Now that’s not that’s not an efficiency metric. That’s that’s enablement right there. Like you’ve fundamentally changed what is possible for the business. And I think that’s a really important evolution because you know, efficiency metrics, they uh they have a ceiling, right? So you can only save so much time or so much money.

William Collins • 09:27

Um, but enablement metrics, like those can fundamentally change the trajectory of your organization and what your org can actually accomplish. So I’d say to folks out there, start with efficiency, of course, you know, because you have to, but always be thinking about how you Evolve towards enablement as you mature because that’s where the real uh transformation value, et cetera, et cetera, is kind of tucked away at.

Holly Holcomb • 09:54

Oh man, I have got a couple of a couple of customers that I that come to mind what that have like really leaned into the enablement side of things, like um on the service provider side of the house, like the value that you get out of formalizing orchestration around like really important business process like service activation, just like as an example. The ability to do it consistently, um, according to an SLA, um, that, you know, the data is captured and stored in a really consistent way. All of the doors that that opens up for that organization to understand better optimizations, whether they can hook into the process for additional business process, um, consistency of the data in your sources of truth every single time that can help, you know, uh boost an AI strategy. All of those things are very much like enablement centric. You know, it’s all the things that you can do now that you’ve got um a core orchestration in place that help evolve and transform the way that the business um functions.

William Collins • 11:06

Yeah, I love that.

Holly Holcomb • 11:08

William, you mentioned something a moment ago around customers like finding their North Star and understanding what it is um that’s really going to drive the direction of their roadmap and what they target in terms of the work that they do. Um, and we’ve got this visual that we oftentimes will use that kind of helps customers identify where their core um value driver is in terms of like a North Star. And then some tactical metrics um that align with those. So this is kind of a fun tool that we’ll use sometimes to spitball ideas with our customers on what the value drivers are that are going to help decide where their roadmap lands in terms of measuring and prioritizing work and outcomes.

William Collins • 11:56

I love it. Yeah, you have to have an overlay. You have to have some sort of uh best practice framework or something that you can start trying to fit yourself into. And I think this is a great representation of that. And something, I don’t know, it’s kind of like um when I think about just you don’t do things just to do things. You know, if you if you’re doing things just to do things, then like what’s what’s the point? How do you show the value?

William Collins • 12:27

How do you show progress? How do you measure anything? Like, no one like I don’t do household chores just for the sake of doing household chores. Like there’s a specific outcome, there’s a purpose. Um kind of like if you think of uh Uh you know, say they’re visiting for the weekend, and in this hypothetical scenario, the the in-laws are pretty much auditors. So they’re coming to inspect.

William Collins • 12:52

A picture of chaos, if you will, Ollie. So um talking about the outlaws, I mean the in-laws, there are very short-term goals that you have to accomplish, but there’s also the really like the long-term strategy. And, you know, these metrics and these frameworks can change a little bit based on what it is that you’re actually trying to accomplish. So just something to something to keep in mind there.

William Collins • 13:11

They’re they’re gonna be checking like behind the fridge, you know, looking in the kids’ closets, like, what’s going on here? You know, and they’re they’re gonna look at things you didn’t even know existed or could be inspected. You know, you have no clue. So suddenly speed to completion prior to their arrival becomes the critical metric because you have an external forcing function that kind of has the ability to bring, you know, morale as we know it, crashing to the ground around the house. So I guess what I’m trying to say is you don’t measure things just to measure things, you don’t measure chores the same way every time. You measure them in different ways, both based on like where you are in like this maturity cycle and what the circumstances are, because I’d say, you know, say, like, okay, let me just paint a picture.

Holly Holcomb • 14:22

Oh, I love that analogy. I feel like forevermore when I talk to customers about like the um percentage of like vulnerabilities on across their network, um, that I’m gonna like imagine an in-law asking them like, what is the percentage of compliance that your network has for known vulnerabilities?

William Collins • 14:42

Yeah, I mean, the last thing you want is your mother-in-law like going down the stairs and stepping on a stack of Legos. Like I did that not too long ago. Well, I do that frequently. So it’s yeah, yeah. Hot wheels routine.

Holly Holcomb • 14:54

That’s what we have in our house.

William Collins • 14:56

Yeah, these hard animals too. They have like these big hard animals now, and my daughter has like a thousand of them. And I’ve stepped on those things so much, they’re they’re like the worst. They they don’t budge. Yes, it’s brutal.

Holly Holcomb • 15:08

So I guess transitioning to another topic that is not always um top of mind when it comes to impact and measurable impact is culture. Um tell me, William, do you have anything that comes to mind when you think about uh orchestration’s impact or measurable outcomes of orchestration when it comes to an organization’s culture?

William Collins • 15:32

It’s huge. Uh, culture eats technology for breakfast, lunch, and dinner. So it’s, you know, obviously really important. So just top of mind, I think what I’ve seen with organizations that really get this right. And I’m actually thinking about one in particular just popped into my head, um, who has hit this out of the park in the real world. Um, and what they did is they built a culture basically around, you know, a bias for action. And let’s unpack that.

William Collins • 16:08

So what that means is, you know, they they set they set these guiding principles and you know, they define their their big rock initiatives, and then they stuck with them. So they they didn’t and they continue to not deviate from the strategy every time there’s like a new shiny object or a new change in leadership, a new VP gets hired, or the CIO changes or something. Like they they stick with their strategy. And this is very important to facilitate measuring impact on culture, and really measuring impact on a lot of things in any capacity, um, but especially culture. And you know, that’s really hard to do, by the way. You know, as everybody listening knows, because there’s always going to be pressure to pivot. Um there’s there’s always gonna be pressure to chase the latest trend to, you know, to respond to the the loudest voice in the room, I guess.

William Collins • 17:07

But the organizations that really succeed are the ones that say, hey, no, we we defined our strategy for a reason and and we’re gonna see it through. And this, you know, the particular company I was just referring to, um, and I actually just talked to them not too long ago, and you know, they handle a few things in very interesting ways that are just kind of novel when you think about how most big companies work. But they one of the great things they do is they treat innovation. Uh kind of like the NFL draft. So they bring in interns or new team members who are completely disconnected from the day-to-day processes. So these folks aren’t they’re not bound by, okay, this is how we’ve always done these things, you know, and they’re really enabled to go and build. They’re given a problem, um, they’re given limited guardrails and and a safe environment to work in, and they’re empowered to go figure out the art of the possible for new and novel ways to solve industry problems.

William Collins • 18:12

And what happens then is like leadership then comes in and extracts the the meaningful and and and reasonable changes from their ideas and their work and implement something, you know, somewhere probably in the middle, you know, not too radical to break everything, but not the status quo either. And I think that’s such a really cool balance. And you know, remember when you when when you create, you know, we always talk about tech debt, but cultural debt is huge, and that rent comes due. So you you have all the um you can have all the technical success on the planet, but if the culture isn’t there, you’re it’s not gonna be sustainable. So great cultures build great teams and you need teamwork to make the dream work. So I’d say find a way to measure engagement, contribution levels and and sentiment. And you know, if you see red flags, try and address them as early as possible.

William Collins • 19:07

You know, because again, culture compounds and both positive and and negative ways.

Holly Holcomb • 19:14

That’s such a good example of how um you embed kind of like an innovation metric inside of your program. I love that. I think the million dollar question at this stage is how I can embed a culture of cleanliness in my kids um to avoid those uh Hot Wheel landmines that we were talking about.

William Collins • 19:35

Yeah, or get them to consistently uh the funny thing with that is like just they’ll hide things. Like I will find things. Like we we miss my my kids have like no socks ever. Because when they put things away, they’ll just stick them somewhere. So there’s never there’s never any socks. So let’s talk about one of the most common challenges that I hear a lot from customers and other folks in the industry. Um I know you hear this all the time too, Ollie, and it’s around capturing data on the current baseline.

William Collins • 20:12

That you’re actually going to measure against. You know, I’ve been in so many conversations where a team, you know, wants to automate something. And, you know, we talk and, you know, we’re having a conversation. And it’s like, okay, so what’s your baseline today? Like how does the process, like how long does it currently take? How many people are involved? Just getting, you know, what’s the error rate?

William Collins • 20:33

Just any sort of information, any sort of data points. And a lot of times it’s like, oh, I don’t I don’t have that data. We need to think through that and get a lot of people on a call. But um, yeah, I’d love to get your thoughts on that, Holly.

Holly Holcomb • 20:46

Yeah. I feel like we have this conversation a lot with our customers. Um, and initially the it comes through the form of asking help on, you know, how to prioritize their backlog or their roadmap. Um, and usually the call um progresses and it turns into me asking lots of questions about what exists today, what data do you have access to today um that can help you give that baseline information for a lot of customers? It’s going back and digging through their ITSM to figure out um, you know, if especially in the um In the case that we talked about at the top of our conversation, where it’s delivery time, how quickly they can deliver a network resource or service to their internal customers. A lot of times it’s going through the ITSM and trying to find a baseline.

Holly Holcomb • 21:42

And so it’s really being specific about what you’re trying to affect change on with your orchestration. You know, if it is delivery times, if it’s just coverage of orchestration based on the types of requests that come through to your team. I feel like we see this really frequently. Without that baseline, trying to make a case for how we’re improving it or the impact of orchestration at the end of the life cycle, like the development lifecycle, we release it, we see the results coming through. Um it’s huge, you know. So you really want to spend time to understand what your baseline is and how you’re going to impact it with the orchestration.

William Collins • 22:42

I love that. And if you don’t know, like, hey, if you don’t know where you started, how do you how do you measure anything? How are you going to demonstrate improvement in any capacity? I mean, you can’t just say, like, hey, well, this used to be hard and now it’s easier. Like, that is not going to cut it when you’re trying to justify getting budget or people or resources. So yeah, I mean, one thing I’ve learned is, you know, this is something that I think is really important is that you need to do some, and like you were saying, you’ve got to do the research. You have to put in the work up front.

William Collins • 23:16

And I know we talked about it on the last webinar, I think we recorded, but there’s there’s not really shortcuts to that. There, there is work that you have to put in up front if you really want to see this get off the ground. Um, very important. And one last thing is even, and I think this is The most controversial sometimes, but a lot of times you’ll see some strategy, and it’s like we have to automate every little thing, no matter what it is. 100% automation at all costs. And I mean, even that kind of decision, like, you know, one thing to start with is hey, you know, you might have two different buckets of things.

William Collins • 23:57

You might have, hey, these are things you should automate. And maybe these are things you probably shouldn’t automate for various reasons. So even taking the tasks and putting them into the right buckets to categorize them. Because I mean, here’s the reality: not everything should be automated. You know, I work for litential and automation company, but you know, this is this should be standard knowledge. And, you know, when the thing you’re thinking about automating maybe isn’t even measurable, and you can’t demonstrate meaningful change if you do automate it. You know, that’s a great opportunity to pause and decide if this body of of work should even be prioritized because there’s a lot of other stuff that probably has a you know higher priority.

Holly Holcomb • 24:43

Oh, that’s such a good point. I think a lot of folks um have great intuition. A lot of our customers have great intuition around the areas where they should focus their time and energy. Um and that intuition will sometimes yield like excellent results. Um, but doing doing the baseline research and also like to your point, understanding what we should be targeting, what maybe we shouldn’t be targeting, um, is like great, a great aspect of that research.

William Collins • 25:15

Yeah. One last thing. I don’t want to rant here, but this is something I learned the hard way is like I always underestimated. How complicated infrastructure is and can be. So we get, you know, a lot of times we’re humans, we get tunnel vision. You’re usually focused on a very specific piece of the overarching like ecosystem. And in making sure you do your homework on like the larger effort, especially when you’re going from like automation to true orchestration, that can yield incredible results, you know, that will help justify additional budget and resources in the future.

William Collins • 25:53

So don’t just look at this tiny little slice of the network that you manage. Look at your, you know, how that automation and orchestration fits into this larger picture. Because oftentimes the ROI story gets way more compelling when you can connect, you know, connect a few dots across a few different technical domains. So we’re starting to wrap up here. And I think it’s worth taking a step back and asking, uh, maybe what what is success even look like here? And I think the answer might be pretty straightforward. If you want to build a sustainable uh successful automation program, and you and you need to demonstrate the value with it.

William Collins • 26:39

You have to take ROI seriously, not just once, not just like at the very beginning when you’re getting funding or you get some time committed to it, but continuously through um the the whole entire process. Uh and the the organizations that do this well, the ones that have data-driven you know, prioritization built into the process, they’re gonna probably see continued investment and support. But uh do you do you have anything for the audience um as we close, Holly?

Holly Holcomb • 27:09

Yeah, I think what we see most is that um most automation programs and orchestration programs stall when they can no longer effectively answer the question of why are we doing this? And the truth is it’s not enough to say we just save some time or cut some costs. Um leaders want to know did it move the business forward? When teams treat ROI as a discipline instead of a checkbox, um, they get like some really key benefits that you just mentioned. Um so like there’s pieces of it uh related to the ability to measure and justify expansion of your program. So easily mapping the outcomes of your program to business outcomes that all stakeholders inside of the organization can understand, whether it’s like a reduction in MTTR, whether it’s faster onboarding or activation of customers, um, whether it’s um quicker application deployment, like we were talking about earlier, and getting access to those network resources faster. Um, they all earn buy-in beyond just the technical teams.

Holly Holcomb • 28:18

So ensuring that the outcomes of your program align with business outcomes, and those business outcomes are easily understandable to all of the peers and leaders within the organization is super, super important. Using the same types of metrics that help you prioritize things, um, like the things that are going to best align with those value drivers that we talked about previously, um, it’s going to help you filter out the noise. Like we talked about before. A lot of people have really great intuition, but there are times when we make mistakes, we try and focus on work that maybe shouldn’t be automated, like you mentioned. Um, making sure that you’ve got some like hard data and metrics that will help you filter out the noise and really focus on high impact work that has measurable outcomes is super, super important. And we see that we see that in our customers um that have that very um I think about hygiene when I think about this effort, but like they have the roadmaps that very clearly track business outcomes and have very specific metrics associated with why the work got prioritized.

Holly Holcomb • 29:26

So those are some of the key attributes that I think we see that really make a big impact with customers when it comes to like how this somewhat theoretical sounding activity comes into play with the actual tactical pieces of running a program. And the real answer is we’re doing it because we’re trying to justify expansion and growth and innovation within the program. And it’s not necessarily just about tracking, you know, in the past. It’s not about saying, this is what we worked on and this was, you know, why we prioritized it. It’s it’s really about making sure that uh moving forward, as you mentioned, like when you’re shifting what kinds of value drivers or metrics you’re using within your program because they should transform with time, um, that they continually align with the direction of the overarching organization. Like they should, they should transform with time, they should represent the market on your company, all of it. It it it’s all it all interweaves, I guess is what I’m trying to point out.

Holly Holcomb • 30:42

Um it’s not a one time activity, like you mentioned. It’s something that should define the overall leadership and direction and roadmap of your of your um orchestration program.

William Collins • 30:55

I love that. Or so if anybody wants to learn more, um, what can they go to? Do we have any resources uh to send ’em?

Holly Holcomb • 31:05

We do. Yeah, I mentioned it at the beginning of our conversation, but we’ve got a white paper that can get give you lots of examples of what some great metric ideas are that are outside of the box of, you know, normal productivity or that type of thing that um oftentimes usually are like the seed that starts the automation program. But um there’s lots of great ideas in that white paper that is all about aligning automation to business value and identifying the metrics that are gonna get you there.