William Collins • 00:05

Welcome, one and all, on this fine and beautiful Wednesday afternoon. My name is William, and I work as Director of Tech Evangelism here at Itential, and with me, I have the privilege of getting to talk through some pretty valuable topics with Holly Holcomb. How are you doing, Holly?

Holly Holcomb • 00:25

Not too bad. Not too bad. I’m really excited to dig into some of this content together.

William Collins • 00:31

Yeah, yeah, I’m really excited too. This is such a great topic because, you know, a lot of times the technology is just… I mean there’s means out there you know the technology is the easy part and you know when you throw in a lot of people into this big machine it gets harder and harder and adds complexity and you know that’s kind of what we want to talk through a little bit today is the the people side of a lot of this and I really can’t think of a better person to talk to so with itential your your work like day in and day out is with some of itential’s larger in high value accounts you want to just kind of start out by you know what does that mean what is a program director of strategic accounts actually do day to day absolutely like you said I have the opportunity to work with some of our more complex accounts that are trying to figure out ways to accomplish more and push closer towards like digital transformation initiatives and so it’s been really fun to work with you.

Holly Holcomb • 01:37

It’s been really fascinating to get to understand some of the unspoken or invisible challenges that they’ve run into and some of the innovative solutions that they’ve come up with to try and overcome them to reach their goals.

William Collins • 01:49

Awesome. Awesome. Well, let’s get started.

Holly Holcomb • 01:53

So one of the first topics that usually comes up when we talk with some of our customers about challenges, operational challenges with their orchestrations is related to fragmented or inconsistent inventory or multiple sources of truth. And I know that this can be a topic that folks have lots of opinions on. William, what’s your take on a single source of truth and the kind of, you know, how many customers are getting there today? How many organizations do you see achieving that today?

William Collins • 02:23

That’s a really good question. I worked in the enterprise for a really long time, 15 plus years, and I would say if there was one lesson I learned, it was tool sprawl is a real thing. Tools get purchased, they get operationalized, and they ultimately get controlled by different teams. With that being said, my take is single source of truth is kind of a myth. Now, I will say that there’s source of truth platforms out there that do an incredible job. Basically, what they do is they aggregate other sources of data for different scenarios and use cases, etc. My take on the problem and solution is that it’s just extraordinarily fragmented. I’m saying this from past real-life experiences. When you say single source of truth and you’re aggregating this data, but say that there’s a source of data somewhere that you don’t have control over, or maybe you just have a small subset of control, and maybe you can’t turn off. the magic admin button for other people updating and modifying that source of data. So then how is it a source of truth or a single source of truth rather or not? But again, I’m actually saying this from past experiences and I’m actually rather new here to Itential. I’m pretty new, about three, four months in and that this is my reality. But what have you seen, I’m more interested in that across your experiences with some of these larger organizations in the real world that you’ve worked with.

Holly Holcomb • 04:21

Yeah, I think that the myth point that you bring up is like very palpable in the industry. I think a lot of folks are trying to strive towards reaching that goal. And I think there are a lot of people who think that in order to begin their orchestration journey they have got to get a single source of truth ironed out for inventory in addition to some of their other sources of truth or systems of record. And it is just what we have seen is that folks that stall and they wait until they have a pristine source of truth never get started. And the reality is that you are going to have to figure out a way that the orchestration strategy and the data integrity strategy work together and like the collaboration and overlap between those two yield results. So as an example for folks that are. working towards making a more consistent source of truth for green-filled deployments.

Holly Holcomb • 05:25

That is something that’s built into the orchestration. In other situations where you don’t always have green-filled as the option, you’re working the brownfield, you’ve got to figure out how to bring all of the assets that are out in your network, the legacy devices, all of that into some level of data integrity, into a source of truth. In those situations, the orchestration is built and developed with that goal in mind. It is an objective of the orchestration roadmap. And so I think that one fallacy that we’ve noticed is that those two are almost seen as like sequential tasks. I’m going to first work on my data integrity and make sure that my source of truth is completely pristine, and then I will start orchestration strategy. I think that when we see customers really achieving innovative solutions in their network, it’s when these two coexist and they collaborate together.

William Collins • 06:19

I love that. And you make a few really good points there. And one thing I would point out is most larger enterprises out there, a lot of times what happens is you go through a whole year, you have a project or you have a budget to do X, Y, and Z. You have programs and different environments you’re building that are going to support applications that are going to make a company money. And what tends to happen is you come in and you adopt and you build this infrastructure and it reaches a certain level of maturity, but then you get yanked. And it’s like, hey, we’ve got this next thing. And you don’t have… Staffing is not unlimited. You only have certain amount of staff, certain amount of operations. So then that greenfield then becomes a brownfield that maybe wasn’t completed 100%. So then what ends up happening is you have lots of brownfields out there, lots of different brownfields that were built with architectures and design patterns and principles that were like best case scenario maybe at the time. But hey, we all know that technology is moving at light speed. Don’t get me started on the AI stuff. So you have all these brownfields and then you have all these new greenfields coming in. And the… I’d say the likelihood that any large company is just going to be able to say, you know, hey, halt. We’re getting our source of truth in order. You know, we got to like, you know, pause until we get all this stuff ironed out so we can do good network automation. That’s a, if you work in these, you know, you just know that that’s not going to happen. It’s not realistic. It’s not reality.

Holly Holcomb • 08:02

Absolutely. Yeah. A more realistic perspective is like team A uses this system for their source of truth. Team B uses a spreadsheet. Team C uses ServiceNow, CMDB, all of them are using a different source of truth and figuring out what makes the most sense for your roadmap, for your orchestration roadmap that not only achieves the goals, whether, you know, whether it’s new device onboarding or decommissioning, whatever it is, but also is contributing to better data integrity across the network. Like, that is a, that is, I think, a little bit more realistic perspective on how those two work together.

William Collins • 08:44

For sure. So, configuration consistency, Holly, it’s challenging, but it’s also absolutely crucial in network engineering and then tying in automation. And there’s so many reasons for this. We could probably talk about this for an hour, but if you think about at the highest levels, you have just reliability in general of a whole network as a single organism. So how do you think about that? So, when you want something to be reliable. you need predictable behavior and that means you need consistent configurations you know to basically ensure that your network is gonna behave you know the way that it’s intended to behave but on top of that when you have consistency I think you also you know it simplifies other things like like troubleshooting you know when you’re you’re troubleshooting a network of all these interconnected things knowing what to expect and you know knowing that your configuration and your outcomes are gonna be the same you’re gonna be able to isolate you know problems much quicker you you just have this baseline this this foundation to work from you know on top of like security posture like access control enablement change management it just all sort of ties together so what have you seen as far as like your your experience working with you know larger environments around config consistency and like the importance here

Holly Holcomb • 10:17

Yeah, it is definitely the elephant in the room a lot of times. I think a lot of folks assume that as a whole we’re further along than we actually are when it comes to like nailing down what the formal standard looks like for day zero, day one. And usually the result is that there’s a team that’s come in, that’s being brought in to help solidify some of those standards and implement them and actually bring visibility to what the network looks like today. All in an effort to highlight the things that you’re talking about like reducing MTTR, making sure that from a resiliency perspective the network is, you know, everybody can use it and it’s working effectively and efficiently. And so When we see folks, which this has popped up a good bit on roadmaps that we’re privy to, everybody is trying to get to a better place when it comes to consistency across their network. And especially when it comes to acquisitions, mergers and things like that, there’s giant swaths of the network that are introduced that you’re now responsible for managing and maintaining and having the same quality and output as the traditional network. It is so important to formalize that standard and figure out what makes the most sense for your organization based on the security standards, based on just like general hygiene. So I think that config consistency is one, again, that’s popping up on a lot of roadmaps and making sure that you look at it as like a practical and pragmatic process. Like step one, we’ve got to define what the standard looks like. And the biggest pitfall that we see is that folks try and boil the ocean.

Holly Holcomb • 12:06

They start by trying to figure out what the entire config needs to look like and not just some of the basics around like SNMP or NTP. Like you’ve got to start somewhere, start small, iterate, kick off little chunks that make the most sense that are gonna give you the biggest bang for your buck. And I think that’s like probably the punchline for this particular invisible issue is like don’t try and boil the ocean.

William Collins • 12:31

I love that you said so many good things there but just kind of I want to pull out something individually in this slide deck that make it easy for teams to stay consistent by giving them the right tools and guardrails. Now there’s so many different security vendors, security protocols, security practices out there and the ones that do really well and really keep things locked up tightly are the ones that make it easier to be secure. Like you don’t have to read a 100-page manual for how to do basic security with the platform. You know it’s just kind of there. I mean even if you go and you sign up for some of the newer like cloud-based services there’s some services out there that will actually make you do two-factor auth and that’s huge especially for a you know something that’s publicly hosted and available on the public internet. Those are you know again they’re guardrails. They’re there for safety and they just take the whole the whole idea, the whole concept of the developer, the consumer, out of the equation for having to think through how to put security as a first-class citizen in these scenarios.

Holly Holcomb • 13:48

This next one, William, I am super curious to hear your thoughts on. It’s one that we hear from a lot of different customers, but when you reach the low-level design stage, when you’re building something out, and it involves sifting through API docs, do you ever feel overwhelmed, or do you ever feel like it’s taking entirely too long to find what you need?

William Collins • 14:13

You say overwhelm, and I think overwhelm is actually a natural part of the process when you’re digging into APIs in any sort of low-level design. Call it part of the people pipeline. I need a runner and a job manager just to manage my stress at this point because I’m about to pull my hair out. So I think the key is, at least the way I think about it, is recognizing that feeling that it’s happening and knowing that there is a better way, and building a strategy to work through it. Organizations only have so many slots for operations and engineers. You can only have so many full-time employees, and it’s tough. And I think…

William Collins • 15:04

Honestly, as we think through this, I think the bigger question is, what are the possibilities of eliminating some of that tedious and time-consuming work? So teams can focus on deploying the reliable infrastructure that really underpins liable applications that make the company money. So how do you do that and how do you take away those tedious things that take up a lot of time and take your skilled professionals away from doing value-added work, which is where you want them. It’s where you want their brain cycles. So what about you? Do you have any wisdom here from your time and experience?

Holly Holcomb • 15:49

I think you put it really well when you talk about overwhelming and searching for the right word. A lot of our customers aren’t comprised of software developers. It’s rare that you have a team of resources that are really adept at navigating all different API docs to figure out the needle in the haystack and figure out which one’s going to best serve their needs and get them what they need. We had a customer recently tell us that they are very intent on burning down boring work and figuring out how they can get people away from navigating API docs that are not going hundreds of pages of PDF that are not super easy to dig through to find what they need. And a lot of what we’re seeing is that folks are trying to use technology like AI to figure out what makes the most sense that helps reduce the noise and focus specifically on what they need to accomplish their goals within the orchestration. So in terms of like trends and ways that we’re seeing people that burn down the boring work, it is very much like using the tech to be more effective and have it work for you.

William Collins • 17:04

I love that. On to the next thing, and I really, I think at the end of the day, like, if you think of like true transformation, like truly transformational network automation, you know, with goals, you know, really big goals, it’s often going to span like lots of different teams, lots of different domains of technology, you know, which ultimately sees to, you know, complexity is going to just slowly increase exponentially, because you have more silos, you have more people, you know, you’re growing. So I wanted to give an example, to try to add some like additional color to this conversation. And I want to try and paint a little Picasso picture through the lens of scale, you know, is, you know, to me, I think the gap that exists between just doing a little automation, and really doing like full orchestration at scale. I think, you know, in my conversations, anyway, that’s a hard thing to really understand that these are two very different things, especially, you know, the larger of a company that it is, that’s, you know, doing the automation and orchestration. So I’m going to use a cooking example. So Holly, do you cook? Do you do you love to cook?

Holly Holcomb • 18:21

I’m a terrible cook. Terrible.

William Collins • 18:24

Well, today, you’re going to be the best chef on the planet. So, so think about it like this.

Holly Holcomb • 18:29

Restormation.

William Collins • 18:31

So say that you, you, you Holly, you’re cooking a single meal for yourself. You know, it’s only what you want to taste, you know, your preferences, like you’re if you have any allergy, like accommodations, and hey, you’re only doing dishes for one person, you know, how must that feel? Oh, must be amazing. So. Now, imagine that you want to scale Holly’s cooking expertise, like you made a dish the other day and it tasted so good and you’re just like, well, I’m the best cook on the planet. Let’s scale. So, let’s say you open up a restaurant. Now, you have multiple teams that are going to be involved with this restaurant. Maybe it’s a chain of restaurants. You have multiple locations.

William Collins • 19:23

You have a lot of cross-functional disciplines here. You know, you’re coordinating the kitchen stuff. You know, you have the waitstaff. You have schedules. You have inventory, supply chain, and reservations, you know, because Holly’s restaurant is going to be very fancy. So… This to me is honestly the difference between a single automation and true orchestration is, you know, Holly sitting in her kitchen just making her meal for herself and then Holly opening up this gigantic five-star, you know, culinary empire.

William Collins • 20:01

So when we think about just all the things that have to go into building out that program for that restaurant chain or this orchestration, there’s a lot of people, a lot of process, a lot of things involved. It gets much bigger and much more complex. How can organizations think about approaching this? Like how do you even begin to think about it? You know, what have you seen?

Holly Holcomb • 20:28

I think if I’m gonna use your analogy here, it’s almost like a localization strategy. How do you like maintain the brand, but in another market where maybe that type of cuisine is not as popular? How do you like change it ever so slightly? Because at the end of the day, every organization within an organization, there are different silos and they all have a preference around the tooling that they use. Some teams will lean really heavily into tools like Ansible. Others will prefer to use a controller-based automation solution that’s got like a GUI. And so making sure that whatever it is, that you have got some level of governance that ensures consistency on the outcomes of the automation or orchestration program is really, really key.

Holly Holcomb • 21:16

So for example, if I’m going to open up my five-star restaurant in a place that is, you know, vegetarian, I need to make sure that the cuisine is altered in a way that makes the most sense for there. So making sure that you’re using the right tool in that specific domain, what’s the most effective for this team? What is going to yield the best results? And making sure that whatever type of overarching orchestration program and governance is on top of that automation tooling, that they’re able to contribute in an effective way. They think oftentimes. when we try and build a governance and kind of centralized program over different silos of automation, the challenges and obstacles that folks encounter is that, you know, this automation doesn’t play nicely with this other domain tooling. You know, like there’s a lot of mismatching in terms of tooling and efficacy.

Holly Holcomb • 22:15

And so it’s really important that as you’re considering your orchestration program and how folks can contribute into the orchestration program that it is flexible and works for all the tools that are most impactful for that specific domain.

William Collins • 22:34

I love that you made you made a lot of great points there but one I think one point and it’s kind of clearly outlined in this slide here so choose a platform that plays well with others across CLI, APIs, scripts, and more because you know what most companies out there are using CLI to some degree even if it’s in pipelines really they’re using APIs they’re using scripts they’re using a of things and you can’t just come in and say hey we have this new thing and we just need you to hey stop doing everything the way you’re doing it today just kind of like flip that light switch and just start doing everything our way It’s not that easy like these are mountains that you know an enterprise has to push and it’s just it’s a lot of hard work It’s a lot of Just taking I mean even taking existing applications platforms tools and stuff and refactoring them Everything’s so complex and interconnected that you may start poking around and you may break something You know as you’re trying to adopt a new solution that wants to do something a different way So I I love that idea of you know a platform coming in and meeting consumers like where they’re at meeting consumers with what they are using how they’re using it security wrapped around it the compliance and Enhancing it so I guess pro tip Play nice with others, you know, there’s a lot of different tools platforms and services out there The technology is available to us the way that you know The API ecosystem is really just blown up over the past five eight ten. However long it’s been at this point API’s proliferation of API’s proliferation of tools and there’s gonna be a really random Combination of these platforms and tools that are adopted by different companies. So how do you go in and have like a Way to actually pull everything together and wrap compliance and such around it I think that’s a great a great point and that is a good a good tip that I would definitely recommend So we really can’t talk much without talking security because security is ingrained, you know, it’s many layers of a big giant onion and every layer that you peel back, you know, security is in scope in every single layer of technology that we have today. Imagine a world where, you know, William’s working for this big company and this company doesn’t, you know, they maybe put a little bit too much trust into their developers and just, there’s not a lot of guardrails around. And, you know, William, he decides, hey, I’m trying to move fast. I clone this project out there that I saw on GitHub. I have access to whatever our CICD platform is that’s maybe hosted in our data center.

William Collins • 25:37

And I’m going to, like, pull this down. And not only that, but I’m going to actually integrate some other open source tech out there in a way that’s fast. Because you know what? William’s got a lot of work to do at this job, and he has other things he’s got to do. So he’s just trying to get it done, okay? So he pulls everything in. He has a lot of defaults running.

William Collins • 25:57

Maybe he even, oh, I dare to even say this with my name publicly, but dare I say, he even has some secrets that he accidentally committed to version control. You know, William’s just a hot mess when it comes to open source security and, you know, building things in this manner. So that’s what he’s doing. leads me to my next, you know, and our next sort of talk track here is, like, you can’t really do automation at scale or orchestration without guardrails, you know, because your reputation, your brand, the integrity of the company you work for, hey, it’s on the hook. So I just want to, like, after this, you know, this completely random example that I hadn’t even thought through until now, have you ever seen the impact of, like, reputational damage for new technology, you know, when there is new vulnerabilities discovered and things happen? Like, what does that look like? Because everything’s so automated and so interconnected. So what’s the story here, Holly? I think all it takes is one incident for a team and the outcome of the team to get impacted by the reputation of something that’s insecure, something that comes out of the team that is insecure. And that is part of the reason why it is so critical that security is ingrained in the DNA of the orchestration or automation program.

Holly Holcomb • 27:27

It has to be in the governance, it has to be in the reusable bits that all of the builders are expected to reuse, that represent those best practices, that represent the ways in which you should be using your vaults. It is so critical because as soon as as soon as an incident does occur within the program, the level of visibility and scrutiny that then comes down on the program that is such that everything is painted through the lens of, do you remember that incident that happened last quarter? So that’s why it’s really important, not just as like a, as a kind of an ethos, but very much in the tangible and practical assets that are produced by the orchestration team that ensure reuse of best practices when it comes to design standards and security standards.

William Collins • 28:27

Yeah great great answer and you know one thing that I’d point out is like in that that bullet right there all it takes is one misstep to derail trust and with with bigger companies especially companies that have been around a while that’s also all it takes to get your automation ambition shut down. So if you you know say that you push a change or you do something and there’s a security event that happens after it it’s gonna get heavily scrutinized and you know there is a chance that you’re gonna get you know access you know that they may just say hey we’re not gonna automate right now anymore this is a big thing that happened you know we don’t want to end up on the news we need to slow it down and you may you know it’s like that whole you know for every one step forward two steps back kind of thing you know this is a very real thing in these scenarios because you have to remember whether it’s an outage that happens you know say that your automation causes an outage or it causes a security event and you end up on the news or something you know something happens along those lines you know what that that can ultimately cost that company just millions and millions and millions 50 you know 60 70 million dollars plus potentially with all the things that go along with it so thinking about the security I don’t want to try not to use shift left that much but you do want to shift at least the way that you’ve used security and how you’re going to implement the security earlier on in the process you know into the standard into the documentation and you know like this the pro tip there says you know make them easy to follow but make it very hard to ignore it’s such a good piece of advice there Return on investment is gigantic. You know, when I when I worked in the enterprise, whenever we would bring in a new vendor, oftentimes it looked something like, okay, this new vendor is coming in. Is there overlap with our existing vendors? You know, we have to do this big calculation, you know, because things get expensive and, you know, again, tool sprawl. It’s a real thing. So when you think of ROI when it comes to like an enterprise, like a lot of times what happens is you’ll have a program, you know, you have this budget that you got, you have a program that kicks off, and maybe that program’s like eight months to a year. Maybe it’s, you know, it could be multi-year. Now, a lot of times there’s a lot of, you know, maybe professional services help or some, you know, partner help that comes in and, you know, you go through this ROI exercise early on at a kickoff. But after that, it just kind of fizzles. And that’s something that’s really important is if you invest time, you invest people, you make a purchase to bring in a platform or some new way of doing things, you want to make sure that not only the technical outcomes are squared up correctly, but you want to make sure that you have a good understanding of your return on investment. And you want to keep doing that. You want to keep measuring it.

William Collins • 31:40

And even knowing how to measure it, that’s an exercise in and of itself, because it’s going to be a little bit different for most companies. But what is your take on ROI here, Molly?

Holly Holcomb • 31:52

Yeah, my take is that we that one time kind of justification that you mentioned when we’re about to purchase something is usually pretty disciplined. But then afterwards, the decisions on how to operationalize the tool or what we’re going to accomplish with the tool or where we’re going to target our efforts, that is sometimes less disciplined. And where we have seen our customers really do a great job at promoting the outcomes of their program is when they’ve got a lot of data that drives the decisions on what it is that they’re going to build in terms of orchestration, and how they define value associated with those orchestrations. So just as an example. There are situations where you get a request in an email or a Slack message or whatever it is about this thing that needs to be automated or orchestrated. Can somebody please do it? And those, you know, sometimes squeaky wheel gets the oil, as they say, and they just do the work because of the person that was that requested it.

Holly Holcomb • 32:59

In other situations, it’s built into the governance of the program. When they have a request come through and they put it in their backlog, there is a very like standardized way in which the business justification is also included within that body of work. So for example, whenever we do this today, it requires three engineers to be on a change window at 2 a.m. on a Sunday night. That’s a lot of money. And when it goes wrong, it goes from, you know, a one-hour effort to a five-hour effort or something like that. So ensuring that all of that is tracked at the beginning of the program before it’s even prioritized in your backlog.

Holly Holcomb • 33:40

And then once the work is actually done and released and people are consuming it, that you’re also tracking executions of it and also tracking how it’s impacting your KPIs. Is it moving the needle? Is it accruing value for the organization? All of that I think is a level of like hygiene and rigor that customers that built really strong fortified programs have baked into the process.

William Collins • 34:13

I love that. Those are, again, just great call-outs. I love that sort of thought there about build an intake process that ties every automation to a tangible value before the work begins. This is really important. I can say that from experience. A lot of times, in my experience in the Wayback Machine, something that really helped in one of the companies I actually worked for was Cloud came along. Cloud, since you’re not dealing with any iron, you don’t have physical gear out there that you’re building. Cloud sort of brought in that app dev approach of, hey, we have build and release. We have CICD. We have these pipelines. We’re doing the same things every time in really small buckets of changes, tiny changes. And one of the things that helped us align the physical, dare I say, traditional infrastructure was how the Cloud folks brought on that intake process. We had an enterprise intake process where we would come in, the design would get validated by a domain expert and such, but then we had this program slash center of excellence, whatever you want to call it, that would go through and actually build these things out and Then tie everything back together back to the cost center back to all the different things all the tagging would apply and basically like the way that we structured the tags and the You know tie back to our asset inventories and stuff with the cloud We could basically go through and like type in a few Tags and it would pull up all the infrastructure the cost and like where the cost was coming from. It was very Very neat and it’s it’s really hard to do this with physical infrastructure. By the way, it’s different because There’s a concept of a mutable and mutable configuration. So mutable is kind of like what we see usually with physical network infrastructure. Like the asset’s already out there, it’s running, you know, all the airflow is working, everything’s great.

William Collins • 36:28

And you go in and you make a configuration change to it while it’s running in production. But immutable stuff is like, hey, we’re building this artifact in a pipeline, like in the build process, and we’re just deploying that whole artifact. And you know what, if we need to make changes, we’re wiping it out and we’re just replacing that with the new artifact. So while these worlds are different, that whole idea of an intake process and planning things out in that way, you know, kind of like what you laid out is super critical. It’s been really fun, you know, getting to talk through, you know, some of these critical things with you, Holly. And I guess if anybody wants to learn a little bit more, they want to reach out, they want to talk, you know, where can the folks go?

Holly Holcomb • 37:16

Yeah, obviously, they can reach out to you and I on LinkedIn. That’s one option. You can also reach out to us on our website. Or if you are an existing customer, please feel free to reach out to any of your intentional point of contacts. And we would love to help talk about how others have overcome the invisible issues together.

William Collins • 37:41

Awesome. Until next time. Until next time.