How to Audit Network Security Vulnerabilities from Cisco PSIRTs with Itential’s Automated Configuration Compliance

Dan Sullivan • 00:00

Hello everybody and welcome to another Itential webinar. Today we’re going to be focusing on auditing network security vulnerabilities from Cisco P-certs in this example, but this is not something that’s specific to Cisco, using the Itential automated configuration compliance. And this solution is focused around the Itential platform on how we can use the tooling that we have to audit configurations to look for those vulnerable configurations that we get thrown at us every time we see a bias series. The timing of this, I think it’s probably due. I think we had one bundled security advisory from Cisco last week for iOS XR. This is totally applicable to any operating system, Cisco, Arista, Juniper, the usuals and not the so-usuals, to detect configurations that are impacted by any CVEs. Also, we wanted to release this because the iOS bundled security publication is due either this week or next week. So it’d be a good idea to go through and do a refresher on how to use Itential’s capability to detect some of these and then produce some reporting.

Dan Sullivan • 01:13

There are various challenges with creating reporting for advisory, right? This is one of those tasks that engineers have to do and network engineers have to do more specifically, but don’t necessarily prepare for this, right? A lot of people don’t necessarily focus or prepare for doing this throughout the entire year. I did several years of consulting, I worked for Cisco for 10 years, and one of the things that I had to do throughout the many years that I worked there was actually support our customers throughout the releases of these advisories, right? And sometimes we have to build custom reporting due to demands from executives, right? Somebody says, I want to know from our entire infrastructure, what devices are impacted by this PSIRT that just came out last week. Or sometimes we have to do it on demand based on some of these industry wide advisories that we get hit with every now and then.

Dan Sullivan • 02:08

And we get these advisories on the vendor’s website and we get some details from those advisories, but there’s not a lot of clear path on how we get from there to actually identifying what is vulnerable on our infrastructure, right? Sometimes we get an email, we get a link from our support people saying, hey, you might be impacted by these, but we don’t have necessarily the tooling necessary a lot of times to actually go ahead and check on the environment against our real devices. What we’re trying to do today is demonstrate how Itential provides the capability to add new advisory checks very quickly and compare them against your environment, so you can assess any potential vulnerabilities and potential threats in the future to your environment, how to prepare these so that they can be executed on-demand or scheduled as well. One of the things that becomes really important is once we do the first check against the environment, we want to go and figure out an action plan and remediate those as we can. A lot of times we are not able to just submit configuration immediately to remediate those, or sometimes you have to upgrade software, and that requires planning, it requires maintenance windows, approvals, and so forth. From then on, we actually want to make sure that we execute ongoing checks against the environment to make sure that we’re tracking what gets remediated and what is left to be remediated. The schedule piece of this becomes really key, and having the reusability of whatever automation we designed so that we can use it in the future for any new advisories that get released.

Dan Sullivan • 03:55

One of the other pieces that’s really important is that some of these advisories are very easy to identify by just looking at potential configuration parameters that are identified as vulnerable, right? Some of the ones that we’re going to be keying on today are the usual suspects, right? Things like Telnet, things like HTTP server running on the devices, HTTP secure server and so on. But there are also some other parameters that may not be as easy to identify, right? Things like crypto, like IKEv1, for example, is one of those that you don’t necessarily need to configure in order for it to be running on the device, right? If the device may have some features that you don’t necessarily need to configure, right? in the back-end that are using some of those protocols that may be affecting the device and you may not know it.

Dan Sullivan • 04:42

That requires also not just auditing configurations, but also auditing the operational state of the device. One of the things that we’re going to be doing is actually looking at both aspects of this. How do we audit configuration that’s existing on the device? Also, how do we audit operational data by issuing show commands live on the device and actually checking against certain parameters that we want? For example, executing a show crypto ISA camp peers and making sure that we don’t have any peers active on IKEv1, or making sure that we don’t have any SAs active and so forth. Lastly, the reporting piece of this. We actually are going to showcase how we can utilize basic HTMLs or basic tooling that we have available, and we can use things like ChatGPT and so forth to build that basic HTML templates that we can use for reporting and using Itential’s integrations actually display this in the tool itself.

Dan Sullivan • 05:42

But not only that, but also be able to send it as an email. Today we’re going to demonstrate how we send the report over email and that becomes very useful for those scheduled runs where we just get an email at night and we say, hey, this is compared against like the environment and so forth. But then there’s also the other plethora of integrations that exist within the Itential platform that you can use for this, namely Microsoft Teams, Slack, or any other systems where we could actually have these reports being attached to a change request, something like ServiceNow or incident creation in Salesforce and so forth. For the demo portion of this, what we’re going to do is we’re going to be focusing on compliance piece and the Golden Config application of the Itential Platform to identify those vulnerable features on iOS XE devices. We’re going to use two or three iOS XE devices I have in my lab for this and they have some configurations there that purposefully are impacted so that we can just go ahead and showcase those and see how we go about creating reporting for those in the format that we have decided. We’re going to go live and create a Golden Config to do this for the actual configuration parameters identified. We’re also going to use command templates to audit operational features.

Dan Sullivan • 07:06

Like I was explaining before, some of these vulnerabilities cannot be identified by just looking at the config, but we actually have to execute show commands to identify the operational state of some of those features. We’re going to go ahead and do those. We’re going to use those two together, and we’re going to execute a reusable workflow that we have that will scan the devices and will generate a report for us and show it to us in the tool on the platform, as well as be able to send it via email so that we can notify our teams or everybody else and showcase how we use the Itential integration to actually make those reports meaningful to the organization, not just us. Lastly, we’re going to demonstrate how we can publish this automation via a manual method so that we can actually expose the automation to be executed on demand. We can also create an API for the automation so that it can be executed by somebody else using their scripts, and they can get their reports on demand as well, and also scheduling so that we can showcase how we can execute this, let’s say, every day at 1 AM or the like. With that said, let’s switch into the platform and see how we can go about this.

Dan Sullivan • 08:23

Okay, so now that we’re in the platform here, the first thing that we’re going to do is we’re actually going to design a golden config. For that, I’m going to go into Configuration Manager, and I am going to create a new golden configuration, and I’m going to call it, let’s see, PSIRTBundleSeptember2024, let’s do March. I’m using March PSIRTs mostly as the example for this, so I’m going to use some of those PSIRTs that were published in March, so we’re just going to do that for this. PSIRTBundleMarch2024Report, and that’s going to be very important because the automation that we built, that is reusable, we can actually point it to a golden configuration that we have designed, and we need to remember this name. This name is actually going to copy it here and save it on my handy-dandy notepad here for later because we will use this. The automation that we have designed is actually reusable, so we can point it towards any golden configuration that we had designed, and it’ll create the report dynamically at the time that we execute that automation.

Dan Sullivan • 09:30

Go ahead and create, that automation is for Cisco iOS, this one specifically. We’re actually going to keep things fairly simple, and I’m not going to get one of the things. If you have seen the Itential webinars in the past, we actually have a hierarchical capability to design the golden config trees in here that allows you to actually drill down into different levels of creating children nodes in here, so that you can create golden configs for things like regional or you can create different nodes for different platforms and different devices and so forth. Today, we’re actually going to keep this fairly simple, so I’m actually going to delete that, and we’re just going to use the base node. Since we’re just looking across the board on our devices, we want to look for these config commands that we have identified. I’m actually going to start pulling commands here from my list that I have pulled up from the March PSIRT Bundle. One of the things that we wanted to look for was Crypto Isakmp Fragmentation, which is an IKEv1 fragmentation vulnerability.

Dan Sullivan • 10:26

Let’s see, we will also have an IP DHCP snooping vulnerability. In this case, we’re not going to find any of these because we’re actually using routers for this and this is not applicable, but I’m just going to put it in there anyway so that we get an idea of how this works for all those. We also had a LISP command vulnerability. I’m just going to look for router LISP. Obviously, the devices running router LISP are not just going to remove it. Remediation strategy there has to be different, but in this case, we’re not focusing necessarily on remediation, but just an identification of the features that are enabled. Then I added a couple of the usual suspects.

Dan Sullivan • 11:02

I added HTTP server, HTTPS also being enabled, which is one of those features that not necessarily everybody runs but sometimes are enabled on the devices. I’m also going to get a little fancy here and I’m going to add a check for Telnet inside VTY lines. Sometimes you see one of those obscure ones, then we say, no, we’re not running Telnet on our devices, but it’s good to check because I’ve been surprised very, very, very many times by customers saying, we disallowed the use of Telnet 10 years ago, and we go and walk through the infrastructure and we find 100 devices running Telnet. I have added a checking here. Now, one of the things that is interesting and different here is that what this denotes is we’re using the regular expressions for this. What I have done here is I have said, and because we’re identifying vulnerable commands only, I want to look at the stuff that I don’t want to be in the environment, and we’ll do that in a second. But I actually want to check if we have transport input telnet and SSH, I have it as allow.

Dan Sullivan • 12:04

Essentially, what I’m saying here is this is a conditional statement. If SSH exists, I don’t care. That’s completely fine. By me, I only want to focus on if the command has telnet in it. Hopefully, there’s something else in there. Nobody should be configuring our login or anything like that these days, but you never know. I just want to focus on those. The other thing that we want to do here is because I am looking for the existence and I want to disallow these commands, I don’t want them to exist, is I want to flag the stuff as disallow on the platform.

Dan Sullivan • 12:34

I’m going to actually change the colors here from dark into white background just so we can see this better. I actually want to go here and say disallow. and that’ll do it for these here as well as this one. What that is saying is it’s telling the platform, whenever we walk the config of the device, if you find these specific commands, let’s flag them. They should not be on the device. That’s the logic. It’s a little inverse here. Traditionally, the use of Golden Config is we want to put every single command that we want on the device, and we’re auditing to see if they’re missing.

Dan Sullivan • 13:13

In this case, we’re actually using inverse logic. We’re using the functionality in the platform to detect the stuff that’s in the config that we don’t want. This is why we’re disallowing the majority of these commands. I am not disallowing line VTY 0-4 because we want that to exist. We want to log into the devices. I just want to disallow this sub-command, in this case, which is transport input telnet with the optional SSH parameter that I don’t care about. I will hit an update here so that we can save that.

Dan Sullivan • 13:40

Then the other thing that I got to do is add devices into this GoldenConfig. Now, this is very important, especially for this use case. I want to add devices here because what happens is the way that this automation is designed, we are actually going to only focus on the devices that are added into this GoldenConfig. We’re going to ignore the rest of the devices. As such, I need to add them in here first. That way, when I go and run my operational checks, look at my show commands and I use my command template feature for that, I can actually pass those device names to the command template. There are multiple options to address this.

Dan Sullivan • 14:20

I can actually have the user provide a list of devices in the form or I could have other mechanisms where I can read the list of devices from some source of truth, some inventory tool, I could go and query, let’s say something like a DSM or a NetBox or something like that, that I can say, hey, give me all the devices that are iOS routers and I will audit against those. Multiple options to go about this. In this case, I wanted to keep this as simple as possible because what we want to showcase is how you can do this on-demand very quickly a couple of days before the PSIRT or a couple of days after you get the PSIRT so that you can address that demand or that ask from your executives, or somebody that’s requesting a report, showcasing what devices are affected. Now, our config compliance is built. We have our devices added. We have our name recorded. I’m just going to keep that tab in there just in case.

Dan Sullivan • 15:15

But from here on out, we can actually move on and build our second piece of our checks, which is our operational commands using our command template feature. I’m actually going to jump into my project here inside of IAP, which is a self-contained set of assets that belong to the automation, and right now we’re looking at the Workflow Canvas. But before we go ahead and focus on the Workflow Canvas, I want to go ahead and build that command template that we talked about. Let’s build a new asset in here, and we’re going to build a command template, and we’re going to use the same name that we use for the golden config. They’re not going to clash because they’re different asset types, but I just want to make sure that I keep everything consistent across the board, so that when I will go ahead and execute my automation, I remember the things that I’m going to execute against. Let’s build that command template. Now, from here on, I have several commands that I have picked out on things to go ahead and check.

Dan Sullivan • 16:14

I have some commands that are very simple and some that are more complicated. I’m going to start by some of those that are a little bit simpler. I’m going to go with the usual suspects, the HTTP server. What I’m going to do is I’m actually going to copy and paste here from my handy-dandy notepad. I want to check for show IP HTTP server status, and I’m filtering this to include status only. That’s the only one that I really care about. I want to make sure that it says that if it’s enabled or not.

Dan Sullivan • 30:27

We’re going to go ahead and select our GoldenConfig and our operational checks template. Our GoldenConfig tree was called PSIRTBundle and that’s there, PSIRTBundleMarch2024, perfect. Our command template is called PSIRTBundleMarch2024Report as well. That’s also selected there and I’m just going to go ahead and click ”Run Now” and see what happens. Now we see the automation running. One of the things is that this is going to take a second because it’s actually going and executing live against the devices. It’s running against three devices, so it actually has to go and get a show run from each one of those and then compare them against the GoldenConfig.

Dan Sullivan • 36:51

Again, like I said earlier, the whole idea was to promote and give visibility to these capabilities ahead of the September PSIRT bundle for Cisco. One was released a week ago or so on the 16th, I believe it was, or a few days ago. and there’s another one coming at some point. So hopefully this is useful for those of you that are our customers and those of you looking to become our customer. So let us know how we can help with this and hopefully this was informative. Thank you and have a good rest of the day.