In the financial services industry, routine infrastructure maintenance is essential for maintaining security and meeting regulatory compliance requirements. Yet, infrastructure teams often face a complex challenge: developing and validating automation that must be fast, reliable, and scalable, all while minimizing operational risk.
One of our largest banking customers encountered this challenge firsthand. They needed a way to safely test and update critical automation already in production. However, the only opportunity to run that automation at full scale was during a live production maintenance window where the stakes were high and the room for error was zero. Without access to a production-scale testing environment, proposed improvements to the automation remained unverified and potentially risky. Sound familiar?
At Itential, we don’t just deliver software, we work side-by-side with our customers as trusted partners. In this case, our close collaboration allowed us to design the right solution for their unique business and technical needs.
We had previously partnered with this bank to build an automation capable of executing commands across hundreds of devices in parallel. Speed and reliability were key. The automation needed to run multiple times within each limited maintenance window without fail.
The Challenge: Testing Improvements at Scale
While the bank maintained development and QA environments, these lacked the scale of production (specifically, hundreds of network devices), making it difficult to evaluate performance optimizations. Building and maintaining a lab of that size would require significant investment in both hardware and personnel.
The Solution: A Temporary, Scalable Testing Environment
To meet this need, the organization collaborated with Itential to design and implement a temporary, production-like test environment. This on-demand environment enabled both the customer and Itential to safely test performance improvements at scale before deploying them to production.
Key Components of the Solution:
🧪 Creating Temporary Production-Like Environments
Itential built an environment that closely mirrored the organization’s hardware and network configuration. This included a combination of AWS EC2 instances and virtual machines hosted in Itential’s on-premises performance lab. The result was a realistic, scalable environment for performance testing and configuration tuning.
🛠️ Simulating Network Devices with Containerlab
Using Containerlab, Itential created a mock network topology that reflected the customer’s production environment. This allowed for realistic behavior testing without the need for physical devices.
🚀 Testing & Optimizing Automation Performance
Itential identified a range of potential enhancements to improve automation efficiency from adapter configuration tweaks to input file formatting strategies. Each change was evaluated for impact on speed and reliability.
The Outcome: Confidence Through Consistent Testing
With a scalable and repeatable testing process in place, the organization gained confidence in its ability to safely evolve and optimize its automation. Itential’s recommendations led to more stable automation behavior and the potential to reduce runtime by approximately 10%.
This approach not only reduced risk during maintenance windows but also demonstrated the power of dynamic, on-demand environments for validating automation in highly regulated industries.
Looking to increase confidence in your automation?
Let’s explore how scalable testing environments can help you innovate faster and deploy with peace of mind.
Nick Andreano
Platform Engineer ‐ Itential
Nick is a Platform Engineer at Itential, responsible for building and maintaining the core infrastructure and tools that empower development teams to efficiently build, deploy, and operate applications.
