A cyber incident can leave applications and data intact while making the infrastructure behind them unsafe to operate. Compromised administrator credentials, altered firewall rules, or an untrusted network controller can interrupt the connections and controls that critical business services depend on.
Recovery means re-establishing those infrastructure functions using approved configurations, appropriate operating authority, and evidence that required services work. It reaches beyond restoring a server or switching traffic to a second site.
This guide explains how infrastructure and security leaders can prepare for that situation, and how Itential can coordinate recovery across existing controllers, automation, approval systems, and teams. Including a financial services recovery program that provides a running customer example of the architecture and operating requirements.
The recovery objective is a working business service supported by verified infrastructure functions.
If production management can no longer be trusted, can we restore the infrastructure our critical services need, under controlled authority, and demonstrate that it works?
Infrastructure resilience supports the business through disruption. It depends on reducing exposure, controlling everyday changes, maintaining service dependencies, and restoring operations when an incident gets through. Recovery is the point at which those disciplines must work together under pressure.
Consider an attacker who obtains privileged access to network management. The organization may need to treat controller settings, device configurations, automation credentials, or recent backups as suspect. A healthy second environment does not resolve that uncertainty if it shares those dependencies or automatically receives the same altered state.
Other failures can produce related recovery requirements: an unavailable controller, a damaging configuration change, or loss of shared identity services. The response depends on what failed and what remains trusted. A cyber incident adds a consequential decision about which state and operating authority are acceptable.
Vulnerability analysis, validated patching, configuration compliance, and certificate management reduce exposure and help keep recovery usable. AI-assisted vulnerability discovery adds pressure to the work of assessing impact and deploying changes safely. The recovery obligation remains even when those preventive processes are effective.
Leadership needs both a way to reduce the likelihood of disruption and a way to restore services when normal operating assumptions no longer hold. Recovery planning should identify the infrastructure needed by each critical service and the conditions under which it can safely return.
Trusted recovery is the controlled restoration or activation of infrastructure using an approved recovery state, appropriate access and execution authority, and functional acceptance tests. Trust is established against defined criteria and evidence. It is not implied by a green dashboard or the age of a backup.
A recovery program must therefore answer three questions:
A recent copy can preserve an unwanted change as faithfully as a valid one. Recovery policy must determine which state is acceptable to use.
A financial services organization is developing an isolated recovery environment for critical network services. The program addresses the possibility that production infrastructure cannot be trusted after a cyber event. Its objective is an authorized, automated activation process with no undocumented recovery steps.
The scope crosses routing and switching, firewalls, load balancing, DNS, management access, certificates, proxy services, compute, edge services, and monitoring. The requirement is functional service readiness across those domains, with an approved configuration and an accountable owner for each operation.
In the financial services example, Itential Platform runs in both production and the recovery environment. A Recovery Control Plane, the Itential-based layer that coordinates recovery activation, ties the two together. ServiceNow is the entry point for governed requests. Versioned configurations, software, and recovery artifacts are held in repositories and transferred through a constrained artifact path.
A central design decision keeps device credentials out of the external Recovery Control Plane. Execution uses site-local authority inside the recovery environment. Where domain controllers own configuration and restoration, Itential orchestrates through them; other components require directly managed automation.
The financial services example defines four recovery operations, each with a distinct purpose:
| Operation | Defined Purpose |
|---|---|
| Apply Changes | Coordinate approved changes across production and recovery, with domain-specific execution. |
| Capture & Synchronize | Collect configurations and backups, reconcile changes, and maintain recovery assets. |
| Rollback | Return a domain to a selected approved recovery point after a bad change or incident. |
| Rebuild & Activate | Sequence infrastructure reconstruction, supporting tools, verification, and service activation. |
Some recovery functions deliberately use a different vendor from production. That requires equivalent policy and service behavior across different implementations, rather than copying configuration syntax.
The program is being developed. Its documented architecture and agreed operating decisions provide the customer example in this guide; full activation, recovery timing, and remaining domain procedures still require validation.
The Itential Platform provides the coordination layer across systems that each own a part of recovery. A request becomes a controlled sequence with explicit dependencies, a selected recovery version, responsible executors, and checks before the next stage can proceed.
The customer architecture separates recovery coordination from site-local credentials and domain execution. The design must also define operation when central services are lost.
Fabric and security controllers (for example SDN and firewall controllers) retain supported configuration, backup, and restore responsibilities. Itential invokes their operations and evaluates results. For components without a suitable controller, workflows and Itential Gateway can coordinate and run local scripts, playbooks, or other supported automation.
Each recovery task needs prerequisites, approved inputs, an execution owner, an expected result, and a failure response. Out-of-band access and local identity may be required before controllers can be repaired. Compute may be required before a proxy or monitoring platform can be installed.
The recovery coordinator itself has dependencies. Local policy, execution assets, credentials, and protected artifacts must support the defined failure scenario. The customer program identifies the bring-up of Itential within the isolated environment as a design item that must be resolved and tested.
In the customer program, controller backups, device configurations, repository versions, and recovery manifests serve different purposes. Itential must coordinate operations against explicitly selected inputs; it should not silently decide which conflicting record represents an acceptable recovery state.
What was approved?
Policies and intended settings
AUTHORITY: Designated policy owner
What is running?
Device and controller evidence
AUTHORITY: Live domain systems
What may we activate?
Approved assets and test results
AUTHORITY: Recovery policy
Define the authority and promotion rules that connect approved intent, running conditions, and recovery assets.
A recovery package should identify the intended configuration, software versions, required images, dependencies, site parameters, and validation criteria. A backup that cannot be read by the available controller version is not a usable restoration input. Configuration ownership and software compatibility need to be known before activation.
Customer security and infrastructure owners define acceptable sources, integrity checks, approval rules, and protected recovery generations. Itential can coordinate those checks, retain references to the selected artifacts, invoke the authorized operations, and route failures for review.
The program uses one-way artifact transfer into recovery. That constrains connectivity; it does not by itself establish the integrity or suitability of each artifact. Promotion policy and local validation must determine whether received content becomes eligible for recovery.
Who approves the state, who executes each operation, and who accepts the service result.
The customer program treats readiness as continuing work. Production changes, new services, software versions, and expiring certificates can leave an isolated environment unable to support the business even when its hardware is healthy.
The general change flow begins with one ServiceNow request. Production Itential applies the approved operation, and the recovery execution follows after production succeeds. Itential coordinates the appropriate domain controller or local automation at each target and retains the result against the request.
This ordering is a program pattern, not a universal rule for every technology. Domain-specific flows and permitted access into the isolated environment must be reconciled with the isolation policy. The essential contract is that each target result is visible and an incomplete recovery update becomes a managed exception.
Scheduled collection complements per-change propagation. It captures backups and differences that occur outside the normal request flow. The program includes recurring capture of production and recovery state, with DNS records providing a practical example of changes that may arrive through other systems.
Collected state must be evaluated against recovery policy. A recent snapshot may show what changed without proving that the change should be activated after a cyber incident. Retain approved generations and distinguish a capture timestamp from an acceptance decision.
The program includes different implementations for selected recovery functions. A request to permit a service path must become the appropriate policy on each target. Itential can coordinate target-specific actions, while the domain design supplies the translation and acceptance rules.
Check three results: the controller accepted the change, the resulting object expresses the intended policy, and the required service behavior passes. Translation feasibility and feature differences must be validated per domain; successful API responses do not resolve them.
Certificate issuance, renewal, installation, and authentication tests belong in the readiness process. The same applies to software images, controller exports, local secrets, and supporting compute. Itential can coordinate these operations through defined integrations and workflows, with the exact coverage established during implementation.
If production succeeds and recovery fails, record the gap, assign an owner, and apply a defined risk response. A working production change should not be reversed automatically without considering the consequences.
The customer program defines a bring-up sequence spanning access, orchestration, network functions, supporting tools, and edge activation. Each stage has defined inputs, execution responsibilities, and acceptance criteria that must be exercised together.
Each stage advances only after its defined prerequisites and acceptance checks pass. Independent domain operations can run in parallel when their dependencies allow.
Leadership authorizes the recovery scope under the incident process. The selected recovery version identifies eligible artifacts and operations. Itential coordinates checks for local management access, credentials, software availability, and reachable execution systems before dependent changes begin.
The customer sequence brings up access control and fabric functions before dependent services. Itential invokes supported controller operations or site-local automation according to the domain ownership model. Parallel work is appropriate only when prerequisites and failure containment are explicit.
Monitoring recovery may require fresh installation, configuration, database restoration, and discovery. The customer design gives this its own stage. After required functional tests, controlled edge or traffic activation can proceed under the defined release authority.
A recovery workflow needs more detail than a list of tools to restart. Itential coordinates the method defined for each component, with the relevant software, state, authority, and verification inputs.
| Component Condition | Execution Approach | Evidence Before Advancing |
|---|---|---|
| Standing & Trusted | Validate existing state and activate required functions. | Required identity, policy, reachability, and service checks pass. |
| State Requires Restoration | Invoke a supported controller restore or local automation using the selected recovery assets. | Compatible versions, restored configuration, and functional tests. |
| Management Stack Requires Reconstruction | Rebuild with approved images and manifests, then apply eligible configuration. | Successful bootstrap, local access, and dependent-service readiness. |
| Supporting Tools Require Fresh Deployment | Install software, apply configuration, restore required databases, and run discovery. | Reachable targets and usable monitoring or validation results. |
The customer design includes two-person approval, scoped release inputs, dependency and policy validation, and evidence collection. These are controls to implement in the recovery operation. They do not establish that every approval workflow is already complete or that security decisions can be delegated without a policy.
Itential workflows should make the authorized scope and selected recovery version visible, evaluate required checks, and prevent dependent steps from advancing when a gate fails. Customer-defined permissions, secrets handling, and site-local execution boundaries govern who or what can perform each operation.
AI may assist with interpreting incident information, identifying dependencies, or preparing a recovery plan. Such assistance should feed the same approval and execution controls as other inputs. AI-assisted planning is an optional extension of this approach, rather than a prerequisite for executing the defined recovery workflows.
The recovery sequence can use deterministic workflows for defined operations. Any added reasoning capability requires scoped tools, appropriate oversight, and separate validation of the decisions it is allowed to influence.
Extensive automation can reduce manual coordination. Decisions about acceptable recovery state, destructive reconstruction, and service activation still need accountable authority.
The most consequential part of a recovery operation is often the failed branch. Itential should coordinate a defined response rather than treat an incomplete step as a successful domain recovery or continue through dependent operations.
| Failure Condition | Defined Orchestration Response | Accountable Decision |
|---|---|---|
| Recovery Update Fails After Production Succeeds | Retry eligible transient errors or retain a tracked drift exception; hold dependent promotions where required. | Domain owner determines the acceptable gap and resolution deadline. |
| Artifact or Policy Check Fails | Reject the input for that operation and stop affected branches. | Security and recovery owners approve an alternative state or remediation. |
| Controller Restore Fails | Retain error evidence and invoke only a validated recovery method. | Domain owner selects repair, reconstruction, or escalation. |
| Functional Test Fails | Keep dependent traffic activation closed and collect diagnostic results. | Service and domain owners decide the next authorized action. |
| Central Coordination is Unavailable | Use the independently tested local recovery path defined by policy. | Recovery leadership confirms local authority and scope. |
Identify which operations can be retried, which require a state check first, and which could duplicate or destroy work if repeated. A recorded checkpoint should state what actually completed and what remains eligible to run. Recovery after an interrupted workflow must be tested against real domain behavior.
A suspected component may require clean reconstruction, but wiping everything by default can remove evidence or damage functioning dependencies. Incident leadership, security teams, and domain owners must determine the permitted method. The workflow executes the authorized choice and records its result.
The customer program has open implementation decisions, including some rollback procedures, vendor translation, and local platform bring-up. These are concrete acceptance items for the program, rather than details that orchestration makes disappear.
Leadership should ask for evidence of specific recovery capabilities. The existence of workflows, backups, or a secondary environment is an input to that assessment. Demonstrated service behavior is the result that matters.
Leadership can test infrastructure recovery readiness with four exercises; for each, request the evidence listed below.
| Exercise | Evidence to Request |
|---|---|
| An Approved Change | Request, production and recovery results, functional checks, and the failed branch. |
| An Out-of-Band Change | Detection, reconciliation decision, promotion or rejection, and tracked exceptions. |
| Loss of a Domain Controller | Independent access, compatible artifacts, successful restoration or reconstruction, and dependent-service tests. |
| Loss of Production Management | Site-local authorization, bootstrap, domain recovery, and end-to-end readiness under the stated failure conditions. |
A useful scorecard records the service scope, dependency independence, accepted recovery version, unresolved drift, functional test results, elapsed exercise time, and manual interventions. Each gap needs an owner and a next action. A passed test in one domain should not conceal an untested dependency in another.
Use these five questions to assess whether your organization can recover critical infrastructure after a cyber incident:
Choose a service with meaningful network and security dependencies. Map its required infrastructure, identify which production services may be lost, and name the accountable owner. Include both a controller-managed function and a component requiring local automation so the exercise tests coordination across execution models.
Use Itential to connect authorization, artifact selection, prerequisites, domain execution, a failed branch, and service verification. Measure what the exercise demonstrates and what assumptions it still relies on. Expand only after the complete request-to-verification operation has been exercised successfully.
The scope follows the critical service. It may include routing, switching, firewalls, DNS, load balancing, certificates, management access, compute supporting infrastructure tools, and monitoring. The guide focuses on those functions; application and data recovery must coordinate with their own owners and plans.
A second site may share credentials, management systems, repositories, or altered configurations with production. It can be available while still depending on the failed or compromised environment. The design must establish which dependencies remain acceptable and verify required service behavior.
No. It means recovery state, access, execution, and service results are assessed against explicit security and operational criteria. Customer security owners and incident leadership retain the decisions about acceptable risk and activation. Orchestration coordinates the checks and execution that support those decisions.
Itential coordinates authorization, dependencies, supported controller operations, local automation, exception handling, and validation. Existing domain tools retain their supported recovery functions. Protected repositories supply recovery assets, and customer owners define the state and tests acceptable for use.
That requires an explicitly designed and tested local operating capability. Local authority, policy, artifacts, credentials, and execution assets must survive the failure scenario. The architecture should identify the minimum required footprint and demonstrate its bootstrap and recovery sequence.
Defined execution steps can be automated extensively. Zero-click recovery is an outcome to demonstrate within an approved scope, rather than an assumption. Trust decisions, exceptions, destructive actions, and activation authority need policies and accountable owners even when workflows execute the work.
Select one critical service and define the disruption it must survive. Map its dependencies, assign owners, establish approved recovery assets, and exercise the complete operation with a failed branch. Use the evidence to prioritize the next integration, domain, or service.
See how Itential connects AI reasoning to governed execution across your entire infrastructure.