Question 1

How do you design an SD-WAN SLA policy that meets the latency requirement while maximizing broadband usage?

Accepted Answer

I design a **multi-transport policy** with two tunnels per site: MPLS as the fallback and dual broadband (cable + DSL/4G) as primary. In Cisco SD-WAN (Viptelo/Catalyst), I create an SLA class named `POS_CRITICAL` with latency threshold 150ms, loss threshold 0.5%, and jitter threshold 30ms. I bind this SLA class to an **App-Aware Routing policy** that steers POS traffic (TCP 443 to payment gateway IPs) through the transport that currently meets the SLA. If both broadband links violate SLA, the policy falls back to MPLS automatically. I also configure **FEC (Forward Error Correction)** on the POS application list to recover from packet loss without retransmission. For stores in rural areas with poor broadband, I add a **4G/LTE transport** with higher cost so it is only used when both fixed lines fail. The key is `preference` ordering: Broadband 1 > Broadband 2 > LTE > MPLS, with strict SLA enforcement on the POS data prefix list.

Question 2

What SD-WAN feature enables direct spoke-to-spoke tunnels, and what control-plane protocol makes it possible?

Accepted Answer

The feature is **TLOC-enabled On-Demand Spoke-to-Spoke tunnels** (also called Dynamic On-Demand Tunnels or SD-WAN Fabric Shortcuts). In Cisco SD-WAN, the control plane is **OMP (Overlay Management Protocol)**, which runs inside DTLS/TLS tunnels between vEdges/cEdges and the vSmart controllers. OMP carries TLOC (Transport Locator) routes that include the color, encap (IPsec/GRE), and public IP of each tunnel endpoint. When Site A needs to reach Site B, the vEdge at Site A receives Site B's TLOC routes from vSmart. If the policy allows `allow-service all` on the TLOC, the vEdges build a dynamic IPsec tunnel directly between their public IPs, bypassing the hub. I verify this with `show omp routes` and `show ipsec sa`. If the tunnels are not forming, I check the **control policy** on vSmart to ensure `tloc-action` is not set to `restrict`, and I verify that NAT-T is enabled for sites behind PAT. The alternative is to pre-provision **full-mesh static GRE** tunnels, but that does not scale beyond ~200 sites.

Question 3

How do you enforce centralized traffic steering in SD-WAN without breaking the dynamic spoke-to-spoke capability for non-firewalled traffic?

Accepted Answer

I use a **hub-and-spoke topology with service-side VPN chaining**. First, I segment traffic into two service VPNs: VPN 10 for `Internet_Bound` traffic that can use local breakout, and VPN 20 for `Inter_Branch` traffic that must transit the hub firewall. In the SD-WAN policy, I apply a **centralized data policy** on the vSmart controller that matches inter-branch prefixes and sets the next-hop TLOC to the datacenter hub's TLOC. This overrides the default OMP best-path and forces traffic into VPN 20 at the hub. The hub vEdge has a service-side VPN 20 interface connected to the firewall cluster; after inspection, the firewall returns traffic to the hub vEdge, which forwards it to the destination branch via VPN 20. For non-firewalled traffic (e.g., SaaS apps), I leave it in VPN 10 with local Internet breakout and dynamic spoke-to-spoke enabled. I also configure **Zone-Based Firewall (ZBFW)** on the vEdges as a secondary inspection point for traffic that bypasses the datacenter. The key design principle is: **use service VPNs as security boundaries**, not just routing instances.

Question 4

What timer controls OMP route expiration, and what is the recommended tuning for a 500-site deployment?

Accepted Answer

OMP routes expire based on the **OMP graceful restart timer** and the **TLOC holdtime**. By default, OMP routes are held for **12 hours** (`holdtime 43200`) if the vSmart connection is lost. This is designed to prevent mass blackholes during controller outages, but in your scenario it caused stale forwarding. The recommended tuning depends on the transport diversity: if sites have at least two transports (e.g., MPLS + Internet), I reduce the OMP holdtime to **1 hour** (`holdtime 3600`) because the site can still reach the secondary vSmart or rebuild tunnels via alternate transports. If sites are single-homed, I keep 12 hours but enable **BFD** on the WAN transports for faster path failure detection independent of OMP. I also enable **OMP send-path-limit** and **advertise connected** with **ecmp-limit 4** to ensure diverse paths are always known. The exact command on vEdge/cEdge is `omp graceful-restart timer 3600` under the OMP configuration. I verify with `show omp tlocs` and check the `expire` column. For a 500-site deployment, I also consider deploying **vSmart clusters** with anycast IP so controller failover is sub-second.

Question 5

What SD-WAN QoS and transport settings prioritize UDP telemetry over user web browsing?

Accepted Answer

The issue is likely that UDP telemetry is being treated as best-effort and competing with bulk HTTPS traffic on the same broadband link. I implement a **Rewriting QoS policy** on the vEdge that classifies the IoT subnet (e.g., 192.168.100.0/24) into a dedicated QoS class with strict priority. In Cisco SD-WAN, I create a **Localized Policy** with a class-map matching the sensor subnet and destination AWS IoT Core IPs (port 8883 for MQTT over TLS). I assign this to queue 0 (PQ — Priority Queue) with a guaranteed bandwidth of 20% of the link. All user web traffic goes into queue 1 (CBWFQ) with a maximum threshold. I also enable **traffic shaping** on the WAN interface to 90% of the broadband contracted rate to prevent bufferbloat in the ISP's upstream gear. For packet loss specifically, I enable **Packet Duplication** for the IoT application list across two transports (MPLS + Internet) so if one path drops a packet, the duplicate arrives via the other path. This is overkill for most apps but standard for industrial IoT. I verify drops with `show interface tunnel statistics` and `show policy-map interface`.

Question 6

What are these ports, and how do you secure them without breaking SD-WAN overlay connectivity?

Accepted Answer

These are the default **SD-WAN tunnel ports**: 12346 for DTLS, 12446 for TLS, and 12646 for IPsec ESP in some firmware versions. They must be open for the vEdge to form control and data plane tunnels with vSmart and other vEdges. To secure them, I implement three controls: (1) **Restrict the TLOC color** so internet-facing tunnels only accept connections from known peer public IPs using a `control-policy` on vSmart with `sequence match source-ip` from an allow-list of all branch public IPs. (2) **Change the default ports** to non-standard high ports using `control-port` and `crypto` configuration to reduce automated scanning exposure. (3) **Deploy a CGNAT or stateful firewall** in front of the vEdge that only permits UDP 12346 from the datacenter's public IP ranges. Critically, I never expose the vEdge's management plane (HTTPS/SSH) to the internet — management uses the out-of-band VPN0 or a dedicated MGMT VRF via MPLS only. I also verify with `show control local-properties` that the device is in `vbond-provisioned` mode and not running default credentials.

Question 7

What are the three realistic integration options, and which do you recommend for fastest unified policy deployment?

Accepted Answer

Option 1: **Rip-and-replace** the VeloCloud edges with Cisco cEdges. Fastest for policy unity but expensive and disruptive. Option 2: **Interoperability via IPsec tunnels** between VeloCloud Hubs and Cisco vEdges, treating each SD-WAN as a separate AS. This preserves both fabrics but requires manual route redistribution and no unified policy. Option 3: **Unified orchestration layer** using Cisco Nexus Dashboard or a multi-vendor SD-WAN orchestrator like Versa or Aruba EdgeConnect that abstracts both underlays. For the 90-day constraint, I recommend Option 2 as the immediate bridge, followed by a phased migration to Option 1. Specifically, I establish IPsec GRE tunnels from VeloCloud gateways to Cisco hub vEdges, redistribute OMP routes into BGP at the Cisco side, and use a centralized prefix list to control what each fabric learns. For unified visibility, I deploy a **single Splunk/DNA Center** instance that ingests syslog and SNMP from both platforms. The key interview phrase: "I never promise a single-pane-of-glass for multi-vendor SD-WAN in 90 days — I deliver secure interoperability first, then migrate."

Question 8

How does Cisco SD-WAN App-Aware Routing handle this, and what specific policy configuration steers video traffic to LTE before fiber congests?

Accepted Answer

App-Aware Routing does not react to congestion — it reacts to **SLA violations**. If the fiber link is congested but latency is still under the SLA threshold, SD-WAN will keep video on fiber. To fix this, I implement **BFD-based path liveness** with aggressive timers AND an **SLA class that monitors packet loss**, not just latency. Video traffic (classified by App-ID as `microsoft-teams` or `zoom`) is matched in a policy sequence with an SLA class `VIDEO_QOS` that has a loss threshold of 0.1%. When the large file sync causes queuing drops on fiber, the loss threshold triggers, and App-Aware Routing steers new video flows to LTE. For existing flows, I enable **Bidirectional Forwarding Detection (BFD)** on the application path so that if the fiber transport fails BFD, all flows are moved immediately. I also configure **per-flow load-balancing** (`ecmp-dampening`) so that video flows are pinned to one transport while bulk flows use the other. The critical config is `set preferred-color lte biz-internet` with `set fallback-to-lte` in the app-route policy. I verify with `show app-route stats` and `show bfd sessions`.

Question 9

What causes a Site-ID mismatch, and what is the correct provisioning workflow to prevent it during zero-touch deployment?

Accepted Answer

A Site-ID mismatch occurs when the **chassis number / UUID** of the vEdge is associated with a different Site-ID in vManage than what is configured locally on the device. This commonly happens when a device is RMA'd and the old chassis record is not removed from vManage, or when the bootstrap config has the wrong `system-ip` or `site-id`. The fix is to delete the stale device record in vManage (Administration > Device Management), re-add the device with the correct chassis/serial, and push a new provisioning file. For ZTP prevention, I use a **golden template workflow**: (1) Generate the device certificate from vBond before shipping; (2) Embed the `cloud-init` or `OTP` file with the correct `system-ip`, `site-id`, `organization-name`, and `vbond` FQDN; (3) On first boot, the vEdge contacts vbond, authenticates with the certificate, and vManage provisions it into the correct site using the serial-to-site mapping in the device inventory. I also enable **validated smart licensing** so unauthorized devices cannot join the fabric even with valid certs. I verify with `show certificate` and `show control local-properties` on the vEdge, and `Monitor > Devices` in vManage.

Question 10

What are the two tunneling options to Zscaler from SD-WAN branches, and which provides better failover?

Accepted Answer

Option 1: **IPsec tunnels from each branch vEdge directly to Zscaler**. This is simple but requires 200 IPsec SAs on the Zscaler side and does not leverage SD-WAN intelligence. Option 2: **IPsec GRE tunnels from regional hub vEdges to Zscaler**, with branches sending internet traffic to the hub via the SD-WAN fabric. This centralizes the Zscaler tunnels but introduces a hairpin. The best option is **Option 3: Zscaler Internet Access (ZIA) via Tunnels from the vEdges using Zscaler AppConnect or IPsec with SD-WAN service chaining**. Cisco SD-WAN supports native **Zscaler API integration** where vManage pushes tunnel configs to Zscaler automatically. For failover, I configure **two Zscaler data centers** (e.g., London + Frankfurt) as primary and secondary tunnel destinations. App-Aware Routing monitors both tunnels with BFD; if the primary Zscaler tunnel fails, traffic fails over to the secondary in under 3 seconds. I also enable **local Internet breakout** for trusted SaaS apps (Office 365, Salesforce) using O365 URL lists to avoid sending that traffic through Zscaler, reducing latency and Zscaler licensing costs.

Question 11

How do you enforce data residency in SD-WAN without deploying separate appliances per country?

Accepted Answer

I use **Localized Data Policy** with geolocation-based prefix lists. First, I obtain the IP ranges of all UAE government services and create a localized policy on the Dubai vEdge that matches these destinations. The policy action is `accept` with `set service local` — meaning traffic exits directly via the local internet transport and is never tunneled back to the European datacenter. For global SaaS, I create a second policy sequence matching O365 and Salesforce URL/FQDN categories with `set service vpn 0` (local breakout). For inter-branch traffic within the UAE, I enable spoke-to-spoke TLOCs within the `biz-internet` color but restrict them to a regional TLOC list. The critical piece is **Geo-DNS steering**: I configure local DNS forwarders on the vEdge so that `gov.ae` resolves to in-country CDN edges, while `microsoft.com` resolves to the nearest global anycast node. I also enable **NetFlow/ipfix** on the local interfaces to prove compliance to auditors. All policy is pushed from vManage as a device-specific template attached to the Dubai site group.

Question 12

What is the root cause, and what is the rollback procedure?

Accepted Answer

The root cause is almost always a **root CA certificate mismatch** or **CRL/OCSP unreachability** introduced by the firmware upgrade. Newer firmware may use a different cipher suite or require SAN (Subject Alternative Name) fields that the old device certificates lack. I immediately check `show certificate` on a failing vEdge and compare the root CA fingerprint with the vSmart's advertised CA. If they differ, I verify whether the upgrade included a CA rotation. The rollback procedure is: (1) Roll back the vSmart firmware via vManage to the previous version; (2) If the CA was rotated intentionally, re-provision device certificates using the **vManage PKI re-sign workflow**; (3) For urgent restoration, I temporarily disable certificate validation on the vEdge with `no control dtls tls-auth` (only in lab/emergency — never in production permanently). To prevent recurrence, I always stage firmware upgrades in a **device template test group** with 5% of devices before fleet-wide rollout. I also maintain a **certificate expiry dashboard** in vManage to catch expiring certs 30 days in advance.

Question 13

Why does traceroute fail after the second hop, and what SD-WAN behavior causes this?

Accepted Answer

Traceroute fails because SD-WAN **encapsulates traffic in IPsec/GRE tunnels**, and intermediate hops (ISP routers, MPLS P routers) do not decrement the TTL of the inner packet — they decrement the TTL of the outer tunnel header. When the inner TTL expires inside the tunnel, the tunnel endpoint (the next vEdge or hub) returns the ICMP Time Exceeded, but many SD-WAN implementations do not generate or forward this ICMP back to the source correctly. Additionally, if the vEdge has **`ip mtu` or TCP MSS clamping** set incorrectly, large traceroute packets may be silently dropped. To verify connectivity properly in SD-WAN, I use `traceroute` with the `no-resolve` option and small packet sizes, or I run `ping` with sweep sizes to find the path MTU. The authoritative SD-WAN tool is `traceroute mpls ipv4` or using the vEdge CLI `traceroute vpn 0 8.8.8.8` which sends probes through the tunnel and reports the tunnel endpoint as a single hop. For troubleshooting, I also use `show tunnel statistics` and `show bfd history` to confirm transport health without relying on traceroute.

Question 14

What Cisco solution integrates SD-WAN with AWS, and how does routing work between the on-prem fabric and AWS VPCs?

Accepted Answer

The solution is **Cisco SD-WAN Cloud OnRamp for IaaS** (formerly Cloud Interconnect). It deploys a virtual vEdge (vEdge Cloud) or Catalyst 8000V in the AWS VPC as an EC2 instance. This virtual router forms OMP peering with the on-prem vSmart controllers over the internet or AWS Direct Connect. Routing works as follows: (1) The AWS vEdge advertises VPC CIDRs into OMP; (2) On-prem vEdges learn these routes and treat the AWS VPC as another site; (3) Traffic from branches to AWS flows through the SD-WAN fabric to the AWS vEdge, which decapsulates and forwards into the VPC. For high availability, I deploy two vEdges in different Availability Zones with **OMP graceful restart** and **BGP peering to AWS Transit Gateway** as a backup path. I also use **AWS Route 53 Resolver endpoints** so that on-prem DNS can resolve private VPC hostnames. The security policy is unified: a branch in London and an EC2 in `eu-west-2` both belong to the same service VPN and are governed by the same vSmart data policy. I verify with `show omp routes` on the AWS vEdge and `show ip route vrf 10` on the branch vEdge.

Question 15

What three metrics do you present, and how do you collect them automatically from the SD-WAN fabric?

Accepted Answer

Metric 1: **Transport Cost Per Mbps**. Before SD-WAN, I collected MPLS invoices showing $400/Mbps/month. After SD-WAN, broadband + LTE averages $40/Mbps/month. I collect this from vManage's **Transport Interface Statistics** correlated with ISP billing APIs. Metric 2: **Application Experience Score (Apdex)**. I measure SaaS app latency and availability before/after. vManage collects this via **App-Aware Routing stats** and I export to Splunk for trend analysis. A score improvement from 0.65 to 0.92 proves user experience gains. Metric 3: **Mean Time to Repair (MTTR)**. Before SD-WAN, WAN outages required ISP tickets taking 4+ hours. With SD-WAN, transport failover is automatic; MTTR drops to under 30 seconds. I collect this from vManage **Alarms & Events** and calculate the delta between `interface-down` and `tunnel-recovery` timestamps. For the sponsor dashboard, I use vManage's **built-in reports** or export via REST API to a Grafana dashboard. I also track **policy compliance percentage** — what % of sites are running the latest approved policy template — to prove operational control at scale.