Automatic Cisco XR provisioning with ZTP

This article explores how to optimize the provisioning of a hundred Cisco XR devices so that they are configured with minimal human intervention. Depending on the context, the devices might be shipped directly from the factory and configured automatically on-site, or more likely, they could be automatically configured in a lab environment, verified, and then shipped for installation. We will rely on what is known as ZTP or Zero Touch Provisioning. The ZTP protocol allows configuring a factory-delivered device without any human intervention. ZTP is vendor-specific, and specifically on Cisco XR it requires:

setting up a DHCP server to assign an IP address to the devices;
setting up a web server that generates configurations for each device;
powering on the devices so they can acquire an IP address via DHCP and reach the web server.

In the scenario we envision, we set up a lab environment where the devices will be unboxed, automatically configured, packaged, and shipped to the final destination, where non-technical personnel will proceed with physical installation, network cable connections, and power-on. We assume no technical skills at remote sites and no ability to send technical personnel.

Let’s delve into the details.

Lab Network Configuration

We set up a lab environment using a Linux system with two network interfaces: the first pnet0 will connect the system to the corporate network, and the second pnet9 will be connected to a dedicated and isolated network. The devices to be configured will then be connected to a dedicated physical switch, whose gateway is the pnet9 interface of the Linux system.

On the Linux system, we’ll use Dnsmasq configured as follows:

port=0
interface=pnet9
dhcp-range=169.254.1.0,169.254.1.253,30d
dhcp-option=67,http://169.254.1.1:8080/ztp
log-dhcp

Dnsmasq will serve the 169.254.1.0/24 network by assigning free IPs to clients and maintaining the assignment for 30 days. DHCP requests will include the URL for ZTP auto-configuration. DHCP requests will be logged via syslog, typically saved in /var/log/syslog.

Configuration Setup

The web server, reachable at the configured address in the previous paragraph, must:

make a Bash script available to all devices;
generate a specific configuration for each device;
make the correct configuration available to each device.

We’ll use Flask to create a small web application capable of dynamically generating configurations for each device. The application will respond to two URLs:

/ztp (GET), which will make the Bash script available to Cisco XR devices;
/config (POST), which will make the specific configuration available to each device that must send its serial number.

Once ready, the Flask application can be run with:

flask --app ztp run -p 8080 -h 169.254.1.1

Ensure the application is listening on the correct interface and that both URLs respond correctly:

wget -q -O- http://169.254.1.1:8080/ztp
wget -q -O- --post-data="serial=1234" http://169.254.1.1:8080/config

The /ztp URL will return text similar to the following, identical for all devices:

#!/bin/bash

export CONFIG_FILE="/tmp/config.txt"
source /pkg/bin/ztp_helper.sh

SN=$(dmidecode | grep -m 1 "Serial Number:" | awk '{print $NF}')
PN=$(xrcmd "show inventory location 0/RP" | grep -m1 "PID" | awk '{print $2}')
RESULT=$(wget -O- --post-data="serial=${SN}&model=${PN}" {{ url }} > $CONFIG_FILE)

xrapply_with_reason "Initial ZTP configuration" $CONFIG_FILE

After acquiring an IP via DHCP, the device will retrieve the described script. This script fetches the serial number (via dmidecode) and the model (via show inventory). Using these parameters, the device requests the second URL to use as its initial configuration.

The /config URL will return, based on the parameters passed in POST, the minimal configuration to make the device accessible via SSH:

username cisco
 group root-lr
 password 0 cisco
!
hostname device01
!
domain name example.com
!
vrf OOB address-family ipv4 unicast
!
router static vrf OOB address-family ipv4 unicast
    0.0.0.0/0 169.254.1.1
!
interface MgmtEth0/0/CPU0/0
 vrf OOB
 ipv4 address 169.254.1.11 255.255.255.0
 no shutdown
!
ssh server v2
ssh server vrf OOB
!
line default
 transport input ssh

Power On and ZTP Process Verification

We’ve reached the final but most critical step: powering on the device and verifying that the ZTP process works correctly. When starting a factory-new Cisco XR device, the ZTP process should start immediately after the cryptographic features legal notice:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply third-party
authority to import, export, distribute or use encryption. Importers,
exporters, distributors and users are responsible for compliance with
U.S. and local country laws. By using this product you agree to comply
with applicable laws and regulations. If you are unable to comply with
U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be
found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.



RP/0/RP0/CPU0:Oct 18 08:11:22.659 UTC: ifmgr[363]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Down
RP/0/RP0/CPU0:Oct 18 08:11:22.661 UTC: ifmgr[363]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Up
RP/0/RP0/CPU0:Oct 18 08:11:29.975 UTC: pyztp2[330]: %INFRA-ZTP-4-CONFIG_INITIATED : ZTP has initiated config load and commit operations
RP/0/RP0/CPU0:Oct 18 08:11:47.203 UTC: pyztp2[330]: %INFRA-ZTP-4-CONFIG_FINISHED : ZTP has finished config load and commit operations
RP/0/RP0/CPU0:Oct 18 08:11:53.034 UTC: pyztp2[330]: %INFRA-ZTP-4-PROVISIONING_COMPLETED : ZTP has successfully completed the provisioning
RP/0/RP0/CPU0:Oct 18 08:11:59.329 UTC: pyztp2[330]: %INFRA-ZTP-4-EXITED : ZTP exited

If the ZTP process has worked correctly, the DHCP server will have assigned an IP:

2023-10-18T11:32:42.306237+02:00 kali dnsmasq-dhcp[3818]: 548912439 vendor class: PXEClient:Arch:00009:UNDI:003010:PID:N540-ACC-SYS
2023-10-18T11:32:42.306699+02:00 kali dnsmasq-dhcp[3818]: 548912439 user class: xr-config
2023-10-18T11:32:42.306856+02:00 kali dnsmasq-dhcp[3818]: 548912439 DHCPDISCOVER(pnet9) 40:14:82:c1:11:11
2023-10-18T11:32:42.307036+02:00 kali dnsmasq-dhcp[3818]: 548912439 tags: pnet9
2023-10-18T11:32:42.307208+02:00 kali dnsmasq-dhcp[3818]: 548912439 DHCPOFFER(pnet9) 169.254.1.11 40:14:82:c1:11:11
2023-10-18T11:32:42.309068+02:00 kali dnsmasq-dhcp[3818]: 548912439 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
2023-10-18T11:32:42.309925+02:00 kali dnsmasq-dhcp[3818]: 548912439 requested options: 15:domain-name, 6:dns-server, 12:hostname,
2023-10-18T11:32:42.310094+02:00 kali dnsmasq-dhcp[3818]: 548912439 requested options: 67:bootfile-name, 43:vendor-encap, 143
2023-10-18T11:32:42.310183+02:00 kali dnsmasq-dhcp[3818]: 548912439 next server: 169.254.1.1
2023-10-18T11:32:42.310289+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  1 option: 53 message-type  2
2023-10-18T11:32:42.310366+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option: 54 server-identifier  169.254.1.1
2023-10-18T11:32:42.310433+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option: 51 lease-time  30d
2023-10-18T11:32:42.310531+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option: 58 T1  15d
2023-10-18T11:32:42.311451+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option: 59 T2  26d6h
2023-10-18T11:32:42.311641+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option:  1 netmask  255.255.255.0
2023-10-18T11:32:42.311775+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option: 28 broadcast  169.254.1.255
2023-10-18T11:32:42.312018+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size:  4 option:  3 router  169.254.1.1
2023-10-18T11:32:42.312554+02:00 kali dnsmasq-dhcp[3818]: 548912439 sent size: 25 option: 67 bootfile-name  http://169.254.1.1:8080/ztp

And the web server will have been contacted on the two URLs:

 * Serving Flask app 'ztp'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://169.254.1.1:8080
Press CTRL+C to quit
169.254.1.11 - - [18/Oct/2023 14:47:06] "GET /ztp HTTP/1.1" 200 -
169.254.1.11 - - [18/Oct/2023 14:47:33] "POST /config HTTP/1.1" 200 -

At this point, the device will be reachable and ready to be configured using secondary automation systems like Ansible.

Development and Troubleshooting

The ZTP process described might appear simple and straightforward. However, it actually required several hours of testing to understand where and how the ZTP process was stalling. Let’s see what tools we have for troubleshooting.

The ZTP status can be seen using the show ztp status command:

State             : Terminated
Current Fetcher   : No active fetcher

The ZTP process runs only if the router has factory settings; otherwise, it’s skipped. If our router isn’t factory-new, we have two options, both useful.

We can force the ZTP process from the command line:

ztp initiate

Or we can, also from the command line, delete the configuration and reset ZTP:

ztp clean
configure terminal
commit replace
reload

ZTP process logs are stored on the device and can be viewed with show ztp logging. However, relying solely on these logs is inconvenient, considering that each boot takes several minutes. The best solution I found was to use the Bash available on Cisco XR devices and manually invoke the script:

wget -O- http://169.254.1.1:8080/ztp | bash -x

The above commands, typed from the console, open a Bash instance and execute the script made available by our developed application. Upon completion, the device should be configured as expected.