Using pyATS and Genie for automated network validation

Most teams meet pyATS as a way to pull structured data off devices. That is useful, but it undersells the point. The interesting part is that Genie turns raw CLI output into structured models you can assert against — which is what makes a validation harness possible.

From parsing to assertions

A classic script prints show ip bgp summary and calls it automation. A validation harness does something different: it parses the output into a model, then makes a claim about it.

from genie.testbed import load

tb = load('testbed.yaml')
dev = tb.devices['leaf-01']
dev.connect(log_stdout=False)

parsed = dev.parse('show ip bgp summary')
total = parsed['vrf']['default']['neighbor'].keys()
assert len(total) == 4, f"expected 4 BGP neighbors, got {len(total)}"

That assert is the whole point. It encodes an engineering decision — “this device should have four BGP neighbors” — into something a pipeline can check every time, on every device.

Build a baseline, then diff

The most practical pattern is a before/after comparison:

1. Capture parsed state before the change.
2. Apply the change through the pipeline.
3. Capture state after.
4. Diff the two and inspect.

Genie gives you the parsed structures; genie diff (or your own comparison) gives you the difference. For a change window, that diff is the evidence that the network did what you intended — and nothing else.

What is worth asserting

Not everything. Asserting whole outputs is brittle. Assert the things that define a healthy network:

• Protocol health: BGP neighbors established, expected prefixes, OSPF adjacencies full.
• Intent: interfaces up/up with the right addresses and descriptions, VLANs present, MTU as designed.
• Hardware/health: line cards and optics in a good state, no critical alarms.
• Fabric specifics: for lossless fabrics, PFC/ECN counters and queue health.

Keep assertions stable and meaningful. A test that fails every other run gets ignored, and an ignored test is worse than no test.

Connect it to the pipeline

pyATS fits naturally into CI/CD:

• the testbed is generated from your source of truth, not hand-written;
• tests run in a container in the pipeline;
• results gate the change (pre-check, post-check);
• the parsed output is archived as evidence.

This is where the value compounds. The same harness that validates a change can run on a schedule as a compliance check, or on demand during an incident to capture the “before” state.

Practical lessons

• Start small. A dozen high-value assertions across your critical devices beats a hundred fragile ones.
• Version the testbed and the tests. They are code and they should be reviewed like code.
• Treat failures as signals. A flaky assertion usually means the assertion is wrong, not the network.
• Do not hide the details. When an assertion fails, print the parsed values; the person on call needs the context, not just a red X.

Where it does not help

pyATS is not a monitoring system, and it is not a replacement for metrics. It is a validation and data-collection tool. Use it to prove state at a point in time — before a change, after a change, during a compliance sweep — and let telemetry handle the continuous view.

The mental shift is simple: stop thinking of pyATS as “how I get data” and start treating it as “how I encode what a healthy network looks like.” Once the assertions exist, everything else — pipelines, gates, compliance, incident capture — is just where you run them.