Evaluation Protocol

Integrating LLMs with NetBox and Netmiko for Vendor-Agnostic Intent-Based Networking

No. Intent Conception Request Construction Response Correctness System Type Prompt Comment
1 PASS PASS PASS NETBOX SC Create a location FRA1 on the DRT Campus in Frankfurt am Main.
2 PASS PASS PASS NETBOX SC Create a router r1.fra1 in Frankfurt on the DRT campus. The Custom GPT asked a large number of follow-up questions to the user, as it initially had problems populating mandatory fields of the router such as IP addresses.
3 FAIL PASS PASS NETBOX SC Create a site in Barcelona according to existing conventions. The Custom GPT has created a site in Barcelona, but forgot to look up the existing naming conventions in NetBox beforehand.
4 PASS PASS PASS NETBOX SC Create a circuit with the CID “HYPERFIBER-123” between our site in Barcelona and Frankfurt. The Custom GPT has chained several CRUD requests to solve this task.
5 PASS FAIL FAIL NETBOX SU Change the speed of the “HYPERFIBER-123” circuit to 400 Gigabit. The Custom GPT configured 400 Mbit/s and not 400 Gbit/s as the commit rate.
6 PASS PASS PASS NETBOX SU Update the name of the site in Barcelona to BC1.
7 PASS PASS PASS NETBOX SU We have moved the router from FRA1 to Bercelona. Move it in NetBox.
8 PASS PASS PASS NETBOX SU Move my FRA1 location to Equinix in Kleyerstraße.
9 PASS PASS PASS NETBOX SG Give me all the sites.
10 PASS PASS PASS NETBOX SG What routers do we have in Barcelona?
11 PASS PASS PASS NETBOX SG What active circuits do we have?
12 PASS PASS PASS NETBOX SG For which routers does the name not match the actual location? A timeout occurred, the Custom GPT detected it, attempted a retry and succeeded.
13 PASS PASS PASS NETBOX SD Delete the routers whose name does not match the actual location. A timeout occurred, the Custom GPT detected it, attempted a retry and succeeded.
14 PASS PASS PASS NETBOX SD Delete the circuit between Barcelona and Frankfurt.
15 PASS FAIL PASS NETBOX SD Delete the site on the DRT campus in Frankfurt. The GPT has tried to delete the site with ID 1. However, this site did not exist in NetBox. Nevertheless, this error was recognized and reported back to the user.
16 PASS PASS PASS NETBOX SD Delete all sites in NetBox.
17 PASS PASS PASS NETBOX MC Create the location BOB1 in the French Polynesia region.
18 PASS PASS PASS NETBOX MC Create the router r1.bob1 in a new rack in location BOB1.
19 PASS PASS PASS NETBOX MC Create the interfaces ethernet1-0 to ethernet1-10 on the router r1.bob1 and assign them MAC addresses in the range from 00:1A:2B:3C:4D:00 to 00:1A:2B:3C:4D:0A. The Custom GPT offered to update the devices after updating them to NetBox.
20 PASS FAIL FAIL NETBOX MC Create two routers r1.lhr1 and r2.lhr2 at the London location in NetBox and connect them via a circuit with the ID: LondonConnect. The Custom GPT understood the intent semantically correctly. The created request was incomplete, which meant that objects necessary for modelling were not created. The circuit termination was derived from the context. The interfaces on the routers and the cables between interfaces and circuit terminations were not correctly taken into account. Only the routers, the circuit and the circuit terminations were created. The Custom GPT did not recognize this error and returned an incorrect success message to the user,
21 PASS PASS PASS NETBOX MU Update all routers to the standardized naming scheme: r1.[IATA-Code][siteID]. The prompt was deliberately designed to create duplicates, by mapping multiple routers in the same region to r1. The duplicates were recognized based on the system feedback and name suggestions were made.
22 PASS PASS PASS NETBOX MU Please reset the names again.
23 FAIL FAIL PASS NETBOX MU Please maintain and standardize all missing and inconsistent data on the sites. It had to be explained that BOB1 stands for Bora Bora. London was incorrectly placed in French Polynesia. There was a region duplication because no GET was performed beforehand. However, the coordinates were set correctly. The Custom GPT had recognized that it had made a mistake.
24 FAIL PASS PASS NETBOX MU The circuit LAConnect is currently missing the cables and interfaces to connect the r1.lax1 and r2.lax2 routers. The Custom GPT got into a logical loop, repeatedly attempting to recreate a cable without properly handling system feedback. Eventually, it made a syntactically correct request but realized it didn't match the user's intent.
25 PASS PASS PASS NETBOX MG Give me all regions and their sites.
26 PASS PASS PASS NETBOX MG Model the trace of the LondonConnect circuit. The Custom GPT first visualized the trace without the cables, on request they were added.
27 PASS FAIL FAIL NETBOX MG For which circuits is the trace incomplete? Only circuit were checked and not the connection from the circuit termination to the interface of the device, which meant that missing connections were not recognized.
28 PASS PASS PASS NETBOX MG Inventory the network in the form of a topology from NetBox.
29 PASS PASS PASS NETBOX MD We have given up the London location. Delete all data relating to London from NetBox.
30 PASS PASS PASS NETBOX MD Delete all interfaces from r1.bob1.
31 PASS PASS PASS NETBOX MD Delete rack 1 and all devices in it.
32 PASS PASS PASS NETBOX MD Delete all data in NetBox.
33 PASS PASS PASS NETMIKO SC Set up the port ethernet-1/2 on the Nokia router 172.20.20.2.
34 PASS PASS PASS NETMIKO SC Add the first interface to the lag-1 on the Nokia router 172.20.20.2. An inter-vendor bias was recognisable for the Cisco command set and an intra-vendor bias for the Nokia SR OS command set. Through self-correction and web-based RAG, the Custom GPT found the correct Nokia SR Linux commands.
35 PASS PASS PASS NETMIKO SC Create the LAG lag-2 on the Nokia router 172.20.20.2. An inter-vendor bias was recognisable for the Cisco command set and an intra-vendor bias for the Nokia SR OS command set. Through self-correction and web-based RAG, the Custom GPT found the correct Nokia SR Linux commands.
36 PASS PASS PASS NETMIKO SC Add a meaningful interface description to the first interface of the Nokia Router 172.20.20.2.
37 PASS PASS PASS NETMIKO SU Switch off the first interface of the router 172.20.20.2.
38 PASS PASS PASS NETMIKO SU Change the description of the first interface of the router 172.20.20.2 to ADMINDOWN.
39 PASS PASS PASS NETMIKO SU Change the interface speed of the first interface from 172.20.20.2 to 10Gbit/s.
40 PASS FAIL PASS NETMIKO SU Clear the flaps for the first interface on the 172.20.20.2 router. The Custom GPT did not find the correct command and reported this back to the user.
41 PASS PASS PASS NETMIKO SG Connect to the Nokia router 172.20.20.2 and show me its interfaces.
42 PASS PASS PASS NETMIKO SG Which SR Linux version does the Nokia Router 172.20.20.2 have?
43 PASS FAIL PASS NETMIKO SG Which VLANs are configured on the Nokia router 172.20.20.2? The Custom GPT queried incorrect areas of the device configuration.
44 PASS PASS PASS NETMIKO SG Which LAGs are configured on the Nokia router 172.20.20.2?
45 PASS PASS PASS NETMIKO SD Remove the description of the first interface from 172.20.20.2.
46 PASS PASS PASS NETMIKO SD Remove the subinterface of ethernet-1/3 on 172.20.20.2.
47 PASS PASS PASS NETMIKO SD Remove the port ethernet-1/2 FROM 172.20.20.2. The port has been deactivated, which is equivalent to deleting it.
48 PASS PASS PASS NETMIKO SD Delete the LAG lag1 from 172.20.20.2. An inter-vendor bias was recognisable for the Cisco command set. Through self-correction and web-based RAG, the Custom GPT found the correct Nokia SR Linux commands.
49 PASS FAIL PASS NETMIKO MC Set up a port mirror from port ethernet-1/1 to port ethernet-1/2 on the Nokia router 172.20.20.2. An intra-vendor bias was recognisable for the Nokia SR OS command set. Despite self-correction and web-based RAG, the Custom GPT did not build the request correctly for Nokia SR Linux and reported this correctly back to the user.
50 PASS FAIL PASS NETMIKO MC Add the first interface of 172.20.20.2 to lag1. The Custom GPT omitted the web-based RAG and therefore did not use the correct commands. It reported back to the user that it was unable to implement the intent.
51 PASS PASS PASS NETMIKO MC Set up the second interface of 172.20.20.2 and provide it with the description fra_to_nyc. The Custom GPT has recognized through self-correction that the port speed it originally wanted to configure is not supported by the device and has selected a suitable alternative.
52 PASS PASS PASS NETMIKO MC Set up two subinterfaces for the second port at 172.20.20.2. An inter-vendor bias was recognisable for the Cisco command set. Through self-correction and web-based RAG, the Custom GPT found the correct Nokia SR Linux commands.
53 PASS PASS PASS NETMIKO MU Activate the first interface of 172.20.20.2 and change its description to ADMINUP.
54 PASS PASS PASS NETMIKO MU Set up two subinterfaces for the second port at 172.20.20.2. The Custom GPT deployed a faulty configuration, corrected it within the same commit, and avoided downtime. Since this could also happen to a human operator, we accept it.
55 PASS PASS PASS NETMIKO MU Change the interface speed of the first two interfaces from 172.20.20.2 to 10Gbit/s.
56 PASS PASS PASS NETMIKO MU Activate the first two interfaces of 172.20.20.2.
57 PASS PASS PASS NETMIKO MG Retrieve all interfaces from 172.20.20.2 and 172.20.20.3.
58 PASS PASS PASS NETMIKO MG Create a list of the SR Linux versions of 172.20.20.2 and 172.20.20.3. An intra-vendor bias was recognisable for the Nokia SR OS command set. Through self-correction and web-based RAG, the Custom GPT found the correct Nokia SR Linux commands.
59 PASS PASS PASS NETMIKO MG Are both interfaces connecting 172.20.20.2 and 172.20.20.3 in operation?
60 PASS PASS PASS NETMIKO MG Are the port speeds on the opposite sides of 172.20.20.2 and 172.20.20.3 configured consistently?
61 PASS FAIL FAIL NETMIKO MD Remove the subinterfaces of ethernet-1/2 from 172.20.20.2. The request was built incorrectly, as only the first two sub-interfaces were removed, but there were more. The Custom GPT did not realize this and gave the user a success message.
62 PASS PASS PASS NETMIKO MD Delete the description of the ethernet-1/1 interface on the routers 172.20.20.2 and 172.20.20.3.
63 PASS PASS PASS NETMIKO MD Delete the LAG lag2 on the routers 172.20.20.2 and 172.20.20.3.
64 PASS PASS PASS NETMIKO MD Reset the routers 172.20.20.2 and 172.20.20.3 to the factory settings. Everything was deleted, including the interfaces and the management connection, which meant that the Custom GPT could no longer access them. But that was the task.