EEM Tricks: Scheduled Packet Capture

It’s going to be a short note. I’ve finally started to explore the world of automation (EEM) and coding (Python) and I love both! I used to code long time ago using Perl and PHP, and now I regret I’ve ignored these skills for the last decade (at least). I will be publishing some EEM and Python snippets here from now on. So, today I’d like to share a small piece of EEM script that, in short, waits until a specific time, then starts packet capture of all packets destined to CPU on the router, waits for X seconds, then terminates capture, exports it to the FTP server and removes 95% of its traces from the running config. This can be simplified or made more sophisticated for as much as your imagination allows…The idea to create this script occured when we tried to troubleshoot one interesting case. Our Cisco ISE infrastructure (TACACS+) has been reporting suspicious authentication attempts to one of our legacy routers (ISR G1, Cisco 2800) for a while. However, a closer look revealed that for some reason ISE didn’t have information about endpoint’s IP address for these VTY sessions. These authentication attempts have been occuring on a daily basis at a specific time, so we expected this to be some kind of scheduled backup job, but no one seemed to know about it.

So, EEM was the obvious choice.

The key feature for this particular script is TIMER event, its CRON-based implementation in particular. At the beginning of every minute CPU generates a cron_tick interrupt, which in turn invokes all CRON TIMER-based EEM policies to check if their CRON configuration matches current clock and if so, all relevant EEM policies/scripts are executed. The following diagram shows what we’ve tried to achieve (it’s rather a simple high level representation).

As you can see there are two flows. Parent Applet is responsible for CREATION and RUNNING of Traffic Capture instance (using Monitor Capture feature available on all modern platforms). It also CONFIGURES FTP credentials which will be used later; and finally it CREATES the Child Applet, which is also using TIMER event, but this time it’s simple COUNTDOWN type. Child Applet is executed when TIMER reaches zero, it then STOPS Traffic Capture and EXPORTS it to FTP server using credentials configured by Parent Applet. Finally, it removes Monitor Capture configuration, itself, Parent Applet and some global configuration applied by Parent Applet. Here’s the code (remember to replace <> with values of your choice:

! Scheduled Packet Capture
! Author: Tymofii Dmytrenko (yakuza.ua@gmail.com)
!
! Local username is required to let EEM play nicely with AAA
! Alternatively EEM Applet can be configured to bypass Authorization (IOS specific)
! If applet is configured to bypass authorization, then both lines can be ignored (not required)
!
username EEM privilege 15
event manager session cli username EEM
!
! EEM Applet Variables:
! qt: quotes (the only envvar that won't be removed after applet completes its life cycle)
! ftpdst: full ftp destination, including filename (e.g. ftp://1.1.1.1/dir1/file.pcap)
! ftpusrn, ftppswd: ftp credentials (username and password)
! captime: capture duration in seconds
!
event manager environment qt "
event manager environment ftpdst <ftp-path-with-filename>
event manager environment ftpusrn <ftp-username>
event manager environment ftppswd <ftp-password>
event manager environment captime <capture-duration-in-sec>
!
! Applet is configured to run once, or periodically (see cron-entry-value description)
! Once traffic capture completes, child instance exports results to FTP and removes all traces,
! except EEM username, event manager session cli username and event manager environment qt
! This configuration is essential for script to successfully complete its lifecycle
! 
! cron-entry-value is "min hr mday mon wday", where
! min: minute from 0 to 59 (or * for any)
! hr: hours from 0 to 23 (or * for any)
! mday: day of the month from 1 to 31 (or * for any)
! mon: month from 1 to 12 (or * for any)
! wday: day of the week from 0 to 7 (or * for any), 0 and 7 are both Sunday
!
! Examples:
! "0 * * * *" - run every hour, on the hour
! "0 0 * * *" - run every night at 00:00
! "40 9 * * *" - run at 9:40 every day
! "*/5 * * * *" - run every 5 minutes
!
event manager applet EEM-CAPTURE
 event timer cron maxrun 60 cron-entry "<cron-entry-value>" name EEM-CAPTURE
 action 000 comment Initialize Monitor and Begin to capture process-switched packets (i.e. destined to the node)
 action 001 cli command "enable"
 action 002 cli command "monitor capture buffer EEM-CAPTURE size 512 linear"
 action 003 cli command "monitor capture point ip process-switched EEM-CAPTURE both"
 action 004 cli command "monitor capture point associate EEM-CAPTURE EEM-CAPTURE"
 action 005 cli command "monitor capture point start EEM-CAPTURE"
 action 010 comment Create Child Applet which will stop capture in X seconds, export it to FTP and remove all traces
 action 011 cli command "config t"
 action 012 cli command "ip ftp username $ftpusrn"
 action 013 cli command "ip ftp password $ftppswd"
 action 021 cli command "event manager applet EEM-CAPTURE-STOP"
 action 022 cli command " event timer countdown maxrun 60 time $captime name CLEANUP"
 action 023 cli command " action 1.0 cli command $qt enable$qt"
 action 024 cli command " action 1.1 cli command $qt monitor capture point stop EEM-CAPTURE$qt"
 action 025 cli command " action 1.2 cli command $qt monitor capture buffer EEM-CAPTURE export $ftpdst$qt"
 action 026 cli command " action 1.3 cli command $qt no monitor capture point ip process-switched EEM-CAPTURE$qt"
 action 027 cli command " action 1.4 cli command $qt no monitor capture buffer EEM-CAPTURE$qt"
 action 030 cli command " action 2.0 cli command $qt config t$qt"
 action 031 cli command " action 2.1 cli command $qt no ip ftp username $ftpusrn$qt"
 action 032 cli command " action 2.2 cli command $qt no ip ftp password$qt"
 action 033 cli command " action 2.3 cli command $qt no event manager environment ftpdst$qt"
 action 034 cli command " action 2.4 cli command $qt no event manager environment ftpusrn$qt"
 action 035 cli command " action 2.5 cli command $qt no event manager environment ftppswd$qt"
 action 036 cli command " action 2.6 cli command $qt no event manager applet EEM-CAPTURE$qt"
 action 037 cli command " action 2.7 cli command $qt no event manager applet EEM-CAPTURE-STOP$qt"

Important! This code was created for Cisco ISR G1 router, so it’s unlikely to be directly compatible with ISR G2, or IOS-XE, but at least you get the idea.

Note! I had to increase maxrun value for both applets to 60 seconds (default is 20 seconds) due to the slow WAN link. EEM, if instructed to play nicely with AAA, virtually authorizes every CLI command. This processes takes time if there’s high latency between the router and TACACS/RADIUS servers. Basically, due to high latency 20 seconds was not enough to execute this script.

The biggest challange you will have with the latest platforms is that they require capture ACL to be configured to protect CPU and CONTROL PLANE. If you want ACL configuration and clean up to be part of EEM script, then it may become quite long. As a compromise, you can ignore CLEAN UP part of the script. In this case the code above can be simplified down to the following (less comments this time):

! Scheduled Packet Capture Simplified (no cleanup)
! Author: Tymofii Dmytrenko (yakuza.ua@gmail.com)
!
username EEM privilege 15
ip ftp username <ftp-username>
ip ftp password <ftp-password>
!
monitor capture buffer EEM-CAPTURE size 512 linear
monitor capture point ip process-switched EEM-CAPTURE both
monitor capture point associate EEM-CAPTURE EEM-CAPTURE
!
event manager environment qt "
event manager environment ftpdst <ftp-path-with-filename>
event manager environment captime <capture-duration-in-sec>
event manager session cli username EEM
event manager applet EEM-CAPTURE
 event timer cron maxrun 60 cron-entry "<cron-entry-value>" name EEM-CAPTURE
 action 000 comment Begin to capture process-switched packets (i.e. destined to the node)
 action 001 cli command "enable"
 action 004 cli command "monitor buffer EEM-CAPTURE clear"
 action 005 cli command "monitor capture point start EEM-CAPTURE"
 action 010 comment Create Child Applet which will stop capture in X seconds, export it to FTP
 action 011 cli command "config t"
 action 021 cli command "event manager applet EEM-CAPTURE-STOP"
 action 022 cli command " event timer countdown maxrun 60 time $captime"
 action 023 cli command " action 1.0 cli command $qt enable$qt"
 action 024 cli command " action 1.1 cli command $qt monitor capture point stop EEM-CAPTURE$qt"
 action 025 cli command " action 1.2 cli command $qt monitor capture buffer EEM-CAPTURE export $ftpdst$qt"

As you can see, it’s less lines now. Pretty much everything was moved out of the EEM applet, which STARTS Traffic Capture using pre-configured Monitor Capture instance and then CREATES Child Applet to STOP Traffic Capture in X seconds and finally EXPORT capture results to FTP server. It then stops without removing any configuration.

As opposed to the Original version of the code, this EEM script can run periodically (i.e. every 30 minutes, or every second hour, etc) because no clean up is performed.

I hope this was useful.

P.S. Oh, I forgot to say that this script helped us to identify the intruder 🙂 It was Cisco Unity Express sitting on the same ISR and trying to authenticate to the router’s WEB GUI using locally configured admin credentials (which didn’t work because box is TACACS-enabled)

2 Comments

  1. John Hinckley says:

    Nice post. You must also be from the future.

Leave a Reply

%d bloggers like this: