Tuesday, November 20, 2012

EEM Scripting



EEM Scripting


Welcome to the Embedded Event Manager (EEM) Scripting Community.

EEM is a flexible system designed to customize IOS, XR, and NX-OS. EEM allows you to automate tasks, perform minor enhancements and create workarounds.

With this community you can get help with EEM and scripting. You can download examples and useful scripts submitted by others to use in your environment. We encourage you to develop EEM scripts and share them with others by uploading them here.
Need help converting from applets to Tcl? Try our EEM Applet Converter.


 

 Monitor MAC Address Table for Exhaustion

This Tcl policy was designed for the Catalyst 4000 series (e.g., 4500 and 4900 series) switches to check for ports that exceed a specified threshold of learned MAC addresses.  If the threshold is exceeded for any interface, a syslog is sent for that interface showing the number of learned MACs.
This policy requires one EEM environment variable to be set:
event manager environment learned_mac_threshold THRESHOLD
Where THRESHOLD is the number of MACs that can be learned on a port before the syslog will be sent.
This policy was tested on a Catalyst 4500 running 15.0(2)SG3.
Attachments:

TCL Script to log all Switch Interface Statistics to a CSV file

Have created a TCL script which can tabulate all the Interface Statisctics seen from the "Show Interface" command in a CSV file
This CSV file is then copied to a TFTP server
This can then be opened up in Excel.
This was created on a 3560 switch.
 Copy the file onto the Flash of the switch and run "tclsh switchstats.tcl" from the command line
 Multiple of such files from different switches can be concatenated and then can be consolidated into one large file.
Attachments:

Flexible Netflow cache table output formatter

Is your Flexible Netflow cache table too big to display on your screen? This simple script filters out unwanted columns of the cache table to show an abbreviated table that fits on your screen. This script is easily customized so you decide which columns you see.
cv.zip (2.2 K) Download

View Top Talkers in "real time" from CLI

This experimental script is similar to the Unix 'top' command except it provides an updating list of Top Talkers. The script works by taking 7-second snapshots of the netflow cache table and calculates the byte/packet difference between snapshops. This calculation is used to find the Bit/sec and Pkt/sec for each flow. Flows with the highest Bits/sec are output to the screen in a 'Top Talker' table. The script output is a growing list so use 'term len 0' before running the script.
top.zip (14.4 K) Download

Email Top Talker reports from your router

Wouldn't it be nice if network devices could collect & process their own netflow data and email you the results? Well, now they can!
This tcl script tracks the local netflow cache and generates "Top Talker" reports. Reports are run from CLI or configured to run automatically from Kron with the results sent via email. The script works by tracking and periodically storing netflow cache data in local log files. The reporting engine analyzes the log files to provide Top Talker reports from any of the previous five days. This script runs locally and does not open or make network connections with the exception of the built-in IOS smtp_send_email function. Performs best on ISR G2 or similar with latest IOS.
netc.zip (19.0 K) Download

EEM Lightweight AAA Server Sample

Sample applet to deny certain users from running commands, and also store each command users have entered into a file. This applet uses a denylist file to compare against who is running each command.  If the user and command is listed in the denylist file, the user will not be able to run the command.  This applet also creates a cmdhistory file that logs all commands.  This applet currently registers against files in disk2:  Please modify disk location when using.   EEM 4.0 is required to run this applet.
Sample denylist file:
cisco show users
cisco configure terminal
Applet:
event manager applet testcli
 event cli pattern .* sync yes
 action 101 file open in disk2:denylist r
 action 102 file open out disk2:cmdhistory a
 action 103 file read in _jpres
 action 104 set tester "$_cli_username"
 action 105 append tester " $_cli_msg"
 action 106 foreach value "$_jpres" "\n"
 action 107  if $tester eq $value
 action 108   puts "$_cli_username is not authorized to run $_cli_msg"
 action 109   file puts out "REJECT: $_cli_username: $_cli_msg"
 action 110   exit 0
 action 111  end
 action 112 end
 action 113 file puts out "ACCEPT: $_cli_username: $_cli_msg"
 action 114 exit 1

Automatically Set Port Descriptions

In preparing for CiscoLive! in San Diego, I am provisioning our access layer 3560-E switches.  Since things have a tendency to change a lot at an event like CiscoLive! I thought it would be best to make sure our port descriptions are always up-to-date when it comes to reflecting what devices are connected.  To help me do that, I wrote up this small EEM applet policy.  It will update the port's description based on the CDP neighbor learned on that port.  This policy requires EEM 3.2, so you're looking at 12.2(55)SE or higher for the 3560s.  It will also work on 3750s and ISR G2 routers running 15.x code.
 event manager applet update-port-description
 event neighbor-discovery interface regexp GigabitEthernet.* cdp add 
 action 1.0 cli command "enable"
 action 2.0 cli command "config t"
 action 3.0 cli command "interface $_nd_local_intf_name"
 action 4.0 cli command "description $_nd_cdp_entry_name:$_nd_port_id"

The result of this will be a description like the following on switch ports:
description SDCC_IDF_1.11:TenGigabitEthernet0/1

Ping Monitor

pingmon offers an easy way to check if, eg. provider links works  fine. It is very useful if you don't have an external monitoring software or you didn't trust it. pingmon is using standard cli ping. So  it is reporting packet loss, min, average and max rtt. It is running at configurable intervals and sends as many pings as configured with the  repeat counter. Results are written to daily rotated files. To ensure that flash isn't running out of memory the parameter "loghist" will keep  an eye on it. With log level "debugging" pingmon will forward messages  to the corresponding lines.
Attachments:
pingmon.tar.gz (3.4 K)

EASy Dynamic DHCP snooping

Assuming some PCs are legitimately using a fixed IP address (ie:  servers), that a switch is having DHCP Snooping/IPSG active, and that  you do not want to manage the switch at the port level, this script will  disable DHCP Snooping for servers assuming they are not trying to steal  someone else's IP / MAC address.

Attachments:

Archive Config if Changes

EEM script will trigger each syslog configuration message, and only  when there is a change in the current running configuration, it will  safe that configuration in the path configured under "Configuration  Replace and Rollback" IOS embedded feature configured in the router.
Attachments:

Send Show-tech via Email

This applet will email the output of 'show tech' using the command  'sendtech'. An alias may be used to create the command 'sendtech' to  manually trigger this applet. This applet may be also modified so the  output of 'show tech' will be attached to an open TAC.
Attachments:

Log User Name Information

Applet will send only the config deltas when a configuration change  notification alarm is detected using the IOS config diff feature.  In  addition, the output of 'show archive log config all' will be provided  in an email to show the user who made the configuration change. Top of Form
Attachments:

Tweet from IOS

Script Info URL:
 Have you ever wondered how to use twitter API from within IOS? Here's a simple example to get you started ...
Attachments:

Send Email

Script Info URL:
http://www.cisco.com/cdc_content_elements/flash/ios/ios_commercial/send_email/Send_Email.html
 Tcl script serves as a template that can be used when sending an  email notification is needed. A training video has been created for this  script. Use the 'Script Info URL' to watch the video.
Attachments:

TCP Syn Flow Detection

The Script checks the number of syn only flows in the given netflow  monitor and raises syslog messages if any abnormal syn flows are  detected.  The flow also checks if there is sudden increase in syn only  flows for given time frame and generates syslog message for the same.   The script can also shutdown the concerned port if specified.
Attachments:

Tclsh menu-driven ACL editor

This is a tclsh script which allows one to edit access-lists  configured on the local device using a menu interface.  To invoke the  script do the following:    Router#tclsh flash:edit_acl.tcl <ACL  number>    For example, to edit ACL 113:    Router#tclsh  flash:edit_acl.tcl 113
Attachments:

Email notify for PAT Changes

I use this to notify me when a PAT address changes at a home router.   We lock down by source route our DMVPN tunnels.  This program emails  the old and the new PAT address.
 Attachments:

WAN Load Alarm

Tcl script sends an alarm via syslog and email if the WAN link  specified exceeds a specified load (wan_load_threshold) for more than a  specified duration of time (wan_load_duration). This script takes  samples of the txload/rxload in the output of 'show interface' at  specified intervals (wan_load_interval) to calculate the overall average  of each over the specified duration (wan_load_duration).
Attachments:

Application Failure Detection

Applet provides examples to monitor application features based on TCP  port availability. Cisco IP SLA and Embedded Object Tracking are used  to trigger these applets.
Attachments:

Server Fail Detection Down

Script Info URL:
http://www.cisco.com/cdc_content_elements/flash/ios/ios_commercial/server_failure_detection_down/Server_Failure_Detection_Down.html
 Applet sends an email notification when the tracked state of  icmp-echo to a particular server goes to a 'down' state. A training  video has been created for this script. Use the 'Script Info URL' to  watch the video.
Attachments:

Watch for NAT pool depletion

This EEM policy runs every 60 seconds, and checks a specified NAT  pool for its usage.  If that usage percentage exceeds a specified  amount, a syslog message is sent.  If the nat_pool_email_template  variable is defined and points to a valid EEM email template, the  threshold violation alert will also be sent via email.
Attachments:


Cisco EEM Best Practices






Introduction

In the years since the introduction of Cisco's Embedded Event Manager (EEM) many EEM policies have been developed inside and outside of Cisco.  In the development of those policies many lessons have been learned about what works best and what does not.  This document strives to outline some of the best practices that have been identified over the years when it comes to Cisco EEM policy design and development.

File Naming Convention

Cisco developed EEM Tcl policies that are included in the operating system (also known as "system" policies) follow a very strict naming convention.

  • An optional prefix of "Mandatory." - If this prefix exists, the system policy will be registered automatically on boot-up unless the configuration specifies that it should not be registered.  For example: Mandatory.go_switchbus.tcl
  • A filename - Containing a two character abbreviation for the first event specified in the policy, an underscore and a descriptive name.  The two-character abbreviation is well documented in the EEM feature documentation.
  • A file extension or suffix of ".tcl" or ".tbc" - This is required for all EEM Tcl policies.  The ".tcl" extension is used for plain text Tcl policies and the ".tbc" extension is used for compiled bytecode Tcl policies.

When naming Cisco EEM Tcl policies is recommended that the name follow this naming convention while adding in a user or customer specific descriptive string.  While it is perfectly acceptable for a user policy to follow the exact same naming convention that system policies use, it is usually advisable to stray from it slightly so there is no potential for overlap with system policies to avoid potential confusion when registering policies.

When EEM Tcl policies are registered the user has the option of including a policy type of system or user in the registration configuration command.  This option is not required and in reality it does not specify a policy type per se but rather where to look for the policy.  If the policy type option is not provided when registering a policy, the system policy directory is searched before the user policy directory.  This behavior is by design but can also lead to confusion if you have both a system and user policy with the same name and register the policy without specifying the policy type - the system policy will be registered even though you may have intended the user policy to be registered.  You can force the registration of the user policy by including a policy type of user in the event registration configuration command.

Applets versus Tcl Scripts versus Shell Policies

There are currently three native policy engines within Cisco EEM.

  • Applets - Supported since EEM version 1.0, these policies are specified and defined in the configuration of the device and were designed to allow simple interface into Cisco's EEM feature.
  • Tcl Scripts - Supported since EEM version 2.0, these policies are defined in separate files stored locally on the device and specified (registered) by adding a single configuration command.  Tcl allows for more complex policies and some EEM features like timer subscribers are (currently) only supported in EEM Tcl.
  • Shell Policies - Supported since EEM version 3.2, these policies are defined in separate files stored locally on the device and specified (registered) by adding a single configuration command.  Shell policies utilize the IOS shell feature.  Support for shell policies is currently (mid-2010) limited to some switching platforms.

NOTE: When using EEM applet policies on devices that support IOS shell (IOS.sh), be careful when enabling shell processing full on the device as this can cause variables (i.e. strings that begin with '$') to disappear from the running configuration.

See this document to determine which version of EEM you are running.

Tcl script execution requires the spawning of a Tcl shell which does require additional resources (memory and CPU) over applet or shell policies.

Applet policies can actually get very sophisticated in EEM versions 3.0 and above.  However, a large applet can often times be very difficult to manage, debug and troubleshoot because of the way applets are stored in the configuration and how actions are sorted based on a tag.  If you find a need to move from applet to Tcl policies, and you are looking for an easy way to transition, the applet converter tool quickly translates applets into Tcl policies.  From there, you can refine the Tcl code to satisfy the additional capabilities.

Shell policies have a very limited scope at this point in time.  They are primarily used in the auto smart ports feature.  As the IOS shell feature becomes more feature rich, IOS shell policies will pick up the ability to do more as well.

Tcl Libraries

A Tcl library is a collection of utility modules for Tcl.  These modules provide a wide variety of functionality - from implementations of standard data structures to implementations of common networking protocols.  The intent of a library is to collect commonly used functions and make them available in a reliable and easy to use manner.

The standard EEM libraries are imported with the following statements:

namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

The event_reqinfo and cli_open procedures are examples of standard EEM library procedures.  Prior to using these procedures the namespace that contains them (::cisco::eem::* for event_reqinfo and ::cisco::lib::* for cli_open) must first be imported.

Users can create their own libraries as well and use them in their own scripts.  User libraries are stored on a local storage device just like a user policy.  The user can then tell EEM where those libraries are using the config command:

event manager directory user library <device>:/<directory>

Once this command is configured, EEM copies all of the .tcl files found in the defined device and directory to a virtual file system for later use.  When an EEM Tcl policy is triggered, the user libraries are made available to the EEM Tcl policy by use of the Tcl source command, for example:

source "user_library.tcl"

NOTE: If you make any changes to this directory (i.e. add new library files), you must unconfigure and reconfigure the user library directory before those library files will be available to your EEM policies.

Or you can also auto load the procedures in the libraries by building a tclIndex file. Building a tclIndex file is documented in Writing Embedded Event Manager Policies Using Tcl.

Please note that bytecode (compiled) Tcl libraries are not supported by EEM Tcl.

Libraries allow for the reuse of common procedures which can ease in the management of those procedures.  For example if you have a procedure that prints debug information out and you use it in all of your policies, would it not be easier to maintain that procedure in a single file rather than each individual Tcl policy?  If the procedure needs to change for any reason you simply update the library rather than every single policy that utilizes that procedure.

Keep in mind that a copy of the library is stored in a virtual file system when the user library directory is configured.  So if you update the library in anyway, those changes will not take affect until you remove the user library directory configuration and add it back.

Tcl Packages

EEM comes with a number of Tcl packages built in.  In addition to libraries found in the ::cisco::eem and ::cisco::lib namespaces, EEM ships with the following additional libraries.

Namespace / Procedure
Description
More Info
parray
Print the contents of a global array
Syntax:
parray arrayName ?pattern?

Where arrayName is the name of an array in the global namespace and pattern is an optional element name pattern to print (by default all elements will be printed)
::http
HTTP client library
::tcl::opt
Command-line option parsing library
::base64
Base64 encoding and decoding library

Each of these packages can be found under tmpsys:/lib/tcl and are automatically loaded for all EEM Tcl policies.

EEM and AAA Command Authorization

By default, if a device is configured for AAA command authorization, EEM will use it.  However, EEM does not send a username to the AAA server by default.  This will result in "Command authorization failed" errors when your EEM policies execute CLI commands.  For EEM to play nicely with AAA command authorization, configure the following.

Router(config)#event manager session cli username USER

Where USER is a username authorized to run all CLI commands in all of your EEM policies.

Even though it is possible to configure EEM to work with AAA command authorization, it may be desirable to allow your EEM policies to bypass authorization.  This is especially true if it takes a bit of time to authorize each command.  In that case, the EEM maxrun timer may be reached causing policies to terminate.  If you will only have one EEM policy running at a time (that executes CLI commands), configure the following AAA commands to dedicate line vty 0 for EEM.

aaa authentication login EEMScript none
aaa authentication login default group tacacs+ local
aaa authorization exec EEMScript none
!
aaa authorization command 0 EEMScript none
aaa authorization commands 1 EEMScript none
aaa authorization commands 15 EEMScript none
!
line vty 0
 login authentication EEMScript
 authorization exec EEMScript
 authorization commands 0 EEMScript
 authorization commands 1 EEMScript
 authorization commands 15 EEMScript
 transport input none
 length 0
!

Because "transport input none" is configured on this line, it will not be accessible for telnet or SSH sessions.  However, EEM policies will be able to use this VTY to execute CLI commands without going through AAA command authorization.

Beginning with EEM 3.1, AAA command authorization can be bypassed on a per-policy basis.  The following are examples for registering applet, Tcl, and IOS.sh policies that bypass AAA command authorization.

Applet:

event manager applet myapplet authorization bypass

Tcl:

event manager policy mypol.tcl authorization bypass type user

IOS.sh:

event manager policy mypol.sh authorization bypass type user

CLI Tips

As mentioned in the section above, each CLI session requires its own VTY line.  Since IOS has a limited number of available VTYs, it is recommended to optimize the use of VTYs to prevent exhausting all available lines.  When the cli_open command is called, there must be at least two free VTY lines.  One of these lines will be used for the EEM CLI session and the other must remain free (EEM is a good network citizen).

If your Tcl policy will execute a number of CLI commands, consider opening one CLI session for the whole policy.  This will ensure that the policy will only ever allocate one VTY line.  Alternatively, if you'd like to not tie up a VTY line for the entire life of the policy execution, use the built-in cli_run command to batch a number of CLI commands together.  The cli_run command was introduced in EEM 3.0.

set output [cli_run [list "show ver" "show run"]]

If you do not have EEM 3.0 or later, you can use this procedure in your policies to get the cli_run behavior.

proc cli_run { clist } {
    set rbuf ""

    if {[llength $clist] < 1} {
        return -code ok $rbuf
    }

    if {[catch {cli_open} result]} {
        return -code error $result
    } else {
        array set cliarr $result
    }

    if {[catch {cli_exec $cliarr(fd) "enable"} result]} {
        return -code error $result
    }

    foreach cmd $clist {
        if {[catch {cli_exec $cliarr(fd) $cmd} result]} {
            return -code error $result
        }

        append rbuf $result
    }

    if {[catch {cli_close $cliarr(fd) $cliarr(tty_id)} result]} {
        puts "WARNING: $result"
    }

    return -code ok $rbuf
}
Dynamic Event Registration

The event registration line in an EEM Tcl policy is like any other Tcl code.  You can be quite creative with the content in order to register for events using dynamic data.  For example, you can embed EEM environment variables into an event registration line.

::cisco::eem::event_register_syslog pattern $my_syslog_pattern occurs 3

In this example, the syslog pattern that triggers the policy can be changed without needing to modify the Tcl code.

More advanced code flows can be included as well.  For example, if you want to allow a configurable maxrun timer (via an EEM environment variable), but you also want to set a reasonable default if a value is not set, you can use code similar to the following.

::cisco::eem::event_register_none maxrun [expr {[info exists my_policy_timeout] ? $my_policy_timeout : 36}]

In this example, if the my_policy_timeout EEM environment variable is set, then the value of that variable will be used as the maxrun timer value.  If the variable is not set, then the maxrun timer will be set to 36 seconds.

Maxrun and Default Timers

By default, all EEM policies should run to completion within 20 seconds.  This 20 second timer is known as the maxrun timer.  It is not always possible to accomplish everything your policy needs to do in 20 seconds, however.  For policies requiring a longer run time, the maxrun timer can be increased when configuring your event registration line.  The maxrun time is specified in seconds.milliseconds.  The following examples will set the maxrun timer to 60 seconds.

Applet:

event syslog pattern "CONFIG_I" maxrun 60

Tcl:

::cisco::eem::event_register_syslog pattern "CONFIG_I" maxrun 60

IOS.sh:

##::cisco::eem::event_register_syslog pattern "CONFIG_I" maxrun 60

If a policy exceeds its maxrun timer (no matter what the value), it will print the following message:

Process Forced Exit

"Process Forced Exit" means the process was forcibly terminated, either by exceeding its maxrun timer or by being manually killed.

In addition to the maxrun timer, certain event detectors have a default timer.  The default timer is the number of seconds (and milliseconds) before the default action will be taken.  The default timer exists for CLI, RPC, SNMP Object, and None event detectors.  In the case of the CLI event detector, the default action is to execute the intercepted command.  For RPC, SNMP Object, and None, the default timer specifies how long to wait before the event is published up the stack (i.e. to the next consumer).  If not specified at policy registration time, the default timer is 30 seconds.  It is configured just like the maxrun timer using the default keyword.

It is wise to be mindful of the default timer especially with CLI policies.  If you are using the CLI event detector to extend the parser, your code may take longer than 30 seconds to complete.  If it does, the original command will be sent to the parser for execution, and the user could see a strange syntax error on the screen.

Terminating a Running Policy

Typically, a policy will run no longer than its maxrun time.  However, under certain circumstances, the policy may become wedged in the execution queue.  Prior to EEM 2.4 it was not possible to terminate a running policy without reloading the device.  In EEM 2.4 a command was added to forcibly kill a running policy:

Router#event manager scheduler clear policy JOBID

This command will kill a specific policy.  If you want to terminate all running policies, use the following command:

Router#event manager scheduler clear all

In order to find the JOBID of a running EEM policy, use the following command:

Router#show event manager policy pending
Key: p - Priority        :L - Low, H - High, N - Normal, Z - Last
     s - Scheduling node :A - Active, S - Standby

default class - 1 applet event
 no.  job id      p s status  time of event             event type          name
 1    40          N A running Wed Jun 8 10:29:22 2011   none                sleep

In this sample output, the JOBID is 40.

In EEM 3.0 and later, the pending queue was broken out into two queues.  The pending queue will only show those policies about to run, where as the active queue will show those policies that are actually running.  To view the active queue, use the following command:

Router#show event manager policy active

Around the same time EEM 2.4 was released another feature went into EEM to help prevent runaway policies.  Prior to this fix, a policy could unregister itself from the configuration yet still run to completion.  After this fix went in, if a policy unregisters itself, it will be forcibly terminated immediately.  Therefore, if you are executing a policy that will remove itself from the configuration, make sure the unregistration step is the last thing the policy does.

Comments and Descriptions

Beginning with EEM 3.1, it is possible to document your policies using header metadata.  This metadata can be viewed using the show event manager policy registered description POL command.  Documenting the policy is very useful for long-term maintenance.  It will help you quickly understand what a policy is doing and why it was configured.  The following examples add documentation when registering a policy.

Applet:

event syslog pattern "CONFIG_I"
description "This policy counts the number of configuration changes"

Tcl:

::cisco::eem::event_register_syslog pattern "CONFIG_I"
::cisco::eem::description "This policy counts the number of configuration changes"

IOS.sh:

##::cisco::eem::event_register_syslog pattern "CONFIG_I"
##::cisco::eem::description "This policy counts the number of configuration changes"

Once the policy is register, use the show event manager policy registered description POL command to view the description.

Router#show event manager policy registered description mypol
1    applet    user    syslog                 Off   Wed Dec 29 16:27:27 2010  none  mypol
 pattern {CONFIG_I}
 maxrun 20.000
 action 1.0 counter name config_counter op inc value 1

%EEM description of policy (mypol)
-----------------------------------------------
"This policy counts the number of configuration changes"

In addition to the policy metadata, it is helpful to comment your policies on a line-by-line basis to help you remember what the policy is doing at certain points.  For Tcl and IOS.sh policies, all comments are found on a line by themselves beginning with a '#' character.  The ability to comment applet code was added in EEM 3.0 when programmatic applet syntax was introduced.  An applet comment works like any other action.  The following are examples of script and applet comments.

Applet:

action 1.0 comment "Enter enable mode because file system access is required"

Script:

# Enter enable mode because file system access is required
EEM Environment Variables

EEM environment variables are set in config mode and can be accessed by policies like any other global variable.  However, if an environment variable value is changed in a policy using the set command, the config will not be updated with the new value.  To change the value of an environment variable and have it persist, you must enter config mode and reconfigure the variable.  If you wish to share dynamic data across policies without entering config mode consider using EEM contexts.

Environment variables offer a great way to keep EEM policies (especially script policies) as dynamic as possible.  The code can be written in a way such that configurable parameters (e.g. interface names, syslog patterns, timers, etc.) are set via environment variables.  The environment variable values are then set in config mode without needing to continually modify the code.

If you are writing policies that will be shared with others, it is practically a must to use environment variables for configuration storage.  However, you should be careful not to assume that an environment variable will be properly set.  Use the Tcl info command to test that an environment variable has been set before dereferencing it.  This will avoid cryptic errors.

if { ! [info exists my_env_var] } {
    puts "ERROR: Policy cannot be executed as the environment variable my_env_var has not been set."
    exit 1
}

When declaring your own variables (environment variables or otherwise) avoid using the underscore ('_') as the first character in the variable name.  The underscore is reserved by Cisco-created variables.  As long as your variables begin with any other character, there will be no risk of those variables being overridden by Cisco variables.

Global Variable Usage

Before a global variable can be used in a Tcl procedure, it must be declared as global.  For example:

proc myProc { } {
    global debug
...
}

set debug 1
myProc

It is usually easy to remember this for variables you create in your policies.  However, this rule also applies for Tcl-supplied global variables such as errorInfo.  It is common for scripts to do something like the following:

proc myProc { } {
    if { [catch {cli_exec $cli(fd) "show run"} result] } {
        error $result $errorInfo
    }
}

However, if errorInfo is not declared global, the "error" command will fail with an unexpected stack trace.  The proper solution is as follows.

proc myProc { } {
    global errorInfo   
    if { [catch {cli_exec $cli(fd) "show run"} result] } {
        error $result $errorInfo
    }
}

An alternative to using the global command is to use namespace syntax.  The global namespace is called "::".  Another way of writing the example above using namespace notation is as follows.

proc myProc { } {
    if { [catch {cli_exec $cli(fd) "show run"} result] } {
        error $result $::errorInfo
    }
}

NOTE: Using the "::" notation is known to have a slight performance hit.  It is better to declare variables as global.
Scalability

Consider how a policy operates when it is scaled.  If you are testing a policy that executes a few config commands per interface and it works great on 3 or 4 interfaces, consider what would happen if the number of interfaces is increased to one thousand or two thousand which may be more realistic in a production environment. Sending one config command to several hundred interfaces can take a really long time.  In the case of interfaces, using the "interface range" notation is a way to deploy a set of commands to a large number of interfaces at once.

By default, EEM allocates 32 scheduler threads for applets and one thread each for Tcl and IOS.sh policies.  This means that up to 32 applets can run at one time where as only one of each Tcl and IOS.sh policy can run simultaneously.  If you are making heavy use of script policies, you may find that if many policies are triggered at the same time, events will be lost.  This is because each type of policy has a fixed-size event pool.  If the pool fills up before the events can be serviced, then new events will be dropped.  By default, applet and Tcl policies have event queue sizes of 64 where as the IOS.sh queue's size is 128.

To modify the number of policies that can be executed simultaneously, use the "event manager scheduler TYPE thread class default number NUM" command, where TYPE is the policy type (either applet, shell, or script) and NUM is the number of policies that can be run simultaneously.  For example, to allow for 10 EEM Tcl policies to be run simultaneously, configure the following command.

Router(config)#event manager scheduler script thread class default number 10

The ability tune the number of Tcl threads was added in EEM 2.3 where as the ability to tweak the number of applet threads was added in EEM 3.0.
Tcl Policy Register and Unregister

Script policies are registered with the single "event manager policy POLNAME" command.  When this command is entered, the EEM server using the following rules to locate the policy and load it.

  1. If neither the system or user type is specified then EEM looks in the system area first and then the user area. This allows a user policy to override a system policy with the same name only if the user specifies a type of user. If the policy is not found in either area, the EEM will stop and emit an error. If a policy is found:
    1. in the system area, register the policy as registration run-type of system.
    2. in the user area, if digital signature support is available, look for a valid Cisco digital signature. If a valid Cisco digital signature is found, register the policy as registration run-type system. Otherwise register the policy as registration run-type user.
  2. If a type of system is specified look in the system area. If the policy is not found, the EEM server will stop and emit an error message. If the policy is found register the policy as registration run-type system.
  3. If a type of user is specified look in the user area. If the policy is not found, the EEM server will stop and emit an error message. If the policy is found, look for a valid Cisco digital signature. If a valid Cisco digital signature is found, register the policy as registration run-type system. Otherwise register the policy as registration run-type user.
The class keyword is used to specify which scheduling class the policy will be assigned to. If the scheduling class has no available threads, the policy will be registered, but a warning message will be emitted about the lack of threads.

If the trap keyword is specified, then an SNMP trap (the cEventMgrServerEvent trap from CISCO-EMBEDDED-EVENT-MGR-MIB) will be sent when the policy is triggered.

To unregister a script policy, configure the "event manager policy POLNAME" command preceded with the "no" argument.  When that command is configured, the EEM Server will perform the following steps.

  1. All options after the policy name are ignored.
  2. The policy is unregistered.
  3. If the policy is a system mandatory policy, special rules apply.  When a mandatory policy is unregistered it is not removed from the policy info queue. Instead it is simply marked as not being registered.

NOTE: Simply copying a new version of a script policy into your EEM user policy directory is not sufficient to make the changes go into effect.  The policy must be re-registered first.  To do that, unregister the policy with the "no event manager policy" command, then register it again with the "event manager policy" command.
The Event Manager Update Command

Beginning with EEM 2.4, a shortcut was added to help loading new and modified script policies.  From EXEC mode, the "event manager update user policy name POLNAME repository REPOSITORY" command can be used to copy a policy from a remote server and register it all in one step.  If the policy is currently registered, it will be re-registered using the version copied from the remote server.  For example:

Router#event manager update user policy name mypol.tcl repository tftp://172.18.123.33

If all of your EEM policies are loaded from a central remote repository, you can configure the repository just as you do for the EEM user policy directory using "event manager directory reporitory REPOSITORY" command.  For example:

Router(config)#event manager directory repository tftp://172.18.123.33

The Error Procedure versus the Exit Procedure

Many examples of EEM Tcl policies use the error command to indicate an error occurred.  This function will print out an error message along with a stack trace.  While this can be useful for debugging, the stack trace is not very user-friendly.  Once you have worked out all of the known bugs in your policy, consider replacing the calls to error with puts followed by exit.  If you want, you can add a debug variable and use error if debugging is enabled.  For example, consider the following code example.

if { [catch {cli_exec $cli(fd) "show run"} result] } {
    error $result $errorInfo
}

This code can be re-written to provide users with a friendlier error while retaining the stack trace if debugging is enabled.  Additionally, the code should perform any cleanup operations that may be required (e.g. an open CLI session should be closed).

proc cleanup { fd tty_id rc {msg 0} {errorInfo {}} } {
    global my_pol_debug

    catch {cli_close $fd $tty_id}

    if { $my_pol_debug && $rc != 0 } {
        error $msg $errorInfo
    }

    puts $msg
    exit $rc
}

if { [catch {cli_exec $cli(fd) "show run"} result] } {
    cleanup $cli(fd) $cli(tty_id) 1 "ERROR: Failed to run command 'show run': '$result'" $errorInfo
}

Testing

An easy way to test policies is to temporarily change the event detector from the production ED to the none event detector.  Then, the policy can be executed with the "event manager run POLNAME" EXEC command.  Beginning with EEM 2.4, you no longer need to replace the production event detector with the none ED.  Instead, you can use the multi-event detector feature to add the none ED to your existing policy.  The following are multi-event detector examples with the none ED added.

Applet:

event tag e1 syslog pattern "CONFIG_I"
event tag e2 none
trigger
 correlate event e1 or event e2

Tcl:

::cisco::eem::event_register_syslog tag e1 pattern "CONFIG_I"
::cisco::eem::event_register_none tag e2
::cisco::eem::trigger {
    ::cisco::eem::correlate event e1 or event e2
}

IOS.sh:

##::cisco::eem::event_register_syslog tag e1 pattern "CONFIG_I"
##::cisco::eem::event_register_none tag e2
##::cisco::eem::trigger
##::cisco::eem::correlate event e1 or event e2

While most events cannot be easily generated, syslog messages can be generated easily to make sure your syslog policies are working properly.  You can use the "send log" EXEC command to generate syslog messages to test your policies.  For example:

Router#send log "Configured from console by user"

Debugging

Besides using "poor man's" puts debugging to see what your policies are doing, EEM provides some pretty powerful debugging commands.  To debug a policy that is not being triggered properly, use the following debugging commands.

Router#debug event manager detector DETECTOR
Router#debug event manager policydir
Router#debug event manager server scheduling

Where DETECTOR is the event detector being used by your policy.

If your applet policy is not executing CLI commands properly, enable the following debugging.

Router#debug event manager action cli

If it's a Tcl policy that is having CLI command issues, enable the following debugging.

Router#debug event manager tcl cli

Similar commands exist to debug SMTP (email) interactions.

Applet:

Router#debug event manager action mail

Tcl:

Router#debug event manager tcl smtp
Performance of Regular Expressions

It may be desirable to parse command output in one large block using regexp with the -inline and -all options.  However, if the output is large, this can cause a big performance hit and may result in a CPUHOG.  Instead, consider iterating through the output line-by-line parsing each line with regexp.  For example:

if { [catch {cli_exec $cli(fd) "show run"} result] } {
    puts "ERROR: Failed to execute 'show run': '$result'"
    exit 1
}

foreach line [split $result "\n"] {
    if { [regexp {^snmp-server community public} $line] } {
        puts "WARNING: Found insecure 'public' community string."
        break
    }
}

Generic Performance

The effects on performance depend on the platform, and load level (the number of policies registered and the number of event manager scheduler threads configured). The number of policies that can be registered is limited by the amount of available memory.

NOTE: All user Tcl policies are completely disabled by the no event manager directory user policy configuration command.

The maximum number of policies that can be registered depends on available memory.  Each policy takes a bit of memory and NVRAM. Applet policy memory will increase with the number of actions.

The following example is based in one thread: In most cases a negligible impact. EEM event detector processes run at a medium priority, so they have the potential of affected process like IP SNMP. However, most EEM policies are short (i.e. run in less than 20 seconds), and only execute periodically. For example, a typical policy may only execute when a specific syslog message is generated, or when a specific CLI command is run. Other policies may require periodic polling (i.e. an SNMP policy), but still the object check is quick, and does not typically impact operations.

The example router is a 2801 running 12.4(22)T.  Using built-in EEM system policies, you can determine how many parallel policies can be executed.

Router#show event manager policy available system
No.  Type    Time Created              Name
1    system  Thu Feb 7  01:28:15 2036  ap_perf_test_base_cpu.tcl
2    system  Thu Feb 7  01:28:15 2036  cl_show_eem_tech.tcl
3    system  Thu Feb 7  01:28:15 2036  no_perf_test_init.tcl
4    system  Thu Feb 7  01:28:15 2036  sl_intf_down.tcl
5    system  Thu Feb 7  01:28:15 2036  tm_cli_cmd.tcl
6    system  Thu Feb 7  01:28:15 2036  tm_crash_reporter.tcl
7    system  Thu Feb 7  01:28:15 2036  tm_fsys_usage.tcl

Register the "ap_perf_test_base_cpu.tcl" and "no_perf_test_init.tcl" policies.  But first, set the _perf_iterations environment variable.  A value of 100 is generally sufficient to properly test the event capacity.

Router(config)#event manager environment _perf_iterations 100
Router(config)#event manager policy no_perf_test_init.tcl
Router(config)#event manager policy ap_perf_test_base_cpu.tcl

Then execute the "no_perf_test_init.tcl" policy to start the performance test.  The performance test will run for _perf_interations number of iterations.  Note the time when the performance test starts and when it ends.

Router#event manager run no_perf_test_init.tcl
Router#
*Jan 19 15:20:41.776: %HA_EM-6-LOG: no_perf_test_init.tcl: EEM performance test start
*Jan 19 15:20:42.016: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 1
*Jan 19 15:20:42.252: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 2
*Jan 19 15:20:42.484: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 3
*Jan 19 15:20:42.720: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 4
*Jan 19 15:21:04.772: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 98
*Jan 19 15:21:05.008: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 99
*Jan 19 15:21:05.240: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test iteration 100
*Jan 19 15:21:05.240: %HA_EM-6-LOG: ap_perf_test_base_cpu.tcl: EEM performance test end
Router#

Subtract the start time from the end time to get the total elapsed time.

15:20:42.016 - 15:21:05.240 = 23 seconds 224 msec

Each policy iteration takes roughly 232 milliseconds (23,224 milliseconds / 100 iterations).  This means that this device can service roughly 4 policies per second (1000 milliseconds / 232 milliseconds).

Once you know this number, you can tweak the applet, shell, and Tcl script thread counts.

References

Here is a consolidated list of main references based on Device Manageability Instrumentation features:


The way the Embedded Management Instrumentation Service Requests are handled today by Technical Services is to provide customers support for the functionality of the features, but not for the coding. For coding and design questions, Cisco typically recommend customers consult the documentation, and check on main repositories under:

Cisco Beyond Scripting Community: http://www.cisco.com/go/ciscobeyond, or Cisco Support Community, and
Embedded Automation Systems: http://www.cisco.com/go/easy