Cisco Nexus vPC and non-vPC VLANs together on the same platform (Nexus hybrid setup) – NXOS v7.0(3)4(1)

Cisco discontinued “spanning-tree pseudo-information” starting from NXOS version 7.0(3)4(1) on the 9000 platforms. So what is the solution for Nexus vPC and non-vPC VLANS on the same platform (hybrid)? Is it no longer going to be supported on NXOS/9000 platforms?

Although hybrid is not recommended vPC design for “aggregation layer” but you find a lot scenarios where you need to have both vPC and non-vPC within the same platform (mostly in mid-size data centres; where you have a lot reasons you can’t deploy traditional ethernet switches; hence you have considered the Cisco Nexus platforms).

If you carry everything (both vPC VLANs and non-vPC VLANs) over the vPC peer-links (yes, vPC carry orphaned VLANs as well!) – in this case if there is any issue happen on the vPC peer links and that stops the vPC working, the downstream switches those are connected to the Nexus via non-vPC/non-vPC VLANs will stop forwarding frames due to STP (any STP – STP/RSTP/PVSTP/MSTP) blockage as they don’t have any information about what happened between the vPC peer Nexus switches in a vPC peer failed scenario.

Experts suggest that you should have an additional Layer 2 trunk port-channel alongside your vPC peer-link; this Layer 2 port-channel will carry non-vPC VLANs in case vPC stops working (whatever reason; could be a Nexus reboot during maintenance!).

Well, now you have setup a seperate Layer 2 trunk port channel for non-vPC VLANs and shutdown the vPC peer link to test it – but you found it is still not working as expected! STP is blocking the Layer 2 trunk link, whats the problem! Cisco used to have a solution for this called “spanning-tree pseudo-information“; as I have mentioned in the beginning Cisco discontinued this starting from version 7.0(3)l4(1) on the Nexus 9000 platform. So what is your option for this? Should you stop using Nexus if you don’t follow spine and leaf design?

Yes – there is an answer to this, there is a small trick to make this working! By discontinuing psuedo-information, Cisco basically makes it ever easier to configure; you need to do the following –

i. First, you need to set STP root priority for the VLANs to a lower priority number on one of Nexus switch (default priority is 32768, lower is prefer; this should be done on the primary vPC role Nexus switch) and leave STP untouched on the other Nexus switch (this is the vPC secondary role switch).

ii. Second, on the non-vPC trunk port-channel set “spanning-tree port type normal” on “both” the switches; (“spanning-tree port type network” is recommended for vPC peer-link).

Here is an example config -Here is an example config –

On the vPC primary role switch, apply the following -

(config)#spanning-tree vlan 10,20,30,40-50 priority 8192

On both the switches, apply the following -

(config)#interface port-channel10
(config-if)#description "non-vPC trunk port" 
(config-if)#switchport 
(config-if)#switchport mode trunk 
(config-if)#switchport trunk allowed vlan 10,20,30,40-50 
(config-if)#spanning-tree port type normal

Without having the above configuration applied you will find STP is blocking the non-vPC Layer 2 trunk link even if the vPC peer-link is shutdown. Also in this example – the vPC primary role switch will be the STP “root bridge” because of lower priority configured (8192).

To test your configuration – shutdown the vPC peer-link, run “show spanning-tree vlan xxx” – you should see STP put the L2 trunk interfaces in forwarding state immediately.

Here is a vPC and non-vPC VLANs on the same platform diagram –

CiscoNexus-Hybrid

 

Cisco UCS Platform Emulator – UCSPE 3.1(2ePE1)

Cisco UCS Platforms are expensive kit to play with. UCS Platforms are not just a standalone router, switch or a firewall that could easily be emulated on a PC – they are bunch of kits interconnected together to deliver the UCS platform. UCS is not a server – it’s a system or a platform compared to traditional computing; UCS comes with unified (LAN/SAN/FC/FCoE/HBA/others) and stateless computing together.

Cisco has come up with a solution to help engineers to get their hands dirty on UCS Platforms – the solution is called Cisco UCS Platform Emulator (UCSPE). The latest UCSPE is version 3.1(2ePE1). Cisco made UCSPE available to everyone – you only need to have a Cisco login.

The whole UCSPE comes with the following-

i. 2 x Fabric Interconnect (Model: UCS-FI-6332-16UP)
ii. 2 x FEX (Model: Nexus 2348UPQ N2K-C2348UPQ)
iii. 3 x UCS 5108 Chassis with 12 x different model blades
iv. 1 x UCSC C3X60 server (Model: UCSC-C3X60)
v. 7 x UCS C series servers (220/240/460)

The above are enough to emulate a complete decent size UCS Platform!

Yes! you can create bunch of full fledged “Service Profiles” with different configurations settings and applied them to the servers; also you can configure the fabric interconnect ports with different options (LAN uplink/Server uplink/FC/FCoE/NAS/Port Channels etc)

UCSPE comes in “OVA” and “VMware VMX/VMDK” format (in a ZIP file); you can run it on VMware Workstation or Fusion (I use Fusion).

The pre-defined OVA/VMX requires only one (01) vCPU, 1024MB memory and 3 x vNICs.

You need 3 x IP address (could be from DHCP or static) to make it accessible – one IP address is for Fabric Interconnect “A”, second one is for Fabric Interconnect B and the third IP address is the VIP of FI-A and FI-B Cluster.

Thats all! Its a great tool for candidates learning towards CCNA/CCNP/CCIE Data Centre certifications as well.

UCSPE download link is the following:
https://communities.cisco.com/docs/DOC-71877

Following are few screenshots:

UCSPE-VM-1

[UCSPE VM console]

UCSPE-Platform-01

[UCSPE devices list. They covered a lot devices in here! The red arrow is showing the icon of UCSM – you need to click on this to launch UCSM]

UCSPE-Platform-02

[UCSM – Login Screen]

UCSPE-Platform-Topology-01

[UCSM- The Topology]

UCSPE-Platform-Fabric-A-01

[UCSM- FI “A”]

UCSPE-Platform-01-5108-Chassis

[UCSM – UCS 5108 Chassis]

 

Cisco UCS 5108 Chassis Power Policy Options and Redundancy Demystify

Cisco UCS server chassis 5108 comes with three (03) different Power Policy options; UCS power management is efficient and energy saving by default but NONE of the policy option explanation says which PSUs will be full ON and which PSUs will be put on to Power Saving Mode. This is what official Cisco document says about these options:

  • Non Redundant – All installed power supplies are turned on and the load is evenly balanced. Only smaller configurations (requiring less than 2500W) can be powered by a single power supply.
  • N+1 – The total number of power supplies to satisfy non-redundancy, plus one additional power supply for redundancy, are turned on and equally share the power load for the chassis. If any additional power supplies are installed, Cisco UCS Manager sets them to a “turned-off” state.
  • Grid – Two power sources are turned on, or the chassis requires greater than N+1 redundancy. If one source fails (which causes a loss of power to one or two power supplies), the surviving power supplies on the other power circuit continue to provide power to the chassis.

Probably the above definition applied to old version UCS and not the latest. Also I couldn’t find details on what exactly happen on power redundancy when you have 2 x PSUs and 4 x PSUs installed and your power load is not very high due to not all the blades are installed and functional.

Cisco-UCS-PowerPolicy-1

Following are what I captured regarding which PSUs will be ON and PSUs will be put on to Power-Saving-Mode when you set different “Power Policy” options – this is done a four (04) PSU server chassis. Changing in Power Policy takes effect immediately (might be a 10-20 second delay to refresh the Web GUI) and it doesn’t require any system reboot.

Assumption is all the four (04) PSUs are installed and connected to power socket; also the chassis is running 2 blades.

When Power Policy is set N+1, this is what happen:

PSU1 – ON
PSU2 – ON
PSU3 – OFF (Power Saving Mode)
PSU4 – OFF (Power Saving Mode)

When Power Policy is set GRID, this is what happen:

PSU1 – ON
PSU2 – OFF (Power Saving Mode)
PSU3 – ON
PSU4 – OFF (Power Saving Mode)

When Power Policy is set Non-Redundant, this is what happen (only one is ON! not ALL):

PSU1 – ON
PSU2 – OFF (Power Saving Mode)
PSU3 – OFF (Power Saving Mode)
PSU4 – OFF (Power Saving Mode)

Data centre racks are equipped with two power rails – A (left hand side) & B (right hand side) for redundancy. Now interesting thing is – your physical power connection must be in-line with the UCS Power Policy options, otherwise your blades will be rebooted in case any power issue on “power rail A”.

You can have only two different combinations of power connections to connect all the four (04) PSUs to the power rails A & B. The combinations are following –

Option 1:
PSU1/PSU2 (first two) connected to > power rail A
PSU3/PSU4 (last two) connected to > power rail B

Option 2:
PSU1/PSU3 (odd numbers) connected to > power rail A
PSU2/PSU4 (even numbers) connected to > power rail B

I had N+1 configured with “Option 1” and power issue due to maintenance on rail A rebooted my blades!

This is what happen as following – either “you are saved” during a power failure on rail A! or “you are not saved!

GRID Mode with (Option 1) PSU1/PSU2 to A & PSU3/PSU4 to B;You are saved!
PSU1 – A ON
PSU2 – A OFF (Power Saving Mode)
PSU3 – B ON
PSU4 – B OFF (Power Saving Mode)

GRID Mode with (Option 2) PSU1/PSU3 to A & PSU2/PSU4 to B;You are NOT saved
PSU1 – A ON
PSU2 – B OFF (Power Saving Mode)
PSU3 – A ON
PSU4 – B OFF (Power Saving Mode)

Screenshot of GRID mode following –

Cisco-UCS-PSU-GRID-Pic1

N+1 Mode (Option 1) PSU1/PSU2 to A & PSU3/PSU4 to B;You are NOT saved!
PSU1 – A ON
PSU2 – A ON
PSU3 – B OFF (Power Saving Mode)
PSU4 – B OFF (Power Saving Mode)

N+1 Mode (Option 2) PSU1/PSU3 to A and PSU2/PSU4 to B;You are saved!
PSU1 – A ON
PSU2 – B ON
PSU3 – A OFF (Power Saving Mode)
PSU4 – B OFF (Power Saving Mode)

Screenshot of N+1 mode following –

Cisco-UCS-PSU-N1-Pic1

Non-Redundant Mode (Option 1) PSU1/PSU2 to A & PSU3/PSU4 to B;NOT saved!
PSU1 – A ON
PSU2 – A OFF (Power Saving Mode)
PSU3 – B OFF (Power Saving Mode)
PSU4 – B OFF (Power Saving Mode)

Non-Redundant Mode (Option 2) PSU1/PSU3 to A & PSU2/PSU4 to B;NOT saved!
PSU1 – A ON
PSU2 – B OFF (Power Saving Mode)
PSU3 – A OFF (Power Saving Mode)
PSU4 – B OFF (Power Saving Mode)

Screenshot of Non Redundant mode following –

Cisco-UCS-PSU-NonRedundant-1.jpg

Cisco IOS Events to Splunk – Track IOS Command Execution History

Cisco IOS event details can be send to an external system via “syslog”. Splunk server itself and Splunk Universal Forwarder both can act as a syslog server to accept logs from Cisco IOS devices.

To add more cream to Splunk log consolidation solution for Cisco IOS devices – there are few Splunk plugins already available on Splunk App store! These plugins display IOS events on nice colorful dashboards with graphs & charts.

Let’s talk about how we can get this solution in place.

Technical dependencies to get this solution are following –

1. Cisco IOS devices (routers, switches, wlc, asa) configured to send IOS event to Splunk via “syslog”
2. Splunk Indexer (actually this is the Splunk server)
3. (optional) to get nice dashboards it needs two Splunk Apps – (i)Cisco Networks Add-on (TA-cisco_ios) (ii)Cisco Networks (cisco_ios)

Regarding the solution design, there are two options as following –

1. Send logs to Splunk via Splunk Universal Forwarder; this design suits very well in a large infrastructure. Splunk Universal Forwarder can act as local “syslog” for IOS devices; picture below-

splunk-uf-pic-1

2. Send logs directly to the Splunk server –

splunk-server-pic-1

Installation technical procedures are following –

Step 1: Configure Cisco IOS to Send Logs to Splunk “syslog”

Following is an example configuration on a Cisco router –

router# config t
router(config)# logging trap notifications
router(config)# logging 1.1.1.1   ;IPAddr of Splunk syslog – if syslog is running other than UDP 514 – this needs to be specify here

The following commands will send Cisco IOS command execution history to syslog –

router(config)# archive
router(config-archive)# log config
router(config-archive-log-cfg)# logging enable
router(config-archive-log-cfg)# logging size 1000
router(config-archive-log-cfg)# hidekeys ;this will not send passwords to syslog
router(config-archive-log-cfg)# notify syslog
router(config-archive-log-cfg)#exit

Step 2: Configure Splunk or Splunk Universal Forwarder to Accept Logs on UDP://514

There are multiple ways to ways to do this. Adding new listener & sourcetype to “inputs.conf” works for both universal forwarder and Splunk server running on any platform.

On Linux/Unix the default location of this file is – $SPLUNK_INSTALLATION_DIR/etc/system/local/

On Windows the default location of this file is – x:\Program Files\SplunkUniversalForwarder\etc\system\local\

Add the following to the “inputs.conf” file –

[udp://514]
sourcetype = cisco:ios

Restart “splunk” service or “SplunkUniversalForwarder” service to get this change take effect.

If you add “sourcetype = syslog” – this will also work. The “Cisco Network Add-on (TA_cisco-ios)” transforms Cisco syslog to “cisco:ios” sourcetype automatically.

At this stage you should start getting logs coming on to Splunk. Execute some random commands on Cisco IOS and search for sourcetype=”cisco:ios” on Splunk search tab – you should be able to see logs like similar to following –

splunk-search-ciscoios-2

Step 3 (optional): Install Splunk Cisco Apps to Display IOS Events on Dashboards

Install the following two Apps from “Apps > Find More Apps > search Cisco” –

  1. Cisco Network Add-on (TA-cisco_ios)
  2. Cisco Networks (cisco_ios)

Installation is very straight forward – just click on the icon to install it.

If you still not seeing any logs on the Dashboard of Cisco Networks – this might be incorrect “sourcetype” issue and “TA-cisco_ios” is not doing the source type transformation – in this case change your source type to “cisco:ios” manually or you can log a support case with Splunk support to get the TA-cisco_ios fixed for you.

You should be able to see the following on Dashboards –

(the main dashboard)
splunk-cisco-dashboard

(command execution history – who has done what?)
splunk-cisco-exechistory

There are lot more you can find here on this dashboard – explore it.

Cisco New Aironet 1700 Series & WLC Software Compatibility Matrix

Cisco recently released Aironet 1700 series access points which support high speed WiFi IEEE 802.11ac. One of the key specifications of this 1700 series – they “only” work with Cisco Unified Wireless Network Software Release 8.0 or later.

It’s now time to upgrade your WLC to version 8.x to get this work for you.

Here is the latest APs & WLCs software compatibility matrix (as of December 15, 2014) –

cisco-aironet-wlc-software

Details @ http://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html

Cisco AIP SSM Email Alert – Cisco IPS Manager Express (IME)

Cisco AIP SSM are pluggable hardware modules for advanced intrusion prevention security services (IDS/IPS) to Cisco ASA 5500 series firewalls.

Although lots of AIP-SSM configuration parameters can be set via ASDM > IDM (Cisco IPS Device Manager) or via CLI – however there is no such thing on IDM or CLI to send security events or security reports via email.

So, how do I know > what is happening on the IDS/IPS? Is the IPS device capable of detecting threats? Is IPS is blocking attacker IP address?

The answer is – there is a separate piece of software called Cisco IPS Manager Express (IME) to manage, configure and send email alert notifications for AIP-SSM modules. This software needs to be installed on a Windows machine. As of today the latest version is 7.2.7. Supported windows platforms are – Windows Vista Business+/XP Pro/Windows 8+/Windows 2003 R2/Windows 2008 and above.

Apart from all ASA AIP-SSM modules; this IME software does support following Cisco IPS hardware platforms – 4240, 4255, 4260, 4270-20, 4345, 4360, 4510 and 4520.

Here is the download URL (you need valid Cisco login),

https://software.cisco.com/download/type.html?mdfid=282052550&catid=null

Installation is very straight forward; start the installation > follow next, next and finish.

Once IME installation is finished; add all of your AIP-SSMs or IPS devices to IME console via IP address. Make sure IME Windows machine is able to communicate to AIP-SSM or IPS device’s management interface IP address. You can have bunch of IPS devices under one IME.

Setting Up Email Notification

This is very easy task. Open IME console > click on “Tools” > click on “Preferences”; enter your SMTP server details under “Email Setup” tab; screenshot–

IME-EMailSetup

You should send test email to confirm – IME is OK sending email.

Click on the next tab “Notifications” for IDS/IPS security events – configure your preferred notification parameters here; screenshot-

IME-Notifications

Lastly you might want to see consolidated security events in a report – such as what happened in last 24 hours or last 7 days or last 30 days; go to the next tab called “Reports” – all the report parameters are here; this will send PDF report with colorful presentation of data with graphs or charts; screenshot–

IME-Reports

Questions again –

i. You have done configuration of all the email notification parameters, do you need to keep IME running on desktop? Should you close the IME console and logoff?

Answer: Yes – you close IME console and logoff from the Windows computer; IME is still running on the background as a Windows service.

ii. You have added 4 IPS devices on your IME – is email alert notification working on ALL of them?

Answer: Yes – email notification is a global setting within IME that applied to ALL IPS devices those been added to the console. There is no option here to configure email notifications on individual IPS device within the same IME console.

Cisco IOS Site-to-site IPSec VPN with VRF-lite

Just few years ago if there was a requirement of connecting to destinations with same IP networks address or for a low-level network segregation – the solution was to get separate network devices. These days the same can be done on a single hardware platform using VRF (VRF-lite).

On server platform – it’s virtualization everywhere these days; why not VRF-lite on networking then! I have seen lots of routers they never use above 50% of its capacity! This saves us the following –

i. Buying new router hardware
ii. Less power consumption, less power outlet
iii. Less number of switch ports required
iv. Overall high gain total cost of ownership

That’s why I have started implementing VRF-lite on all my new implementations! Why “all” – because if there any new requirements comes into the picture I still can use the same device; no need to reconfigure the existing platform or buy new devices.

So far I experienced – Cisco IOS does support all IP features with VRF-lite; such as static routing, dynamic routing, BGP, site-to-site vpn, nat and packet filtering firewall. On HP Comware5 platform (A-Series, 5xxx) – VRF-lite doesn’t support Layer-3 packet filtering – other than this they support most of IP services.

Let’s talk about IPSec site-to-site VPN with VRF-lite. Following are the key configurable components of a site to site IPsec VPN –

  1. Remote peer with secret keys
  2. IKE Phase 1 security details
  3. IKE Phase 2 security details
  4. Crypto map
  5. NAT
  6. Access List

On a VRF environment – the whole VPN concept and commands remain same except the following only where we specify network addresses –

1. Remote peer & keys – remote peer is reachable via which VRF domain; instead of global “key” we need to configure “keyring” with specific vrf domain name here.
5. NAT – internal source address belongs to which VRF domain; we need to specify vrf domain name in the NAT rules.
6. Although “access-list” contain of IP addresses – no VRF name need to be specify here.

Following are the command syntaxes for remote peer with VRF details –

(config)#crypto keyring tunnelkey vrf my-vrf-A
  (config-keyring)#pre-shared-key address 10.100.200.1 key 6 mysecretkey
 
 (config)#crypto keyring tunnelkey vrf my-vrf-B
  (config-keyring)#pre-shared-key address 20.100.200.1 key 6 mysecretkey

Without VRF the syntax is (this is called “global key”)-
(config)#crypto isakmp key mysecretkey address 10.100.200.1

Following are the command syntaxes for NAT rules with VRF domain name –
(config)#ip nat inside source static my_src_ip  my_nat_inside_global_ip vrf my-vrf-A

(config)#ip nat inside source static 192.168.1.10 10.100.200.10 vrf my-vrf-A

Show configuration commands for the above are –

#show crypto isakmp key
#show ip nat translations vrf my-vrf-A

If the above are not specified correctly – you might receive the following error on the router log file;

No pre-shared key with 10.x.x.x!
Encryption algorithm offered does not match policy!
atts are not acceptable. Next payload is 3
phase 1 SA policy not acceptable!
deleting SA reason “Phase1 SA policy proposal not accepted” state (R) MM_NO_STATE (peer 10.x.x.x)