UCS management paradigm

UCS Manager (UCSM) exhibits a very unique and interesting set of features for ease of deployment in the cloud. UCSM stores configuration, device information, statistics and policies in an object oriented data model. The "management brain" of UCS is completely data driven. Only interface to UCSM is through XML APIs, and both GUI and CLI internally use XML to communicate with the core UCSM process. With that high level background, let me get in to the details of some interesting characteristics of UCSM.

Named References

UCSM makes maximum use of named references. Templates, pools and policies are used to loosely bind the configuration and to easily share common data. Policies dictate behavior and multiple configuration end-points can share the same behavior. A change in policy would not require to revisit all the end-points that refer to the policy. There are standalone policies for global configuration, for example chassis discovery policy, VM life cycle policy etc. The VLANs are also referred by name. For example, all database servers can be in a "dbNet" VLAN with VLAN id 10 and all the service profiles corresponding to database servers would have VNICs referring to the VLAN by the name "dbNet". Once this is in place and servers are up and running - if network admin changes the network architecture and VLAN id changes from 10 to 20 -- he/she wouldn't have to revisit all the servers to change - it would be changed at only one place.

Policy resolution in hierarchical org structure

UCSM allows to reflect hierarchical organizational structure of a company (or tenants in case of cloud service provider) in the managed object model. For example, you can have classic "coke" and "pepsi" top level orgs. Under "coke", you can have "operations", "research", "legal", "marketing" orgs. Now let's say there are "streaming" and "http" QoS policies defined at "coke" level, where "http" restricts the bandwidth to 1Mbps. But, in the "research", there's a requirement to let web traffic flow up to the line-rate, so, administrator can create a QoS policy with same name at the "research" org level. When policy is referred by name, policy gets resolved to the closest org level that matches the name. So, a port-profile defined in "research" level or sub-org levels would enjoy line-rate if it refers to "http" QoS policy.


Also, the policy gets re-resolved if another policy is added with same name in the org hierarchy. If the previous example, if "http" QoS policy is deleted from "research" level, the port-profiles referring to them would automatically resolve to "http" policy at the "coke" level.

Loose Referential Integrity

Most management system would not let you delete a policy if it is referred by other configuration. However, UCSM does not enforce such strict referential integrity. UCSM referential integrity works in following loose manner:

• For every policy/pool type, there exists a predefined policy at the root level with "default" name.
• If the system doesn't find a named policy in org scope of configuration that refers a policy by name, then default policy is used.
• If a referred policy gets deleted (and no other policy in org scope exists at any other org level), then configuration referring to such policy resolves to the default policy.
• If a more relevant (closer) named policy is defined with respect to the configuration, then policy gets re-resolved to the "closer" policy in org hierarchy.
  

Of course, when system dynamically resolves policy, then it must have mechanism to inform administrator about its resolution. Two mechanism are used for this:

1. An operational property is defined for every named reference which specified distinguished name of the policy that system resolved.
2. If a specific named policy is not found and default is used, then a fault is raised.

Asynchronous Configuration Deployment

In server management systems, many operations take long time to finish - like VM deployment, server reboot etc., so north bound APIs can not be blocked for such extensive period of time. UCSM provides completely asynchronous north-bound experience. It differs from Cisco's networking gears in this regards - for example, when user issues command to create a VLAN, the management system would check the range and maximum VLANs etc, if user input is good, it immediately unblocks the user, effectively telling "consider it done". Later, it deploys the VLAN on the switch and it would only fail if there are other serious issues like control plane running out of memory etc. and if that happens, faults are raised.

Putting it all together


UCSM is designed to keep cloud and data-center virtualization in mind. It is extremely friendly to automation given its XML APIs, data-driven model and asynchronous nature. Loosely coupled policies and maximum usage of pools, policies and templates make it and ideal fit for server procurement in cloud, especially in multi-tenant systems. Field has very warmly welcome these characteristics. Just as a sample, Here is a blog that talks about integration of XML APIs in the power-shell.

Network Virtualization in Cisco's UCS

I would like to start this blog with my favorite project - Network Virtualization in Cisco's Unified Compute System (UCS). Particularly, I would like to write about integration of UCS with VMware's vCenter to achieve seamless network virtualization in data centers. Before I go into the details of the "solution", let me go over the "problem" first.

Issues with virtualization in data center network

As soon as virtualization is added on a host, a software switch or "soft switch" had to be added in the host kernel. Refer to the diagram on left. As soon as you have multiple VMs running on a host (an ESX host in case of VMware), switching must happen in the host kernel if one VM wants to communicate with another VM because of the Ethernet switching 101 - a frame is never sent back on the same interface from where it came from (it would be either sent to a specific interface where destination MAC address is learned, or flooded to all the interface except from where it came from in case of unknown unicast, broadcast and multicast). So, a soft switch was mandated from the day one in VMware ESX host.

However, as soon as the soft switch was added to the host, another issue surfaced in the way data centers are managed. The line in the middle of the diagram is the boundary between server admin team and network admin team, and now a switch is lying in the server admin's domain. In the good old days before virtualization, the port connected to the host was typically put in access mode, with one access VLAN assigned to it and specific QoS and access control policies attached with it. Now, because of the soft switch in the host, the trust boundary has to be extended to the soft switch, multiple VLANs have to be allowed on the port connected the ESX host, QoS trust boundary has to be extended to the soft switch (or more expensive NBAR based classification has to be performed), and access control policy enforcement becomes complex. The link between switch and ESX host becomes a dump pipe (that's why I have drawn it that way), and network policy enforcement become responsibilities of soft-switch - which is typically managed by server admin.

Cisco's UCS solves this problem and streamlines the data center virtualization by offering hardware VNTag solution.

UCS Network Virtualization

In Cisco's Unified Compute System (UCS), customers have choice to replace the ESX host's soft-switch by the hardware based switching in the UCS access switch, aka UCS Fabric Interconnects. In the ESX host, a Cisco's kernel module - Virtual Ethernet Module or VEM is added.

VEM works as a replacement of soft-switch, where instead of performing switching in the host, for every north bound virtual NIC created on the VEM, a south bound instance of VNIC is created in the Cisco's M81KR virtualized adapter. In the current releases, up to 56 such VNICs can be created on the M81KR adapter. The adapter in turn requests the attached fabric-interconnect (access switch) to create dynamic virtual Ethernet interfaces (vEths). vEths are logical entities created in ASIC and enjoys the same status as physical interfaces, so now the frames from one VM to another is switched in the ASIC on the access switch. All the rich set of switching features of Nexus 5000 switch is now available to the vEths, which is managed by the network admin team. M81KR is also industry's first implementation of VMDirectPath technology, which essentially bypasses the switching module in the hypervisor and provides near bare metal performance to VM-VNICs. In addition to solving the domain issues and providing faster switching in hardware, UCS provides smooth integration with vCenter and port-profile based network management.

UCSM / vCenter integration and port-profiles

Another benefit of using UCS in data center is its integration with VMware's vCenter and port-profile based VM network management. A UCSM first registers itself to vCenter as a management extension and UCSM can create multiple distributed virtual switches in the vCenter. A distributed virtual switch is a VEM that expands across multiple ESX hosts. At this point, network admin can create multiple port-profiles. Port-profiles are grouped network policies and configuration identified by a name. For example, "finance" and "hr" port-profiles would contain VLANs, QoS policies, L2 security policies etc. for those two groups. UCSM pushes profile names and description to vCenter, as server admin wouldn't be interested in the nittygritty of network configuration.

When server admin deploys a VM, she would create VNICs for the VM and assign a port-profiles to VNICs. When VM instantiates on the hypervisor, VEM and M81KR would request the access switch to dynamically create vEth interfaces and provide the corresponding port-profile names. As access switch has complete configuration of port-profiles, the vEths would inherit the configuration from the corresponding port-profiles, and thus, the whole loop finishes. As you can see, this architecture has several benefits:

• It decouples server and network admin's work flow and eliminates inter-dependency. For example, network admin can easily change VLAN id in a port-profile without bothering the server admin team to make any changes on server side.
  

• When vMotion happens, the VM-VNIC's configuration also smoothly moves from one host to another. A vEth interface is detached from one access port and is attached to access port connected to the destination hypervisor. All the state and stats of the vEth is preserved after vMotion. Any external script or entity doesn't need to change configuration of access ports with vMotion.
• UCS (which includes the access switch) is aware about the virtualization. You can see the association of physical servers, their associated service-profiles, hypervisors, VM instances, VNICs and port-profiles in the UCSM UI.

You can find white papers that describe more technical details of this solution on Cisco website, my goal here was to iterate over the latest end-to-end virtualization technology as simply as possible.

Disclaimer: Anything explained/expressed here is not an official form of communication by/from Cisco Systems, Inc.