On the Job with a Network Manager Activities performed by network managers Tools used by network managers How management tools and management technology are used to achieve a given purpose Support for network personnel The organization of tasks and workflows How network management personnel work together Procedures that management personnel must follow to complete a job After reading this chapter you should gain a better understanding of the following: The types of operational tasks network personnel face on a day-to-day basis How network management technology supports network operators The different types of management tools that network personnel use to do their jobs The three different scenarios provide an overview of the role that a network manager plays within an organization Responsibilities differ widely depending on the type and size of the organization The examples are illustrative and by no means comprehensive Pat is a network operator at the Network Operations Center (NOC) of GSP Her group is responsible for monitoring the global backbone network and the access network GSP is a multibillion-dollar business that provides global services with “five nines” service availability Any disruption to service would cause revenue losses in the millions of dollars, expose GSP to penalties and liability claims, and put jobs in jeopardy Pat works in command central Big screens map the main sites of the network and show statistics on network utilization Map items appear in green when everything is operational Occasionally entire nodes on the map turned yellow or red when connectivity is affected A smaller screen on Pat’s desk shows a list of trouble tickets, which are used to track and resolve current network problems The trouble tickets have two sources ◦ customer-reported problems and ◦ Network problems Reps provide “tier 1 support” for customers If a service rep cannot help a customer, the customer is transferred to a rep with more troubleshooting experience Pat’s group provides the third tier of support Trouble tickets typically contain ◦ a description of the problem ◦ who is affected ◦ contact information Sometimes this information is missing or in error The second source of tickets Reported by systems that monitor alarm messages sent by network equipment The problem with alarm messages is that they rarely indicate the root cause of the problem In most cases, they reflect symptoms that could be caused by any number of things Pat doesn’t see every single alarm in the network The alarm monitoring system tries to group messages that appear to point to the same underlying problem Viewing grouped messages is much more practical than addressing individual alarms In the past, Pat has seen 20 tickets all related to the same problem GSP has recently made significant progress in reducing redundancy Despite the progress made many tickets still relate to the same underlying cause Pat must first acknowledge that she has read each ticket If she does not acknowledge the ticket, it is automatically escalated to her supervisor Next Pat analyzes the ticket information She uses a monitoring application for the affected part of the network to see what is going on She starts by verifying the symptoms that are reported in the network In some cases, Pat decides that a piece of equipment needs to be replaced If equipment needs replacement Pat opens a work order system to create an order for a new part It is not Pat’s responsibility to dispatch a field technician Sometimes technicians are dispatched to a remote location to inspect and/or replace the equipment In other cases the rack-mounted equipment in the NOC is affected When the technicians exchange a part, they use a bar-code scanner that automatically updates the central inventory system Pat enters what she did and when resolution is expected on the trouble ticket For now, she is finished When the work order is fulfilled a notification from the system will be sent to her in-box Pat used to close the tickets before receiving notification from the work order system She now understands that procedures are essential for GSP to control quality of the services it provides Chris and a colleague are responsible for the computer and networking infrastructure of the RC Stores headquarters and 40 branch locations RC Stores’ network contains almost 100 routers ◦ Typically, an access router and a wireless router in the branch locations and ◦ additional networking infrastructure in the headquarters and at the warehouse A managed service provider (MSP) is used to interconnect the various locations of its network ◦ The MSP has set up VPN tunnels between the access routers at each site that connects the branch locations to the headquarters ◦ The entire network can be managed as a single network Chris runs a management platform from a workstation at his desk ◦ The application provides a graphical view of the network that displays the network topology ◦ Chris has built a topology map that groups multiple routers into “clusters” ◦ A typical management application screen is shown in the next slide Management applications support some statistical views as well Cisco Packet Telephony Center Cisco Works IP Performance Monitor Chris adds a phone for a new user and troubleshoots a problem that he suspects might be caused by problems on MSP’s network Chris handles multiple responsibilities because of the smaller size of his organization and network Sandy works in the Internet Data Center for a global Fortune 500 company, F500, Inc. The data center ◦ is the center of the company’s intranet, extranet, and Internet presence ◦ hosts the company’s external website and connects customers to the online ordering system ◦ is host to all the company’s crucial business data ◦ hosts the company’s internal website which provides access to the data given the proper access privileges Since the functions provided by the data center are crucial to its business, F500, Inc. decided not to outsource them The way F500, Inc. organizes and manages its processes and supply chains differentiates it from its competitors Sandy is to develop a plan to accommodate a new supplier It will involve configuring ◦ server and storage infrastructure critical for the business relationship as well as ◦ an extranet over which the shared data can be accessed Sandy has a list of the databases that need to be shared and backed up and storage and network capacity must be assessed Sandy’s main concern is security Sandy needs to set up a new VLAN to separate traffic on the extranet from other network traffic A typical networks configuration screen is shown in the next slide Access control lists (ACLs) on the routers need to be updated to reflect the new extranet security policy Finally, authentication, authorization, and accounting (AAA) servers need to be configured Sandy must determine where the data will be hosted and how her decision will impact the internal data center topology Sandy uses a performance-analysis application to make these determinations Data are stored in multiple places. This will be transparent to applications. A common uniform resource identifier (URI) will be used to address the data Pat, Chris, and Sandy handle their jobs in different ways The differences depend upon the size and purpose of the network and the network functions that individual manages. Chris manages numerous aspects of his network. Different tools are used by the managers to carry out the required management tasks Pat’s job is determined by guidelines, procedures, and the way the work is divided at her organization Both manually-executed and automated tasks exist Management tasks required abstract concepts that had to be broken down into numerous subtasks Running a network involves monitoring operations, diagnosing failures, configuring services, analyzing historical data, planning, security, and workforce management Integration affects operator productivity ◦ Pat used integrated applications to manage trouble tickets ◦ Chris had to repeat some work because of a lack of integration in his phone system applications Device managers and craft terminals Network and element analyzers Management Platforms Collectors and Probes Intrusion Detection Systems Performance Analysis Systems Alarm Management Systems Trouble Ticket Systems Work Order Systems Workflow Management Systems and Workflow Engines Inventory Systems Service Provisioning Systems Service Order-Management Systems Billing Systems Craft terminals provide a graphical view of the physical configuration of network equipment Craft terminals are used to ◦ ◦ ◦ ◦ ◦ log into a device view its current status change configuration settings perform diagnostic self-tests and downloading software images Managed equipment may provide a “built-in” craft interface that renders a view of the device Field technicians typically load craft terminal software onto their notebooks so they can connect to a device and managed it via a USB or serial interface CiscoView for Catalyst 6500 Network or packet analyzers are used to view and analyze network traffic for diagnostic and and troubleshoot purposes Network analyzers ◦ capture or “sniff” packets that traverse the ports of a router or switch and ◦ Display packets in a human-readable format that an experienced network operator can interpret Similar to craft terminals but ◦ contain a database for retaining device configuration information ◦ receive event messages from devices on the network and ◦ discover equipment deployed on the network Element Managers may have an interface that allows applications to manage equipment through the element manager. This provides ◦ data synchronization between apps ◦ a simpler user interface and ◦ reduced management load General-purpose applications used to monitor and manage the network, comparable to element managers May be proprietary Often include toolkits that extend the management platform Often compared to a PC operating system because it provides a means for communicating with network devices and tracking communications via an internal database Collectors gather and store network data ◦ Netflow is a common ◦ It gathers data about the traffic traversing a router or switch Probes are active collectors ◦ They can trigger events and collect the responses Provide information regarding suspicious activity on a network, such as ◦ break-ins to routers or servers and ◦ denial-of-service attacks IDS listen ◦ for alarms ◦ inspect activity logs and ◦ observe load patterns to determine if a network has been compromised Performance Analysis Systems Enable users to analyze traffic and performance data to recognize trends and traffic patterns Alarm Management Systems Collect, monitor, and group large volumes of data that pertain to alarms that are triggered by different events on the network May be provide an initial diagnosis of the cause of an alarm Trouble Ticket Systems Support network problem resolution Usually generated by users of the network who experience a problem Assign tickets, escalate tickets that take too long to resolve, and report statistics about the resolution process Work Order Systems Assign and track maintenance jobs in a network Help organize workforce that performs jobs Track the maintenance process Help manage execution of predefined processes that consist of multiple steps and different owners Not specific to networks Based upon concept of finite state machines Events are managed by a workflow engine Inventory Systems Used to track assets which may be either physical or services (such as a phone service) Service Provisioning Systems Facilitate the deployment of services, such as DSL, over a network Typically found in operational support environments of large service providers Service Order-Management Systems ◦ Used to manage orders for customers of large service providers, generally not found in enterprise environments ◦ Manages workflows and processes of an organization Billing Systems ◦ Essential for collections of revenues ◦ Analyze accounting and usage data to identify which communications services were provided to whom ◦ Can be helpful in fraud detection Three scenarios illustrating how networks are managed in practice and the variety of tasks that are involved were discussed The service provider scenario emphasized workflows, processes, and interactions The medium-size enterprise scenario was characterized by a variety of tasks that had to be performed by one individual and a greater reliance on the individual expertise and intuition The Internet Data Center scenario focused on the planning phase of a network’s life cycle as well as management practices for both the network and the devices and applications connected to it The scenarios illustrate that network management involves management technology as well as organizational and human factors In each case personnel were supported by a variety of tools A wide variety of different tools exist Running the largest, most complex networks usually involves hundreds of management systems and applications