NOC Theory and Practice
Part II: The Nuts and Bolts
(The Practice Of Making a Good
NOC)
NOC Theory and Practice NANOG 24 - Miami, FL - TCB
DISCLAIMER
I am not affiliated with any corporation. My
views are my own.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Intro - What’s In A NOC?
• Purpose - What Type of Company Are
You?
• People will always be your key asset.
• Systems make the job of people easier.
• Managers assist in identifying gaps in
both.
• Processes allow for flexibility in
operations while still maintaining
standards.
• And, last but
notandleast,
the24facility
NOC Theory
Practice - NANOG
- Miami, FL - TCBis
The Facility - Think Data Center
• Location, location, location.
• Security
• Environmental Concerns
 This applies to both your inside and outside
environment.
• Ergonomics
• Reliability
• Sources of Labor
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The Goals of a NOC
Our conception is:
• At the bare minimum, you will receive
requests from your customers
• You will process those requests in some
fashion (engineering, handoff, ...)
• You will communicate resolution to the
customer, and/or keep customer updated
on status
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Goals II
• NOC as first, second, or third tier support.
 Conception of NOC as “do everything” group
 The engineers should be smart at designing,
configuring, etc. - NOC should be divided into
categories of engineers able to resolve issues
 A smart issue routing process makes this a
very effective model
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The Process
• Receive events from your “inputs” (phone
calls, network monitoring, etc)
• Process inputs into a problem to be
resolved
• Route the problem to the correct individual
• Resolve the problem as appropriate
• Communicate status continually.
• Record the problem for later review and
training.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The Key to Success Is
By constant re-examination of your
operation, your employees, and
what inputs you are getting, you
can reach incredible efficiency.
Communicate this to other teams
and your operations will prosper.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The Facility (cont.)
• Security applies to keeping other people
out, but it also applies to making a safe
arrival for your employees, at any hour.
• Environmental concerns: Just like a
datacenter, consider climate control and
earthquake/act of god resistance.
Disaster planning rules should be in effect.
• Sources of labor: Not just engineers, but
builders.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
A Word on Portability
• As Matt suggests, portability is critical.
• Obvious resources: Computers,
communications, phones.
• Not obvious resources: Vehicles and quick
transport/assembly, good cases for shipping or
relocation, breakdown/reassembly schedules.
• Multiple everything.
• Make sure your resources are distributed. If
your network is large, your footprint of mobility
should be large.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Hiring the Right People
• Skillset
 but more importantly, flexibility
• Stress
• Scheduling
• Stability
NOCs are hard things to staff appropriately.
And good people are hard to retain.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Hiring (cont.)
• Make sure they have lives. But know what you
can’t ask.
• Can they talk? How are their soft skills?
• Do they get flustered easily?
And remember, org charts suck. Classify people
according to their skills, not according to some
hierarchy which has no meaning outside of
some middle-management mind.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Keeping Your People Happy
• Environment
• Ergonomics
• Creature Comforts
 We’re not kidding.
Give some thought to a 24x7 outlook. This
means beds, showers, food, and climate.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Environment - People
• Climate controlled centers get cold at
night. Make sure you have 24H control
over your space.
• When thinking about climate, keep in mind
your working conditions are slightly
different in a NOC.
• 2:00AM to 4:00AM are the critical hours.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Ergonomics - People
• Hire a good ergonomic consultant.
• Positions of lights, phones, keyboards,
monitors, pens - all this matters.
• Observation is key. Your engineers will
show you what the problems are.
(Cameras)
• Seating. If you’re in a chair for eight hours
at a stretch, don’t you want it to be a
comfortable one?
• Screen space.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
About People. About NOCs.
• It’s a messy business. It’s high stress. It’s easy
to let things get out of hand.
• Good management qualities in a NOC:
 cool heads
 methodical approach to solving problems
 having been on the customer side
 micromanagement sucks.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Let’s go over retention.
• Retention of NOC staff is 100% harder. Very
few people live for this kind of job.
• So, innovate. Think outside the box.
• People want to grow. Know what your people
want to do when they grow up.
• Remember your core focus. Do operations, not
sales, or account services, or anything else.
• At the same time, make sure your folks are
informed on everything. Everything comes out
in the wash. All the dirty clothes come to
NOC.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Interviewing NOC folks
• You are interested in three things.
 The candidate’s willingness to learn.
 The candidate’s 1-2 year outlook on their career.
 The candidate’s ability to work funny schedules.
• They are interested in three things.
 Will you offer them growth potential?
 Can you give them enough perks to enjoy their work?
 Do you have your stuff together?
 Are the procedures well-documented?
 Will they be called in outside their shift?
 Will they ever wonder who to go to?
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Productivity
• Momentum is key. Keeping your people busy is
the most important thing towards maintaining
productivity.
• The other component to momentum is keeping
them working on different things. This makes
them better engineers, and helps feed ego.
• “Everything at your fingertips” - your engineers
should never have to make a customer wait
more than four minutes on hold for anything. If it
has to wait longer than that, tell the customer
why, and mark it as something to fix.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The View of the Customer
• A customer wants
 Professionalism: your engineers must treat
the customer as if they are paying you a lot of
money and the engineer wants the customer
to continue to do that.
 ETR: Your customer wants to know when the
problem will be fixed. “I don’t know” is not an
acceptable answer. The ETR is the first thing
you give to a customer and the one thing you
update him on at least every hour (or less if
they prefer).
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
The View of the Customer II
 A description: Concise or not, the customer
wants to know what the problem appears to
be.
Remember your focus. It is not (typically) your
responsibility to engineer your customer’s
network for maximum reliability. Your
responsibility is the operation of that network.
The only thing you should do as a NOC is fix
problems when they arise in the most expedient
and professional manner possible. A customer
wants to beNOC
comfortable,
and assured that that
Theory and Practice - NANOG 24 - Miami, FL - TCB
Things we never, EVER allow a
customer to hear
• I don’t know when it will be fixed.
• I can’t help you.
• I’m not qualified to answer that question.
• Let me transfer you to someone else.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
It’s not always ideal. But • Where possible, the first person to take the call
should be the person who follows it to
completion. The customer hates being thrown
between different engineers - but not having an
answer is worse.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
A Little About Systems
• Don’t just focus on ticketing.
• Make sure your people have information
on everything they need to operate.
• If all the architects fell off the planet today,
could you rebuild?
• It’s about agility.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Policies and Procedures
• If you don’t write it down, it isn’t a
procedure or a policy. There are no
unwritten rules in a NOC. People cycle.
• Focus on procedures that affect the
handling of a customer. The soft skills are
usually the hardest to learn.
• Full disclosure?
• Troubleshooting Techniques
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
And a word on incidents.
• Incidents, tickets, call it what you will.
• Whatever happens in the NOC:
 gets summarized to the account exec if the
issue is of appropriate severity
 gets recorded - nothing EVER gets deleted
from a customer log
 gets archived
 gets resolved - how much time will you spend
on issues that never get fixed?
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Some final notes - Conclusion
• It’s easy to throw a bunch of people in a room. It’s hard
to build an efficient, productive, and well-oiled NOC (the
same goes for networks)
• Record everything. Write everything down.
• Try to remain as flexible as possible.
• Look to your employees for cues on what their ideal
environment should be.
• Remember your focus, and know your customer.
• Know what not to say.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Credits
As far as the practice of NOC building goes,
we must thank:
• herb@tomobiki.urusei.net - thanks for
great private discussions on NOC building
• nanog subscribers - for talking about
operational stuff
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
And finally
• Sean Donelan - for giving network
operations folks an impetus for existing
(due to his backhoe skills)
• Shouts to EFNet IRC #nanog: you know
who you are
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Part III: Evaluation Criteria
• Social engineer your own NOC.
• We like to rate NOC folks on:
 Do they sound professional? Do they have
the answers?
 Do they follow us all the way through to
resolution?
 Can they tackle difficult as well as simple
problems?
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Part III: Evaluation Criteria (cont).
• How many calls a day does your NOC
take?
• Do you survey your customers, Cisco-TAC
style?
• The three ‘P’s:
 Perception
 Passion (Emotion)
 Presentation
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Evaluation Criteria
• They (customer) must perceive you as
the people that will help.
• You (NOC) must present your solution
with total assurance that you can handle
the problem.
• You (NOC) must have passion in dealing
with the customer. Make it friendly, even if
it’s a network down emergency. It will
come back to pay off.
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB
Remember.
• Your goal is to be professional, calm, cool
and collected, even in the face of serious
adversity.
• If you don’t have an answer, you know
where to get it and how long it will take.
• You have to communicate to the customer
and keep the customer’s viewpoint in
mind, no matter what.
• As a NOC manager/builder, you have to
make sure your people have the right
NOC Theory and Practice - NANOG 24 - Miami, FL - TCB