NOC Theory and Practice

Part II: The Nuts and Bolts

(The Practice Of Making a
Good NOC)
NOC Theory and Practice - NANOG
24 - Miami, FL - TCB

DISCLAIMER
I am not affiliated with any
corporation. My views are my own.

Intro - Whats In A NOC?

Purpose - What Type of Company Are
You?
People will always be your key asset.
Systems make the job of people easier.
Managers assist in identifying gaps in
both.
Processes allow for flexibility in operations
while still maintaining standards.
And, last but not least, the facility is where
you do your work.

The Facility - Think Data

Center
Location, location, location.
Security
Environmental Concerns
This applies to both your inside and
outside environment.

Ergonomics
Reliability
Sources of Labor

The Goals of a NOC

Our conception is:
At the bare minimum, you will receive
requests from your customers
You will process those requests in some
fashion (engineering, handoff, ...)
You will communicate resolution to the
customer, and/or keep customer
updated on status

Goals II
NOC as first, second, or third tier
support.
Conception of NOC as do everything
group
The engineers should be smart at
designing, configuring, etc. - NOC should
be divided into categories of engineers
able to resolve issues
A smart issue routing process makes this a
very effective model

The Process
Receive events from your inputs
(phone calls, network monitoring, etc)
Process inputs into a problem to be
resolved
Route the problem to the correct
individual
Resolve the problem as appropriate
Communicate status continually.
Record the problem for later review and
training.

The Key to Success Is

By constant re-examination of
your operation, your employees,
and what inputs you are getting,
you can reach incredible
efficiency. Communicate this to
other teams and your operations
will prosper.

The Facility (cont.)

Security applies to keeping other people
out, but it also applies to making a safe
arrival for your employees, at any hour.
Environmental concerns: Just like a
datacenter, consider climate control and
earthquake/act of god resistance.
Disaster planning rules should be in
effect.
Sources of labor: Not just engineers, but
builders.

A Word on Portability
As Matt suggests, portability is critical.
Obvious resources: Computers,
communications, phones.
Not obvious resources: Vehicles and quick
transport/assembly, good cases for shipping
or relocation, breakdown/reassembly
schedules.
Multiple everything.
Make sure your resources are distributed. If
your network is large, your footprint of
mobility should be large.

Hiring the Right People

Skillset
but more importantly, flexibility

Stress
Scheduling
Stability
NOCs are hard things to staff appropriately.
And good people are hard to retain.

Hiring (cont.)
Make sure they have lives. But know what
you cant ask.
Can they talk? How are their soft skills?
Do they get flustered easily?
And remember, org charts suck. Classify
people according to their skills, not
according to some hierarchy which has no
meaning outside of some middlemanagement mind.

Keeping Your People Happy

Environment
Ergonomics
Creature Comforts
Were not kidding.

Give some thought to a 24x7 outlook.

This means beds, showers, food, and
climate.

Environment - People
Climate controlled centers get cold at
night. Make sure you have 24H
control over your space.
When thinking about climate, keep in
mind your working conditions are
slightly different in a NOC.
2:00AM to 4:00AM are the critical
hours.

Ergonomics - People
Hire a good ergonomic consultant.
Positions of lights, phones, keyboards,
monitors, pens - all this matters.
Observation is key. Your engineers will
show you what the problems are.
(Cameras)
Seating. If youre in a chair for eight
hours at a stretch, dont you want it to
be a comfortable one?
Screen space.

About People. About NOCs.

Its a messy business. Its high stress. Its
easy to let things get out of hand.
Good management qualities in a NOC:
cool heads
methodical approach to solving
problems
having been on the customer side
micromanagement sucks.

Lets go over retention.

Retention of NOC staff is 100% harder. Very
few people live for this kind of job.
So, innovate. Think outside the box.
People want to grow. Know what your people
want to do when they grow up.
Remember your core focus. Do operations, not
sales, or account services, or anything else.
At the same time, make sure your folks are
informed on everything. Everything comes out
in the wash. All the dirty clothes come to
NOC.

Interviewing NOC folks

You are interested in three things.
The candidates willingness to learn.
The candidates 1-2 year outlook on their career.
The candidates ability to work funny schedules.

They are interested in three things.

Will you offer them growth potential?
Can you give them enough perks to enjoy their work?
Do you have your stuff together?
Are the procedures well-documented?
Will they be called in outside their shift?
Will they ever wonder who to go to?

Productivity
Momentum is key. Keeping your people busy is
the most important thing towards maintaining
productivity.
The other component to momentum is keeping
them working on different things. This makes
them better engineers, and helps feed ego.
Everything at your fingertips - your engineers
should never have to make a customer wait
more than four minutes on hold for anything. If
it has to wait longer than that, tell the customer
why, and mark it as something to fix.

The View of the Customer

A customer wants
Professionalism: your engineers must treat the
customer as if they are paying you a lot of
money and the engineer wants the customer
to continue to do that.
ETR: Your customer wants to know when the
problem will be fixed. I dont know is not an
acceptable answer. The ETR is the first thing
you give to a customer and the one thing you
update him on at least every hour (or less if
they prefer).

The View of the Customer II

A description: Concise or not, the customer
wants to know what the problem appears to be.
Remember your focus. It is not (typically) your
responsibility to engineer your customers
network for maximum reliability. Your
responsibility is the operation of that network.
The only thing you should do as a NOC is fix
problems when they arise in the most expedient
and professional manner possible. A customer
wants to be comfortable, and assured that that
will happen.

Things we never, EVER allow a

customer to hear
I dont know when it will be fixed.
I cant help you.
Im not qualified to answer that question.
Let me transfer you to someone else.

Its not always ideal. But Where possible, the first person to take the
call should be the person who follows it to
completion. The customer hates being
thrown between different engineers - but
not having an answer is worse.

A Little About Systems

Dont just focus on ticketing.
Make sure your people have
information on everything they need
to operate.
If all the architects fell off the planet
today, could you rebuild?
Its about agility.

Policies and Procedures

If you dont write it down, it isnt a
procedure or a policy. There are no
unwritten rules in a NOC. People cycle.
Focus on procedures that affect the
handling of a customer. The soft skills
are usually the hardest to learn.
Full disclosure?
Troubleshooting Techniques

And a word on incidents.

Incidents, tickets, call it what you will.
Whatever happens in the NOC:
gets summarized to the account exec if
the issue is of appropriate severity
gets recorded - nothing EVER gets
deleted from a customer log
gets archived
gets resolved - how much time will you
spend on issues that never get fixed?

Some final notes - Conclusion

Its easy to throw a bunch of people in a room.
Its hard to build an efficient, productive, and
well-oiled NOC (the same goes for networks)
Record everything. Write everything down.
Try to remain as flexible as possible.
Look to your employees for cues on what their
ideal environment should be.
Remember your focus, and know your customer.
Know what not to say.

Credits
As far as the practice of NOC building
goes, we must thank:
herb@tomobiki.urusei.net - thanks
for great private discussions on NOC
building
nanog subscribers - for talking about
operational stuff

And finally
Sean Donelan - for giving network
operations folks an impetus for
existing (due to his backhoe skills)
Shouts to EFNet IRC #nanog: you
know who you are

Part III: Evaluation Criteria

Social engineer your own NOC.
We like to rate NOC folks on:
Do they sound professional? Do they
have the answers?
Do they follow us all the way through to
resolution?
Can they tackle difficult as well as
simple problems?

Part III: Evaluation Criteria

(cont).
How many calls a day does your NOC
take?
Do you survey your customers, CiscoTAC style?
The three Ps:
Perception
Passion (Emotion)
Presentation

Evaluation Criteria
They (customer) must perceive you
as the people that will help.
You (NOC) must present your solution
with total assurance that you can
handle the problem.
You (NOC) must have passion in
dealing with the customer. Make it
friendly, even if its a network down
emergency. It will come back to pay
off.

Remember.
Your goal is to be professional, calm, cool
and collected, even in the face of serious
adversity.
If you dont have an answer, you know
where to get it and how long it will take.
You have to communicate to the customer
and keep the customers viewpoint in mind,
no matter what.
As a NOC manager/builder, you have to
make sure your people have the right tools.