September 15th, 2011
To: Scott Newbolds, Director of Site Operations, NEEScomm
Fr: Ross Boulanger, Director, and Dan Wilson, Assoc. Director, NEES@UC Davis
Re: Incident report from August 24th, 2011 – Summary of Mechanical Evaluation and Procedures
On Wednesday, August 24th, 2011, the large centrifuge at UC Davis struck a steel work table during a
centrifuge experiment. The centrifuge was moving at a low speed and the physical damage to the
machine appears to be limited. This memo provides a summary of the mechanical evaluation of the
centrifuge machine following the incident. This is the second of three expected memos regarding this
incident:
Memo 1: Preliminary incident report from August 24th, 2011 (issued 8/30/2011)
Memo 2: Summary of Mechanical Evaluation and Procedures (this memo)
Memo 3: Summary of Incident Impact on NEES MO&M Scope of Work and Budget (pending)
Summary of Mechanical Evaluation
On Wednesday, August 24th, the Center for Geotechnical Modeling was set to perform a centrifuge test
in support of Muralee's NEESR piles project. During the initial spin up of the machine, the centrifuge
bucket impacted a steel work table at about 6.5 rpm (less than 0.5 g centrifugal acceleration). It appears
the centrifuge dragged the table for about 15 seconds, or about one full revolution. The table was not
noticed in the pre-flight checklist inspection by the responsible technician as it should have been.
The experiment was declared a loss due to the loads and strains in the model induced by sudden bucket
movement.
Following the incident, CGM staff analyzed the data collected by health monitoring systems and
examined the physical evidence. The data indicates three distinct loading conditions occurred: 1) the
bucket impacted the table; 2) the arm dragged the table around the room; and 3) the table wedged
between the bucket and the main door to the centrifuge. Each loading condition was analyzed for its
potential impact on the structural integrity of the centrifuge, and the results are discussed below.
The incident of 8/24/2011 followed closely the hydraulics failure on 6/1/2011, where a research model
was lost due to hydraulic oil spilling into the model container. In that incident, a temporarily routed
hydraulic hose caught on a fixed part of the machine during spin up. Neither of these was caught during
pre-flight inspections as we would expect. We have thoroughly evaluated the circumstances leading to
the latest incident and have reviewed similar operating procedures at other NEES equipment sites. We
are initiating corrective procedural and administrative measures to reduce the likelihood of similar
future incidents. These changes are discussed below.
Finally, the research schedule at NEES @ UC Davis has been impacted, at least in the short term, by the
incidents this summer. Two project teams have had direct impacts on their work, and other teams have
had to alter their schedules according to equipment availability. The shift in focus of our staff effort and
Memo: Mechanical Evaluation and Procedures
Page 1 of 8
budget allocation has also impacted our planned MO&M activities for the fiscal year. These impacts will
be discussed in a subsequent memo.
Evaluation of equipment
The physical damage to the centrifuge facility appears to be limited primarily to electrical connectors
and shaker hydraulic lines. Our staff members are already working on these repairs.
Following the incident we performed detailed physical inspections of the machine as well as analyzed
our health monitoring data. Where possible we have tried to quantify the loads imposed on the arm and
compared them to design loads and normal operating conditions. Note that the machine is designed for
extreme loading conditions, operating at up to 75 g of centrifugal acceleration and supporting the highpower servo-hydraulic shaking table. Detailed physical inspections were performed to find any evidence
of impact loads that could lead to stress risers in highly-loaded areas of the machine. We also looked for
any evidence that flying debris may have caused damage that is less obvious.
Based on our review, we believe that the arm impacted the table while the arm was accelerating. Power
was immediately cut off to the centrifuge but the arm continued to spin under its own momentum. The
arm caught the table and dragged it around the room. At one point, the table became momentarily
wedged between the arm and the wall, significantly crushing the table. Finally, there was sufficient
clearance between the arm and the wall such that the arm could swing free of the table. Each loading
condition is discussed below.
Initial impact with table:
The centrifuge was under full power acceleration at the time of impact. The arm was moving at about
6.4 RPM, a little less than 14 mph at the bucket. The initial impact caused a measurable oscillation in the
drive motor speed, as demonstrated in Figure 1.
The drive train includes shear pins to limit the torque on the arm. Per the original NGC stress analysis
summary, the pins will shear at a torque of approximately 182,000 ft-lb. At a 25 foot radius, that
translates to 7300 lb of static force on the bucket.
The impact happened under maximum motor torque and inspection of the shear pins following the
incident showed they were undamaged. A simple
RPM at impact
static approximation then limits the expected
maximum tangential force to less than 7300 lbs. This
simplification neglects the dynamic response of the
arm, but nonetheless it gives us a reasonable
indication of the maximum potential load.
RPM at motor
100
80
60
40
20
0
0.5
1
1.5
2
Time (sec)
Figure 1: Variation in drive motor speed due
to initial impact with table
Memo: Mechanical Evaluation and Procedures
Per the NGC stress analysis summary, the allowable
stress in the bucket wall is 40 ksi. We estimated the
bending stress in the bucket wall due to a 7300 lb
force tangential to the direction of rotation to be
Page 2 of 8
5.1 ksi, a little more than 10% of the allowable. We thus conclude that the stresses induced by the force
of impact were less than allowable for the machine.
Table dragged by arm:
Starting about 25 seconds after the impact there was a period of about 17 seconds where the centrifuge
speed dropped rapidly, followed by a slow, steady decrease in speed. The rapid decrease in speed
indicates additional drag on the arm as the arm was dragging the table around the room.
RPM versus time
Arm speed (RPM)
8
6
4
2
0
0
100
200
300
Tim (seconds)
Typical Slowdown
8.24.2011
Figure 2: Variation in arm speed on 8/24 versus
prior slow downs
The maximum deceleration of the centrifuge
was about 4 seconds per RPM (i.e. 0.5 RPM
decrease in 2 seconds). Looking at prior spindowns, the centrifuge typically takes over 5
minutes to slow from 5 RPM to 2 RPM or about
110 seconds per RPM. We estimated the drag
force required to slow the arm as observed. The
calculation, while rough, indicated a force of less
than 2000 lbs on the bucket. This is significantly
smaller than the force limit used to calculate
stresses due to impact above and is therefore
less than allowable on the arm.
Finally, while the drag force had a measurable impact on the centrifuge speed, the change was less than
typically observed during full throttle acceleration. During rapid acceleration the centrifuge will
accelerate from 0 to 35 RPM in two minutes, or about 3 seconds per RPM.
Table wedged:
After the centrifuge arm dragged the table around the room for several seconds there was a notable
event where bearing vibrations were greatest – see Figure below. Presumably at this point the table
caught the corner of the man door in the rotunda and rotated until it became wedged momentarily
between the bucket and the wall. This likely produced axial, vertical, and tangential forces on the bucket
while crushing the table. The magnitudes of these forces are difficult to estimate, but we can compare
the observed data to what we typically record during shaking at high g-levels. In particular, we tried to
estimate the potential effects of each load of concern to what was observed.
Bearing loads:
The momentary load could potentially have damaged the main centrifuge support bearings, which
would shorten the bearings’ operating lives. However, even if the bearings were damaged, we do not
believe there are any consequences to axial load or vertical load that would cause the machine to be
unsafe to operate. The consequence of bearing damage would be a change in the vibration signature
recorded in the health monitoring data. We monitor radial, axial, and vertical vibrations on the upper
and lower main support bearings of the centrifuge. These bearings support the weight of the centrifuge
and carry any dynamic imbalance force from the machine into the foundation structure. When the table
became wedged we recorded significant vibrations on the bearings in all three directions. These
Memo: Mechanical Evaluation and Procedures
Page 3 of 8
Figure 3: Bearing vibrations measured by health monitoring system (units of g on vertical axis)
Red = example of prior large shake
Black = recorded on 8/24
vibration levels indicate that the loads on the main support bearing were significant compared to our
normal shaking-induced loads. However, it is not possible to accurately estimate the loads based on the
recorded vibrations.
Our best indicator of potential bearing damage will come from comparing future vibration data to our
past data. We will operate the centrifuge up to 10 g to record the current state of bearing vibration.
Vold Systems will then compare the new data to our existing health monitoring data to see if there is
any indication of bearing damage due to the incident. Any long term consequences will be discussed at
that time.
Hydraulic bolt loads:
There was an apparent axial component of load based on measured movement of the counterweight
displacement transducers. The axial load was transmitted down the arm like a hammer blow driving a
pile, and the tip (counterweight end) moved out. After moving out quickly, it slowly retracted, and the
amount it came back was restricted by friction in the pucks. The counterweight displacement
transducers indicate that the counterweight rapidly moved about 0.25 mm outward before returning
slowly to about its original position. Note that typically the counterweights move more than 2 mm at
75 g.
Memo: Mechanical Evaluation and Procedures
Page 4 of 8
Our concern with axial load is related to the potential for the tension straps to slip at the hydraulic bolt –
the point at which they are connected to the main beam. The hydraulic bolt is designed to withstand
500,000 lbs of axial force, so we do not expect the bolt to have slipped. Nevertheless, we did closely
inspect the arm for visual signs of potential slippage and we found none.
Bucket wall bending:
The bucket wall bending gauges showed a rapid transition of moment at this event. The maximum
strains recorded were 150  absolute and a delta of 80  before and after the event. These are small
relative to the bending strains typically observed in centrifuge operations. We have recorded over
500  of strain at 75 g, and it is not uncommon to see 80  of dynamic bending strains during large
shaking events. Thus we conclude that while measurable, the bending strains induced in the bucket wall
were no larger than typical operating strains.
Other inspections:
Potential for bending of the bucket clevis retainer pin due to tangential forces:
It was thought possible that there could have been some prying action on the bucket pins due to the
tangential load on the centrifuge bucket. That is, the bending moment induced on the bucket wall is
transmitted across the pivot point through the clevis retainer pin and bushing. We removed the pin to
inspect for signs of excessive load. Nothing was apparent - no shiny marks, metal slivers, witness marks
in grease or dirt, etc. The gap visually noted between the inner (black) plate and the inner clevis was due
to the big (13" ID) bushing sitting about 1/8" proud of the inner clevis, as it always has. The dirt on the
OD of the bushing makes it obvious that it's been that way for a long time. The same was evident on the
outside; the grease behind the outer plate looks undisturbed.
Potential for stress risers due to impact load damage:
We examined the west bucket wall for
impact damage with a flashlight and a 10X
eye loupe. Numerous paint chips that
were initially assumed to be the result of
the table impacting the bucket appear on
close inspection to predate the impact.
These marks were heavily stained with old
grease and dirt, including the lengthy
scrapes along the bottom edge of the
bucket. The photo of the bucket wall in
Figure 4 shows the dings we believe to be
the result of the incident circled in green
and numbered.
Of the circled dings, number 1 is the most
prominent, and the one we initially
expected to be the most serious. Under
Memo: Mechanical Evaluation and Procedures
Figure 4: areas where paint is removed from the
bucket, indicating possible points of impact loading.
Page 5 of 8
magnification it's apparent that there is no dent in the metal. In fact there is still a thin, but smeared,
layer of red primer present, as if the primer acted as a sort of "grease" when the table hit it. The bare
metal can be seen, but the surface is unbroken mill scale. No shiny metal or denting is visible.
Number 2 is the only ding where there is clearly bright metal. The metal is scraped on the square edge,
with vertical score marks as if something slid up or down the edge, for a distance of roughly one inch.
However the marks are shallow enough that no burr or sharp edge is apparent when a finger is slid over
it. A couple of strokes with a mill file will eliminate the scrape marks completely.
In all other cases, paint was removed, but not to clean, bare metal. Some primer remained in every case.
Our mechanical engineer rated all of these spots as insignificant based on his experience working with
steel. We will visually monitor these points along as part of our annual maintenance inspections.
Ancillary equipment damage:
Research data acquisition system sensor interface: many connectors and interface boxes were damaged
and must be replaced. The work requires many man-hours of effort by our electronics staff.
Shaker Hydraulics: A supply hose on the shaker was damaged and hydraulic fluid leaked out of the
system. The hose is easily replaced. Filling the shaker with oil requires following a careful process, and
the shaker’s performance will have to be verified before we begin research again.
The centrifuge DAQ and control systems appear to be undamaged.
Building Damage:
The table caught on two anchor bolts embedded in the centrifuge rotunda wall. One anchor bolt was
bent and it can be cut off. One anchor was pulled from the wall, requiring some patching of the
concrete. These repairs can be made by our
staff with minimal expense.
The large access door was loaded when the
table caught the man door section while
being dragged around the room by the arm.
After the incident the large door would not
close properly and the corner sagged
enough to drag the floor while it was
opened. The hinges on the large door were
visibly bent by the impact.
We were concerned because if the concrete
wall were to be broken behind the jamb
then the wall would have lost significant
strength. Upon close inspection, it appears
this did not happen. The steel jamb appears
Memo: Mechanical Evaluation and Procedures
Figure 5: These electrical interface panels are used to
connect sensors to the data acquisition system. The
hydraulic lines are part of the servo-hydraulic shaking
table.
Page 6 of 8
to have flexed slightly when it was loaded, causing cosmetic damage as it cracked the epoxy paint at the
seam between the jamb and wall. There does not appear to be any structural damage to the wall or
jamb.
The small man door will have to be replaced and the large swinging door will have to be removed, new
hinges welded into place, and the locking pins will have to be reset. This work will have to be performed
by an outside contractor.
Evaluation of the circumstances leading to the incident and corrective
procedural and administrative measures
Centrifuge operators follow an established checklist each time the centrifuge is operated. The checklist
can be reviewed at
http://binders.cgm.ucdavis.edu/forms/operations/UC%20Davis%20NEES%20Geotechnical%20Centrifug
e%20Operators%20pre-spin%20Checklist.docx.
According to the checklist, an operator is to walk the centrifuge rotunda to inspect for loose items. They
are then to walk the top of the centrifuge arm to inspect for loose or inadequately restrained items. On
8/24/2011, the technician did not walk the centrifuge rotunda prior to the spin. Instead, he inspected
the rotunda from on top of the arm. This was clearly a mistake and did not satisfy the checklist.
The top of the arm provides a view of most of the room, except directly beneath the arm – see Figure 6
below. Unfortunately, in this case the arm was positioned directly over the steel table and so the table
could not be seen from on top of the arm. When the technician came down off the top of the arm and
exited the room the table was blocked from view by steel decking.
The technician executing the checklist on 08/24 was our most experienced operator. He has worked at
Figure 6: At left, the arm blocks the view of the area directly beneath it from anyone standing on the
arm. At right, the table under the arm, as it was positioned before the spin on 8/24.
Memo: Mechanical Evaluation and Procedures
Page 7 of 8
the CGM for more than 10 years and has operated the centrifuge over 100 times, far more than any
other operator in our history. He recognizes more than anyone that he made a mistake in deviating from
the full checklist.
Our staff reviewed operating checklists from NEES@RPI, NEES@UCSD, and NEES@Buffalo for best
practices that might be missing from our procedures. We then held an all-staff incident debriefing to
discuss changes to our operating practices. The debriefing resulted in multiple suggestions for significant
changes to our operating practices that we will implement immediately:
1) We will clarify our operating practices and expectations for researchers and staff to reduce
pressure on staff to work hurriedly on spin days. For example, we will inform researchers that
our practice is to not begin the pre-spin check list after 2:30pm. We will also inform researchers
that our normal practice is to schedule one spin per day. Clearly articulating these expectations
will reduce pressure on our staff to rush getting a model ready to spin when they should be
focused on preparing to safely operate the centrifuge. The information will be incorporated into
our ESUF and/or our facility orientation package as appropriate.
2) We will explicitly modify our pre-flight checklist practices. For example, we will require that the
check list not begin until the researcher and all other personnel have exited the rotunda for the
last time. The checklist must be completed in order, and each step must be executed even if the
step was already done (e.g., close the valve even if you know you previously closed it). If the
checklist is interrupted (e.g., the researcher needs to troubleshoot a sensor) then the checklist
must be started again from the beginning. Combined with the 2:30 time requirement, this will
greatly reduce the sometimes frenetic back-and-forth that happens when a researcher thinks
they’re ready to start but repeatedly needs to check one more thing on the model.
3) We will institute a redundant checklist process whereby a second person will perform a check of
key steps in the checklist. Now two people will sign off on key safety checks like walking the arm
and rotunda inspecting for loose objects or improperly-routed cables.
4) We will install clearly marked and easily accessible emergency shut off switches for the
centrifuge drive motor. The drive motor already includes emergency shut off switches in the
rotunda, but they are not clearly marked or easily accessible.
This incident was the first time in 25 years of operations that the centrifuge was operated with an
object left in the room. The procedural changes we are adopting and the increased awareness and
vigilance within our staff will greatly reduce the risk of a similar incident ever happening again. The
changes will also reduce overall risks from operational mistakes associated with operating the
equipment and model testing.
Memo: Mechanical Evaluation and Procedures
Page 8 of 8