Calibration Software
Uncertainty Implementation
Philosophies and Practices
Speaker/Author :
Jorge Martins
NCSLi 2011 Conference
Laboratory to Document and Approve
•
•
•
•
•
•
Calibration procedures
Uncertainty analysis
Uncertainty budgets
Calibration procedures validation plan
Calibration procedures validation results
Traceability chain for each measurement method
NCSLi 2011 Conference
Early Developments of
Calibration Procedure
Software
• Lack of uncertainty
standardization
• Focus on the needs of a
particular organization
• No international
uncertainty standards to
follow
NCSLi 2011 Conference
ISO/IEC 17025 and GUM
1978 ISO
Guide 25
ed.1
1982 ISO
Guide 25
ed.2
1990 ISO
Guide 25
ed.3
1999
ISO/IEC
17025 ed.1
1995 GUM
NCSLi 2011 Conference
2005
ISO/IEC
17025 ed.2
2008
BIPM
GUM
ILAC-P14: 12/2010 Policy for uncertainty in calibration
Requires that to issue an accredited calibration
certificate a laboratory must:
• Estimate expanded uncertainty for each
measurement
• Estimation is compliant with GUM
• Certificate shall report numerical value of
uncertainty for each measurement
• Implicit references to uncertainty are NOT
acceptable
NCSLi 2011 Conference
NCSLi 2011 Conference
Uncertainty Implementation in Software
Group
Measurement
Uncertainty
Type of SW
I
Not supported
• Open source
• Procedure can be modified by the laboratory
II
Budget is supported
• Open source
• Procedure can be modified by the laboratory
III
User defined per point
• Locked by developer
• Procedure cannot be modified by the laboratory
IV
Developer defines budget
• Locked by developer
• Procedure cannot be modified by the laboratory
V
• Locked by developer
Developer defines budget • Procedure cannot be modified by the laboratory
• Unc. Budget ISO 17025 Accredited by 3rd party
NCSLi 2011 Conference
Group
I.
Group
II.
• Most common in-house implementation of calibration
software
• Configure the instruments, sequence the readings
and record the results
• Measurement uncertainty not supported
•
•
•
•
Each uncertainty component can be defined
SW collects Type A at time of test
Effective DF calculated in real time
Expanded measurement uncertainty for each point
(RT)
• Original readings and Uncertainty components
recorded
NCSLi 2011 Conference
Group
III.
• Calibration procedure is pre-defined
• User provides expanded measurement uncertainty
• SW merges expanded uncertainty with
measurement and saves data in DB
Group
IV.
• Calibration procedure is pre-defined
• Developer provides uncertainty budget
• Developer is responsible for validating Calibration
procedure and uncertainty budget
• Data is saved on a DB
• Usually developed by a third party or the OEM
NCSLi 2011 Conference
Group
V.
• Calibration procedure is pre-defined
• Developer provides uncertainty budget
• Developer is responsible for validating calibration
procedure and uncertainty budget
• Data is saved on a DB
• Usually developed by a third party or the OEM
• Measurement uncertainty budget and calculation
engine is accredited by independent SW laboratory
NCSLi 2011 Conference
Document maintenance and validation responsibilities
Uncertainty
analysis (UA)
Group I
External
maintained
Group II
External
maintained
Uncertainty
budget (UB)
External
maintained
Internal
user available
Not
available
Original
readings user
available
User resp.
User resp.
User resp.
Not need SW
framework
validated
User resp.
User resp.
User defined
User defined
Repeatability
(type A)
collection
Uncertainty
analysis
approval
Unc. budget
calculations
validation
Calibration
Procedure
validation
Traceability
Req.
Group III
External
maintained
Group V
Not available to
user
Internal
External
Internal
Not available to
maintained
user available
user
Internal
Original readings
Not available
Not available to
user available
user
Dvlp. Resp.
Developer
User resp.
3rd party IT test
responsibility
accredited
Not need SW
3rd party IT test
User resp.
framework
accredited
validated
Not need SW
Not need SW
Not need SW
framework
framework
framework
validated
validated
validated
Developer
Developer
User defined
defined
defined
NCSLi 2011 Conference
Group IV
External
maintained
Laboratory Calibration Procedure
implementation Efforts
NCSLi 2011 Conference
Major Characteristics
User has complete control of measurement process
Uncertainty validation is independent of measurement
procedure
Merged data (measurement and uncertainty) is available
to issue calibration certificate
Validation of uncertainty engine calculations
Uncertainty budget is not directly synchronized with
measurement procedure
Type A data collection in real time
Noisy units detection in real time
User technically challenging to make changes on the
measurement process
Requires metrologist with SW knowledge
User defined environmental conditions
User defined Uncertainty analysis constraints and
assumptions
NCSLi 2011 Conference
I
II
III
IV
V
√
√
X
X
X
√
X
√
X
X
X
√
√
√
√
X
√
X
√
√
X
√
X
√
√
X
X
√
√
X
X
√1)
√1)
√1)
√1)
√
√
X2)
X2)
X2)
√
√
√
√
X2)
X
X2)
X
X2)
X
√
√
X
X
X
Calibration SW Validation Efforts
NCSLi 2011 Conference
Commercially Available Software
When the software is:
• Commercially available
• Documented
• Locked by the development house
The laboratory procedure validation:
• Trained technicians
• Operational Capabilities
• Resources
NCSLi 2011 Conference
Calibration SW Development Process
Published
Calibration
Procedure
Measurement
Uncertainty
Analysis
Procedure
Revision?
SW Code Implementation
N
SW Test OK?
Y
SW Validation
SW Validation
OK?
Y
SW Release
NCSLi 2011 Conference
N
Calibration SW Development
Deliverables
SW House Calibration Final Report:
• Reference to Documented calibration
procedure implemented
• Validation Report summary
• Uncertainty analysis Summary
Detailed SW specifications
NCSLi 2011 Conference
NCSLi 2011 Conference
Conclusion
• Significant difference in Laboratory:
 Validation efforts
 Calibration procedure control
 Uncertainty analysis control
Group Group Group Group Group
I
II
III
IV
V
Estimation Laboratory
Approval Efforts
100%
NCSLi 2011 Conference
90%
55%
25%
20%
Questions?
NCSLi 2011 Conference