Process Measurement Assurance Program For U.S. State Metrology Laboratories

by Jerry L. Everhart – JTI Systems, Inc. and Georgia L. Harris – NIST Office of Weights and Measures

This paper describes how the Process Measurement Assurance Program (PMAP)2 is used in the U.S. State metrology laboratories to provide uniformity among the laboratories in measurement control, uncertainty evaluation, control charting, and assurance of standards accuracy and traceability. The program provides a method for electronic submission (via e-mail or diskette) of control charts and standards data in a standardized format for NIST, Office of Weights and Measures (OWM) evaluation. The methodology of PMAP and how it applies in a comprehensive quality management program for State metrologists can be applied to other metrology laboratories and systems.
Introduction
The NIST Office of Weights and Measures provides oversight for the U. S. State laboratory infrastructure through a comprehensive State Laboratory Program described as a “measurement assurance program.”3&4 The current system builds on the foundation of measurement assurance concepts and methods developed by J. Cameron in the 1960s and 1970s. J. Taylor and H. Oppermann developed it further in the 1980s. The latest initiative to improve this system incorporates a specific Process Measurement Assurance Program (PMAP) that includes methods and software for determination and control of measurement uncertainty and evaluation of measurement traceability in the calibration processes. Training and software will be supplied to the State laboratories in support of this initiative during 1998. The objective is to provide a comprehensive program that will be used to evaluate, chart, and control the processes in the laboratory. This program will 1) improve the uniformity of measurement assurance methods among State laboratories through the use of a standard software program5, 2) provide a method for electronic submission of data for OWM evaluation, and 3) minimize resources needed for the development of independent software tools on a State by State basis.
Measurement Assurance
Ingredients of the NIST/OWM State Laboratory Program6 include the following:
periodic calibration of laboratory reference standards7 at NIST for traceability;
general and technical criteria for program evaluation following ANSI/NCSL Z 540-1-19948;
quality policy, procedures, and template quality manual9;
documented calibration procedures10>;
laboratory metrology training at Basic, Intermediate, and Advanced levels11 and technical support;
internal and external surveillance testing;
periodic calibration of working standards and check standards in the laboratory;
a process measurement assurance program for control charting and analyzing these activities; and
six regional measurement assurance programs (RMAPs) which provide annual training and round robin measurement coordination for all 50 States, plus the District of Columbia, Puerto Rico, the U.S. Virgin Islands, the U.S. Department of Agriculture, Master Scale Depot, and Los Angeles County.
A number of industry laboratories and other government laboratories participate in various aspects of the program.
The State Laboratory Program incorporates Total Quality Management (TQM) philosophy for selecting the correct quality and measurement assurance tools for analyzing and improving processes within each laboratory as well as in the overall system. It is important to note that measurement assurance concepts are incorporated into all components of the overall program and specifically into all levels of training. This well-organized laboratory program provides a substantial knowledge base of experience and data on equipment and standards, calibration processes, and round robin results that have only recently been available in their fields of measurements. The measurement assurance tools used in the laboratory are like pieces of a puzzle that, when put together, provide a quality calibration program with continuous evaluation and improvement. The pieces of the puzzle include measurement traceability and a process measurement assurance program that evaluates, charts, and controls the processes within each laboratory.
When measurement assurance tools, such as check standards and control charting, were originally proposed and implemented in State laboratories, each laboratory collected and submitted data to NIST for evaluating and charting. State staff were then trained to develop and evaluate their own check standards and measurement processes using control charts. This system has worked well; however, as each State laboratory developed its own system, using various software tools and spreadsheets, relatively little uniformity was observed. The lack of uniformity among the laboratories made sharing resources among laboratories ineffective and made OWM evaluation difficult and time consuming.
The use of standardized PMAP software and methods in this latest initiative does the following: 1) improves uniformity among state laboratories; 2) provides a method for electronic submission of data for OWM evaluation; and 3) minimizes resources needed for the development of independent software tools in each State.
Process Measurement Assurance Program (PMAP)
The Process Measurement Assurance Program (PMAP) consists of methodology and software to determine, control, and improve measuring processes. PMAP uses check (control) standards that represent the calibration output (or product) to evaluate the total measurement process, including the equipment, standards, personnel, procedure, and environment. Measurements made with check standards that duplicate the process are called control measurements. Data from the control measurements are used to develop control charts and calculate measurement uncertainties as calibrations are produced. PMAP methodology calls for modeling the measurement process in such a way that the data from control measurements realistically evaluate the uncertainty, which allows measuring the uncertainty, rather than theorizing about or estimating uncertainties.
The PMAP methodology includes the following components described in this section:
Modeling the process;
Selecting check standards;
Establishing values for check standards with independent reference standards;
Installing at the time of calibration for standards and equipment;
Controlling the accuracy of all standards and the measurement process;
Assisting in determining calibration needs; and
Using statistical tests to make decisions on the process performance and when to take action.
Modeling the process requires knowledge of the measurement instrument and standards to determine which measurements will evaluate maximum process variability and potential bias in the range of use. The objective of each PMAP application is determined before selecting the required check standard or standards. When possible, the objective includes controlling the measurement process uncertainty and controlling the accuracy (established at calibration) of the working standards used. Control measurements are made to duplicate the measurement of the unknown artifacts or product. In a calibration laboratory, the control measurements must represent the type of standards that are submitted for calibration. Control measurements are made continually throughout the year at various times of the day; therefore, they evaluate environmental influences on the measurement uncertainties. Each metrologist who performs calibrations makes control measurements. Data can be used to evaluate how the operator’s skill and performance influence measurement uncertainty.
Check (control) standards are selected to evaluate process variability and systematic errors in the range of use. They are chosen to evaluate the same variables that influence the calibration of a customer’s artifacts. Check standards should be calibrated with a sufficiently small uncertainty to help detect changes in the working standards by comparing each calibrated value with the history of measurements on the check standard. The value of the check standard should be established with a reference standard that is independent of the working standard and the reference standard used to calibrate it in the controlled process. If the check standards and working standards do not have independent calibrations, the program will probably not detect a bias, or error, in the measurement process, although it may be detected during an outside calibration or a round robin exercise.
After modeling the measurement process and selecting the check standard(s), the PMAP software evaluates and controls the measurement uncertainties. Four screens, called pages, are used to set up the software control program. The first page sets up the control file, the recall date for data analysis, archive file, measurement equipment and units of measure. The second page is used to input information about the standards used. This screen includes input for check standard identification number, uncertainties of standards used, calibration and expiration of working or reference standards used, and the established value of the check standard. The third page contains information on limits for the measurement process including coverage factor (k) confidence intervals desired for uncertainty calculation, information for scaling the control chart, reference data and other limits that can either be calculated by the program or entered manually. The fourth page allows input for titles and labels desired on each control chart.
The PMAP program provides assurance that the accuracy of the working standards is maintained when the check standard is inserted into the measurement process immediately after the calibration of all standards. Control measurements are performed and reference limits are established to develop a performance baseline at the time of calibration. At the same time, the value of the check standard should be assigned based on historical data. Future control measurements are then frequently evaluated against the baseline data to detect any changes in process performance or standards accuracy. Baseline data can also be used as a performance reference for new equipment or major service of equipment. Comparison of control data to reference data can indicate when to re-calibrate standards and when to service or calibrate laboratory equipment.

Figure 1. PMAP Chart of Check Standard.
The reference limits of the program (see Fig. 1) are established by duplicating the standard operating procedure in use and by inserting the check standard in place of the product being measured or calibrated. By duplicating the process, the statistical analysis of the data provides the measurement process variability (measured in standard deviations) and the bias of the process when comparing the mean of the data to the established value of the check standard. Each control measurement is evaluated against the check standard value and then tested against the reference control limits for in/out control status. Each control measurement is also graphed for visual examination of the measurement process performance.
Statistical tools are selectively used whenever it is desired to evaluate the measurement process performance. The “F test” evaluates changes in variability (i.e., changes in standard deviations between old and new data), and the “t test” evaluates changes in bias of the standards or process (i.e., compares the mean values of old and new data). These tools (as shown in Fig. 2) allow the user to decide when to re-calibrate standards, when to service equipment, and to determine if measurement uncertainties have changed.

Figure 2. Statistical Analysis of Current Data Verses Reference Data.
Applications of PMAP
The ability to determine, capture, and test historical data against current performance is a major advantage to the metrologist. PMAP has a “History” feature that allows the metrologist to input calibration history data to determine drift rate or wear on standards due to time and or use. Drift rate is calculated using a least squares analysis of “as left” to “as found” values. The history analysis also determines the variability of the calibrations over time using range calculations. Results of the calibration history analysis can determine the optimum calibration interval for standards and equipment in the calibration process. This tool is also used to determine the time interval that PMAP control chart data should be evaluated for detection of long-term changes in variability and bias change.
Control charting and statistical analysis of control measurements can evaluate the primary standards against external standards to determine if the values of primary standards are drifting between calibrations. When PMAP is modeled to evaluate both the calibration process and the calibration standards, the metrologist can often identify problems as they occur and prevent undesirable measurement results. The software can also be used to control chart, document, and evaluate round robin results over time. Data from each round robin test that is conducted with a particular nominal value and uses a specific operating procedure can be tracked on a chart to analyze and document calibration and standard performance over time. State metrology laboratories have extensive experience in conducting round robins and have collected many years of data that evaluates the calibration output of their laboratories. This data provides valuable history and documentation of processes, equipment, and standards when it is evaluated using PMAP methods.
State laboratories also have data and history from current surveillance programs to evaluate standards and process variability using both internal and external standards. This data can now be entered into the PMAP program for visual examination, more complete statistical analysis, and uniform documentation of periodical evaluations.
The application of PMAP software for the State laboratories provides a program to determine and control measurement uncertainties in the calibration processes (see Fig. 3). These laboratories have used control measurements for uncertainty determination and control for many years. By improving data handling methods, States have extensive real time control over their calibration processes and standards. Using these methods and tools is cost effective because calibrations of all balances, standards, and equipment can be obtained based on need rather than by simply requiring periodic full-blown calibrations. By using this information and sharing it with other laboratories in the State Laboratory Program, the processes and equipment can all be improved for performance, accuracy, and cost savings.

Figure 3 . Measurement Performance and Uncertainty Determination
The PMAP program is a systematic approach to measurement assurance with statistical tools, control charting, and analysis such as those used in NIST calibration MAPs12, but it is applied to all levels of calibration, thus facilitating individual laboratory operation and control. The program is used to create control charts and keep note tags with all control measurements. As previously described, the incoming data is evaluated against reference limits that are established when all factors are in calibration. Data is evaluated at will and/or at prescribed intervals using statistical tools to evaluate changes that might occur. These calibration evaluations are documented and stored in software for future reference.
Provides Uniformity
Uniformity is needed in metrology to ensure consistency and accuracy in data analysis, uncertainty analysis, traceability evaluation, and measurement control. Using software spread sheets and individually written software that is different in each laboratory makes oversight, laboratory assessment, and accreditation more difficult and results in higher costs. By using the PMAP application, the measurement process, procedure, standards, instrument used, and performance can be evaluated in established screens. The control chart, performance results, statistical tests, and resulting uncertainties can be obtained by viewing three screens. Process control is visual and built-in statistical controls are indicated on each control chart. With software uniformity, the total measurement control database file can be electronically submitted to OWM for review and assistance via electronic mail. Control charts and uncertainty results can be shared with other metrologists who have similar processes and equipment. Problem solving and process improvements can be rapidly achieved with this electronic exchange of information and knowledge.
Beyond The Laboratory
PMAP methodology can be applied to other weights and measures product processes. Any measurement process must ensure that the quality of the data on which decisions are based is acceptable; the field of weights and measures is no exception. Regulatory actions are often based on measurement data. In weights and measures, acceptable quality for measurements is defined as that which is accurate and traceable. This can impact the system at any of several levels: 1) State laboratories; 2) State inspection officials and registered service companies; and 3) producers and packagers.
Economic factors provide a critical reason for incorporating quality systems and measurement assurance practices into weights and measures. A few cents per single package sold by weight or volume that is incorrectly weighed or measured may not be critical for a family, or even for a producer. However, nearly one-half of the nation’s Gross Domestic Product consists of items that are bought or sold on the basis of measurement13, an amount over $4 trillion. Quality measurements, which can be assured with appropriate quality systems and measurement assurance concepts, limit the potential negative economic impact in the marketplace.
The goal is to use suitable quality systems and measurement assurance techniques to be able to answer the following questions in such a way that will support weights and measures enforcement14 activities:
Are our standards and measurements accurate and traceable at each level?
What are the pieces of data that we have at each level that can be integrated and used to evaluate measurement assurance?
Can we take regulatory action based on these measurements?
Is there uniformity in the system from official to official and jurisdiction to jurisdiction?
Can we rely on measurements made nationally?
Is there equity between the buyer and the seller in each transaction?
Are the measurements correct, or is someone gaining or losing in the transaction?
In 1997, the National Conference on Weights and Measures adopted Good Quantity Control Practices15 that incorporate measurement assurance methodology and practices into packaging guidelines. Implementation of PMAP on the product measurements allows the user or packager to determine measurement uncertainties that affect product quantity. It is desirable and profitable to measure and build the products or fill the packages to stated contents by knowing the real-time measurement uncertainty versus inspecting to reject or accept product after the packaging process is complete.
A system-wide implementation of process measurement assurance concepts at all levels of weights and measures enforcement can help to ensure 1) that measurement data is accurate and traceable for regulatory purposes and 2) that equity between the buyer and seller of measured items is maintained which limits the economic impact on the system.
A 1996 survey16 of the State Laboratory Program workload revealed that nearly 340,000 calibrations are performed each year for 19,400 customers across the United States. Surprisingly, nearly one-half of the calibrations performed by State laboratories were for applications that were not weights and measures related. These included measurement support for manufacturing processes, government requirements, and such quality of life industries as biomedical and pharmaceutical companies. Therefore, the potential impact of this part of the U. S. measurement system and the quality of measurements provided are dramatically multiplied beyond regulatory activities. Monitoring measurements performed within the State laboratory infrastructure provides the impetus for continuous improvement in uniformity and in measurement assurance methods.
Summary
The formal incorporation of PMAP methods and software in the NIST/OWM Measurement Assurance Program for State metrology laboratories during 1998 is the logical next step in an ongoing process of continuing improvement. Benefits resulting from this initiative include the following:
Improved uniformity among the laboratories in the methods being used to chart, evaluate, and control standards and the measurement processes;
A method for electronic submission of measurement assurance data for NIST/OWM evaluation and oversight;
Savings of time and resources for each State and for NIST/OWM through elimination of the need for independent development and quality assurance of spreadsheets and through improved evaluation capability;
Facilitation of sharing information and data among laboratories electronically.
The potential application of measurement assurance concepts to all levels of weights and measures programs could undoubtedly provide the documented assurance of accuracy and traceability needed for future regulatory actions. State laboratory staff can assist regulatory field officials, as well as other laboratory customers, in applying measurement assurance methods to subsequent levels of measurement systems.

As used throughout this paper, the terms “State metrology Laboratories” or “State laboratories” refer only to the State calibration laboratories responsible for providing calibrations for legal metrology or what is more commonly known as weights and measures.
Note: Reference to specific products in this paper does not imply endorsement by the National Institute of Standards and Technology.
Oppermann, H. V., State Regional Measurement Assurance Programs, NCSL Newsletter, pp. 21-26, March 1983.
Belanger, B. C., SP 676-I, Measurement Assurance Programs Part I: General Introduction, pp. 52-55, May 1984.
JTI PMAPTM Software, JTI Systems, 4200 Meadowlark Ln SE, Suite 2, Rio Rancho, NM 87124-1050, 505/896-2500.
Informally known as the State Laboratory Program.
The terms reference standard and working standard as used in this paper are consistent with international definitions and those contained in ANST/NCSL Z 540-1-1994. The use of the term check (control) standard is consistent with the definition in NIST Handbook 143, State Weights and Measures Laboratories, Program Handbook, June 1997: “a standard that is used as part of a process measurement assurance program to provide a ‘check’ on the process and standards to ensure that the standards, measurement results, and measurement processes are within acceptable statistical limits.”
Harris, G. L., editor, NIST Handbook 143, State Weights and Measures Laboratories, Program Handbook, June 1997.
Lee, G. D., NISTIR 5802, Quality Manual Template, Developed for State Legal Metrology Laboratories, March 1996.
Taylor, J. K., H. V. Oppermann, NBS Handbook 145, Handbook for the Quality Assurance of Metrological Measurements, November 1986.
Training is also available to U.S. and foreign industry and government officials. See the NCWM Fax-on-Demand System, document #503, 1-800-925-2453 or the OWM Internet site at http://www.nist.gov/owm.
Croarkin, M. C., SP 676-II, Measurement Assurance Programs Part II: Development and Implementation, April 1985.
Economic Impact of Weights and Measures, NCWM Fax on Demand System, 1-800-925-2453, document #108, 1997 data.
Enforcement activities are not conducted by NIST, but by State and local weights and measures jurisdictions.
Butcher, K. S., Laws and Regulations Committee Report, NCWM Agenda, Appendix B, Good Quantity Control Practices, Appendix C, Point-of-Pack Inspection Guidelines, April 1997.
Eason, L. F., 1996 NCWM State Laboratory Program Workload Survey, unpublished results presented at the 1997 National Conference on Weights and Measures, July 1997.

Selected Bibliography of Reference Papers on Measurement Assurance and Process Measurement Assurance Programs (PMAP) in Chronological Order
1960s
Pontius, P. E., and J. M. Cameron, Realistic Uncertainties and the Mass Measurement Process, NBS Monograph 103, August 1967.
Cameron, J. M., and H. Plumb, Traceability – With Special Reference to Temperature Measurements, Society of Automotive Engineering Transactions, 78, Section 3, 1586-1590, 1969.
1970s
Cameron, J. M., Measurement Assurance, Journal of Quality Technology, Vol. 8, No. 1, 53-55, January 1976.
Cameron, J. M., Measurement Assurance, NBSIR 77-1240, April 1977.
Cameron, J. M. Measurement Assurance for Tolerance Tests, unpublished, estimated late 1970’s.
1980s
Oppermann, H. V., State Regional Measurement Assurance Programs, NCSL Newsletter 23 (1), 21-26, March 1983.
Oppermann, H. V., The Use of Control Charts in State Laboratories, unpublished, March 1984.
Belanger, B. C., SP 676-I, Measurement Assurance Programs Part I: General Introduction, 52-55, May 1984.
Croarkin, M. C., SP 676-II, Measurement Assurance Programs Part II: Development and Implementation, April 1985.
1990s
Everhart, J. L., Determining Mass Measurement Uncertainty, CalLab Magazine, 23-29, May-June 1995.
Pettit, R. B., Process Measurement Assurance Program, Sandia National Laboratories, SAND96-1241, May 1996.
Clark, J. P. and A. H. Shull, Analytical Chemistry Measurement Assurance Programs: More Than Just Measurement Control Programs, Measurement Science Conference 1997, WSRC-MS-96-0405, January 1997.
Ehrlich, C. D., and S. D. Rasberry, Metrological Timelines in Traceability, Measurement Science Conference 1997 and NCWM Annual Proceedings 1997.
Everhart, J. L., Developing a Process Measurement Assurance Program, Cal Lab Magazine, 22-28, January-February 1997.
Butcher, K. S., Laws and Regulations Committee Report, NCWM Agenda, Appendix B, Good Quantity Control Practices, Appendix C, Point-of-Pack Inspection Guidelines, April 1997
To learn more about how JTI can help your business, contact us at 505-710-4999