Overcoming the Challenges of Metrics Implementation History of Metrics

[Editor's Note: This article is the first in a six-part series from the PDA Quality Management Maturity Team, covering key points for designing and implementing a robust quality metrics program.]

The Industry has traditionally taken a compliance-review or management-review focus on metrics.

The emphasis has been put on meeting the minimal requirements set out by regulatory authorities and industry standards without pushing toward a holistic and comprehensive view into operational capabilities. Metrics alone without an underlying culture of quality have been manipulated to support a desired message to business leaders or regulators. Research by PDA, confirmed by the University of St. Gallen demonstrated that having appropriate quality systems and processes in place enhances the positive behaviors and beliefs needed to support a strong quality culture. Selection and implementation of appropriate metrics and advanced data analytics is one important quality system component that will drive the continual improvement mindset and help set a strong quality management maturity culture With the move towards advanced data analytics and artificial intelligence, it is even more urgent that an appropriate foundation and culture be in place to ensure the use of metrics is grounded a commitment to patient-focused quality.

Steady progress has been made in the understanding and application of metrics over the past century. This understanding has centered around three main pillars: variability, waste, and continual improvement. Each advancement in the understanding of metrics has helped improve the understanding of each of these pillars. To identify where industry needs to go, it is helpful to recognize the progress that has been made in understanding the metrics in support of continual improvement and draw upon the learnings of other industries.

The first discovery commonly acknowledged as paving the way for understanding the variability of manufacturing data occurred in the 1920s at Western Electric Company and Bell Telephone Research Laboratories. During these years, Walter Shewhart first described control charts, which introduced the theory of statistical process control. This was a breakthrough method to understand the variation that existed in any process. This variation was divided into common-cause variation, a natural variation that exists in a process, and special-cause variation, that is, one that is not natural to a process and indicates the potential for an issue (Figure 1) (1).

While the contribution to statistical theory was remarkable, so was the benefit to individuals who were less savvy about the use of statistics. The ability to graph relatively simple-to-understand charts, such as the p-chart, allows any employee to notice visual shifts in a process and act to correct it. This innovation provided an early indicator of whether deeper investigation into process changes was required.

The next contribution to the understanding of metrics and introduction of the concept of continual improvement in manufacturing grew out of Shewhart's work at Bell Laboratories. The Shewhart cycle, more commonly referred to as the Deming or Plan-Do-Check-Act (PDCA) cycle, took the control-chart concept and embedded it into an overall improvement process. This process was pioneered by Shewhart and popularized by W. Edwards Deming during his work in Japan (1).

At about the same time, a separate pathway of innovation started to develop. The concept of Lean manufacturing originated in the Toyota Production System, developed by Taiichi Ohno, starting in the 1950s with the primary objective of eliminating "muda" (waste). The goal of this methodology was to sustainably deliver value by identifying and reducing waste as a means of continual improvement (2).

In the 1980s, the understanding of continual improvement and variability progressed again at Motorola. During this time, Motorola linked the Deming cycle with the Taguchi loss function to focus on creating a process so robust that it produced virtually no defects. To reach this robust process, known as Six Sigma, the methodology used statistical process-control concepts in measuring the standard deviations (sigmas) between the mean of the process and the specification limits (3).

Over time, practitioners have developed a process improvement methodology to complement this statistical theory. The Define, Measure, Analyze, Improve, and Control methodology and the Define, Measure, Analyze, Design, and Verify methodology are evolutions of the Deming PDCA cycle in the areas of process improvement and product development, respectively. These new methodologies share a goal of reducing defects and minimizing variability by progressing to higher sigma levels.

As the Lean and Six Sigma concepts began to permeate manufacturing, companies started to use both in the same manufacturing areas, which became known as Lean Six Sigma. One reason behind this combination is that a foundation of Lean in an organization could make employees and management more open to adopting Six Sigma concepts. When the two practices are successfully established in an organization, "Lean establishes the standard. Six Sigma removes the deviations from that standard" (4).

The final major innovation in the modern understanding of metrics and continual improvement is the concept of Operational Excellence (OPEX), which was developed through the contribution of many different individuals. This approach has added to the concepts of Lean and other process-improvement philosophies. One key innovation of OPEX is the desire to move an organization from thinking about quality as a product attribute (little "q") to quality as a facilitator of a positive customer experience (big "Q"). This theory has pushed process owners to think about the customer when considering process-improvement opportunities and the business benefits that are obtained by this mindset.

The U.S. Food and Drug Administration (FDA) has also contributed to this evolution of understanding metrics. Initially, the Agency portrayed metrics as a means of ensuring compliance and reducing the potential for drug shortages through draft guidance documents and other Agency publications (5-6).

Their position eventually evolved to a continual improvement approach, as part of a pharmaceutical quality system, with the introduction of quality management maturity (7-8). The FDA then showed that the two approaches could coexist, with metrics supporting a continual improvement framework as opposed to being used strictly as a measure of compliance (9-10).

Additionally, the discussion of the use of continuous vs. continual improvement is ongoing. The FDA weighed in with their definitions of the two terms in the guidance MDSAP QMS P0013: Continual Improvement Procedure, definitions we have adopted in this paper (11):

• Continual Improvement (CI) is defined as a "recurring activity to enhance performance." This concept involves systematic, step-by-step enhancements over time, allowing for periodic assessments and adjustments.
• Continuous Improvement is described as "more of a philosophy than a process or system." This approach emphasizes ongoing, uninterrupted efforts to enhance processes, often integrating real-time monitoring and immediate corrective actions.

Finally, as evidenced by the FDA's continuing interest in quality-metrics reporting programs and its quality-management-maturity initiative, the Agency has demonstrated interest in how data can predict performance and support a continuous improvement mindset. Companies would benefit from developing internal expertise in this area to lessen the possibility of a situation in which the FDA finds trends in the data that are not identified (or cannot be reproduced) by a company. A company's data experts can also assist in the transition from metrics being used in a compliance mindset to metrics being used in a proactive, continual improvement mindset. This includes assessing the true need for given metrics, assessing potential unintended consequences, and ensuring that the data governance of these metrics aligns with the firm's quality culture to yield a strong data culture (Figure 2) (12). A strong data culture can transcend departmental boundaries to increase the discovery, use, and trust of information, resulting in better decision-making. Ultimately, these data experts would be well-positioned to choose the most appropriate metrics for entering into any metric-reporting program that allows self-selected metrics such as those presented in FDA's proposal.

Many organizations have researched methods of determining an organization's proficiency with continual improvement, such as Lean, Six Sigma, and OPEX. Research has indicated that quality culture is an enabler of high performance and competitive advantage. To support the determination of an organization's continual improvement proficiency, the Parenteral Drug Association (PDA) has worked to understand the linkages between an organization's quality culture and its quality maturity. Partnering with the University of St. Gallen, PDA has co-published literature that confirms a positive correlation between quality culture and performance (13). Independently, both PDA and St. Gallen performed research to develop lists of the top elements of quality culture and found a strong overlap between those identified in PDA's Quality Culture Assessment Tool and St. Gallen's Top Ten Quality Maturity Attributes as shown in Table 1 (14). This further supports the concept that a strong quality culture is vital for quality maturity and is a part of the overall continual-improvement pillar of metrics understanding.

Conclusion

From the early breakthroughs of statistical process control to the integration of Lean, Six Sigma, and Operational Excellence, the evolution of metrics reflects a shift from simple measurement toward enabling continual improvement and organizational maturity. What began as a way to understand variability has become a strategic tool for reducing waste, strengthening quality culture, and improving performance.

Regulators have mirrored this transition, moving from a compliance-focused view of metrics to one that supports quality management maturity and proactive oversight. This shift makes clear that metrics alone are not enough—they must be grounded in strong data governance and a robust quality culture.

As industry embraces advanced analytics and AI, organizations must ensure their metrics programs are designed not just to report performance, but to drive insight, resilience, and patient-focused excellence.

References

1. Best, M; Neuhauser, D. Walter A Shewhart, 1924, and the Hawthorne Factory. BMJ Qual Saf 2006, 15 (2), 142-43.
2. Womack, J P; Jones, D T, et al. The Machine That Changed the World: The Story of Lean Production. Free Press: Cheney, WA, 1990.
3. Devor, R; Chang, T, et al. Statistical Quality Design and Control: Contemporary Concepts and Methods, 2nd Edition. Pearson: New York, 2006.
4. Wheat, B; Mills, C, et al. Leaning into Six Sigma: A Parable of the Journey to Six Sigma and a Lean Enterprise. McGraw-Hill Professional: New York, 2003.
5. U.S. Food and Drug Administration. Guidance for Industry: Submission of Quality Metrics Data, Rev. 1; U.S. Department of Health and Human Services: Silver Spring, Md., 2016.
6. U.S. Food and Drug Administration. Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments. Fed Reg 2013, 78 (29) (Feb 12 2013), 9928-29.
7. The White House. Building Resilient Supply Chains, Revitalizing American Manufacturing, and Fostering Broad-Based Growth; The White House: Washington, DC, 2021.
8. U.S. Food and Drug Administration. Quality Management Maturity: Essential for Stable U.S. Supply Chains of Quality Pharmaceuticals U.S. FDA: Silver Spring, Md., 2022.
9. U.S. Food and Drug Administration. Drug Shortages: Root Causes and Potential Solutions; U.S. FDA: Silver Spring, Md., 2020.
10. U.S. Food and Drug Administration. Food and Drug Administration Quality Metrics Reporting Program; Establishment of a Public Docket; Request for Comments [Docket No. FDA–2022–N–0075]. Federal Register 2022, Vol. 87 (No. 46), 13295-99.
11. Medical Device Single Audit Program. MDSAP QMS P0013: Continual Improvement Procedure, US FDA., January 18, 2024
12. K. Data Governance + Quality Culture: The Whole is Greater than the Sum of its Parts, Presented at 2022 PDA Data Integrity Workshop, Washington, DC, Sept 15-16; 2022.
13. Friedli, T; Buess, P, et al. The Impact of Quality Culture on Operational Performance—An Empirical Study from the Pharmaceutical Industry. PDA J Pharm Sci Tech 2018, 72 (5), 531-42.
14. Baker, D; Broadfoot, J, et al. How to Measure Quality Management Maturity. PDA Letter 2021.