Best Practice Guide for Using KPI’s/Metrics

The PDA Quality Systems Interest Group (QSIG) provides a forum for industry experts to discuss “hot” topics, which usually relate to rapidly evolving interpretations of current regulations. One such topic is the use of Key Performance Indicators (KPIs) and metrics.

Because health authorities consider KPIs and metrics important indicators for judging the maturity of a company’s quality system (1,2), the QSIG is frequently asked, “Which is the best practice for using KPIs and metrics within a quality system?” That this topic receives so much attention is not surprising, as KPIs and metrics are a good way of judging the state of maturity of selected elements of a quality system or of the entire system. What is surprising, however, is that there is currently little standardization in the industry for using KPIs and metrics.

This background is what triggered the work of a subgroup within QSIG that has been working on developing a Best Practice Guide for Using KPIs and Metrics. They recently presented on the status of their work in the monthly video meeting of QSIG.

The subgroup presented the following list of content to be covered in the Best Practice Guide:

• Purpose of KPIs/metrics
• Essential/desirable features of KPIs
• Areas of application
• Categorization
• Example definitions
• Visualization
• Points to Consider

The “Points to Consider” section will serve as a resource for various aspects of the use of KPIs and metrics that reflect the current thinking on best practices.

• Metrics influencing behavior
• KPIs/metrics categorization
• CMO integration into system of metrics, achieving alignment and understanding
• Compliance vs. quality metrics
• Data source where data come from
• Escalation of process to management
• FDA metrics (2017 approach vs. 2019 approach
• Hierarchy of metrics
• Leading vs. lagging
• Predictive vs. descriptive
• Normalization of data for long-term comparability
• Prioritization
• Purpose of metrics
• Quality culture, metrics as part of
• Targets, how to establish

One item in the “Points to Consider” section may be highlighted here specifically: Standardization of KPIs and metrics. This item is central to understanding the complexity of creating a Best Practice Guide. Whereas standardization of KPIs and metrics is a prerequisite for anyone who uses these tools within an organization for assessing the state of the pharmaceutical quality system and its subsystems, the attempt to standardize between different organizations has repeatedly proven to be a challenge. One reason for this is because good definitions of KPIs and metrics are usually quite specific to an organization, especially when it comes to such details as how to calculate the KPIs from the raw data or how to calculate the targets. Definitions may also vary by size of organization, because smaller organizations may need fewer complex hierarchies of metrics than larger organizations. Furthermore, good KPIs and metrics systems need to undergo continuous review and occasional controlled adjustment of definitions to the dynamic environment in which they are used, e.g., to changes in the competitive environment and coupled changes in business objectives, strategies and targets.

The presentation at the QSIG meeting raised great interest among the participants and stimulated a lively discussion, including suggestions for further work and the most appropriate format for publication of the Best Practice Guide for Using KPIs and Metrics within the available PDA formats.

If you are interested in participating in the activities of the subgroup, additional participants are welcome to assist in the creation of the final version of the Best Practice Guide.

References

1. Submission of Quality Metrics Data Guidance for Industry, Draft Guidance, U.S. FDA, 2016, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/submission-quality-metrics-data-guidance-industry
2. Case for Quality, U.S. FDA, 2020, https://www.fda.gov/medical-devices/quality-and-compliance-medical-devices/case-quality