FDA's Quality Metrics Guidance: Reading Between The Lines
A Q&A with Lou Angelucci
One week before we published the article 5 Important Takeaways From The FDA's Revised Quality Metrics Guidance, Life Science Connect attended a thought-provoking presentation about said guidance at a local ISPE event in Philadelphia. While the article covered several key things the guidance document did communicate to industry, the ISPE presentation — given by Lou Angelucci, currently a project management consultant to Johnson & Johnson — focused more on what the guidance document failed to say and do … and the implications of those omissions. We tracked down Lou after his talk and asked him to elaborate on some of the issues he had raised.
Life Science Connect (LSC): How would you characterize the current view of quality metrics in the pharmaceutical/biopharmaceutical industry?
Lou Angelucci: The word I would use is “confused.” The FDA’s new draft guidance Submission of Quality Metrics Data truly watered down the requirements from the previous version of the guidance, and also made reporting voluntary (for now). In addition, the metrics the agency is requesting are not so much quality metrics as they are production and distribution results.
For example, the new guidance asks for things like number of lots started and number of dosage units distributed, which it will use to calculate the metrics lot acceptance rate (LAR) and product quality compliant rate (PQCR) respectively. These are not quality metrics so much as inventory metrics, in my opinion. While you could say that if the number started is higher than the number distributed, there might be a problem, you are certainly not tying quality directly to the situation.
In another example, the FDA is requesting the total number of out-of-specification (OOS) test results, and those invalidated by the lab, to determine invalidated OOS rate (IOOSR), the third of three quality metrics identified and defined in the new guidance. Too many invalidated OOS observations could be an indication that laboratory testing is faulty, but is this truly a quality metric or just an indicator for a future FDA investigation?
So, both the FDA and industry have some work to do in defining what is meant by the term “quality metrics” — and which quality metrics are most important.
LSC: Does the new draft guidance address any of the concerns industry raised in response to the guidance’s predecessor, Request for Quality Metrics (2015)?
Angelucci: It appears as though the FDA may have indirectly addressed one concern by establishing the voluntary phase for quality metrics compliance. Timing of implementation was an issue raised in a number of the comments submitted.
The agency also removed the statement that foreign firms are exempt from submitting quality metrics data. Some commenters expressed concern that such an exemption would give foreign firms an unfair competitive advantage.
LSC: What industry concerns did the new draft guidance fail to address?
Angelucci: Medical devices were completely left out of the guidance, with no discussion or references. At the very least, the FDA could have referred to a similar effort that is being undertaking on the medical device side, where quality metrics are also being discussed. This topic would be of vital concern to drug-device combination products.
Also, no explanation was given as to why so many biotech disciplines were exempt from reporting quality metrics — in either version of the guidance.
The 2015 draft guidance touched upon other potential quality metrics related to quality culture and process capability/performance. No mention of these metrics was made in the new version, so it is unclear what other metrics may be of concern or interest to the FDA aside from LAR, PQCR, and IOOSR.
In the new draft guidance, the FDA again pointed out that quality metrics play a very important role in activities such as process validation. However, the three metrics requested in the guidance are not even monitored or of any concern during validation activities in the industry.
LSC: Does the new guidance raise any new questions?
Angelucci: For one, the body text doesn’t explicitly state that the new draft guidance is intended to be a total replacement of the previous draft; nor does it say that the previous draft was to be ignored and shelved. In fact, upon first read of the new version, you get the impression that this is a completely new quality metrics guidance, not an update to the original version. Only upon reading the associated foot notes do you learn that it was issued as a replacement to the 2015 draft guidance.
Perhaps most perplexing is the lack of an explanation of the true measure of quality — and a clear definition of the associated metrics the agency is looking for (outside the basic production-related and OOS). There is no discussion of severity or categorization of metrics (such critical, major, or minor). As a result, the industry is left without a clear path forward for implementing quality metrics.
Further muddying the waters, FDA’s Quality Metrics Technical Conformance Guide (2015) was intended to explain how quality metric data should be electronically submitted to the FDA for review. However, the document was written for the Request for Quality Metrics draft guidance has not yet been updated to accommodate the changes in the new Submission of Quality Metrics Data guidance document.
Finally, the earlier draft guidance came across as much more demanding and threatening with regard to compliance and enforcement. In the new version, the enforcement provision paragraphs were all removed, resulting in a much less intimidating tone. While this may sound like a good thing, how FDA will handle compliance after the voluntary program concludes remains an open question.
LSC: Given all the confusion surrounding the new guidance — and quality metrics in general —what should pharmaceutical companies be doing right now to prepare for an uncertain future?
Angelucci: Despite the ambiguity, all companies should be establishing relevant metrics that they can use to effectively monitor their quality process. They should also perform rigorous evaluations of these metrics so that they will be defensible in the case of a potential FDA audit. At the very least, they should pay close attention to 2015 version of the guidance, since it suggested potential metrics that are more closely aligned with current metrics being monitored in the industry. Another important resource are ISPE’s Quality Metrics Wave 1 and Wave 2 reports, which provide some good examples of quality metrics that may be of interest to the FDA.
I would recommend participation in the voluntary phase of the FDA quality metrics program, set to begin 2018, even though it is not mandated. Involvement in this initiative will demonstrate a good faith effort on the part of a firm in supporting quality metrics. That being said, I would not recommend giving the FDA any metrics beyond those they have requested and defined in the new guidance (LAR, PQCR, and IOOSR).
LSC: What tools are available to help manufacturers analyze and standardize quality metrics in their organizations?
Angelucci: There are a number of online and automated tools available, and the standard application of these tools will go a long way towards developing consistency of quality metrics interpretation. There are statistical packages that can be purchased and used to calculate and monitor things such as process capability index (CpK).
For example, while ASQ quality tools are not addressed in the FDA guidance documents, they do offer standard way to measure and record quality metrics. In addition, Minitab is a very good source for the automated and calculated analysis of associated metrics such as CpK. With a little research, you will discover a number of other statistical packages available for compiling and interpreting quality metrics-related data.
LSC: What do you think the future will hold for the FDA and quality metrics?
Angelucci: With the new Trump administration, I can’t imagine that the FDA will do much of anything related to mandating quality metrics, since implementing requirements would be tantamount to issuing a new regulation — or at least a quasi-regulation in this case. Instead, I think FDA should focus on working with industry members and organizations such as the ISPE to develop standards, best practices, and methods that can be shared and taught to all practitioners of quality and to the industry as a whole.
About Louis Angelucci
Louis Angelucci is currently a senior project management consultant with Johnson & Johnson, specializing in engineering initiatives regarding qualification, validation, and compliance. He is a pharmaceutical professional with over 20 years’ experience in quality assurance, quality control, regulatory affairs, validation, consent decree remediation, and cGMP compliance in the pharmaceutical and medical device industries. He has worked at companies including Johnson & Johnson, Bristol-Meyer Squibb, Pfizer, Schering Plough, and Merck.
Angelucci is a degreed engineer with two master’s degrees in engineering. He holds industry certifications with ASQ as a CQE, CQA, and CPGP, and a PMP certification through the Project Management Institute. Angelucci is also a member of the IVT board of directors. He has published numerous articles on the subjects of validation and compliance and has been a speaker to industry groups such as ISPE, IDE, IVT, PDA, and ASQ.