Is GMP Quality System Auditing Fundamentally Flawed? — A Data Integrity Alternative

A quality systems approach to GMP auditing has become common practice among both regulatory authorities and internal audit programs at pharmaceutical companies. The method originated with the FDA Center for Devices and Radiological Health’s (CDRH) Quality System Inspection Technique (QSIT), introduced to the medical device industry in August 1999, and was later adopted by the Centers for Drug Evaluation and Research (CDER) and Biologics Evaluation and Research (CBER) for pharmaceutical drug inspections.

In September 2006, CDER, CBER, the Office of Regulatory Affairs (ORA), and Center for Veterinary Medicine (CVM) published Guidance for Industry, Quality Systems Approach to Pharmaceutical CGMP Regulations. Pharmaceutical inspection approaches rely on the evaluation of several or all components of the quality system: quality, production, laboratory, materials, facilities and equipment, and packaging and labeling.  FDA deems that if one of these systems is out of compliance, then all may be deemed to be out of compliance.  This method provides structure for internal audits, supplier/contractor qualification and monitoring, due diligence efforts, and agency inspections. One area where it has serious limitations, however, is in the assessment of data management and data integrity.

A Blind Spot In The Quality System Auditing Approach

A routine quality system approach to auditing can frequently fail to identify data management and data integrity problems in pharmaceutical manufacturing operations.  As an example, consider the common questions asked and evaluated during a quality systems audit of out-of–specification (OOS) events and how they are managed. I chose this example because it demonstrates linkages among laboratory activities, deviation investigations, and corrective and preventive actions (CAPAs) and thus provides a good way to evaluate the health and effectiveness of a pharmaceutical quality system. The example is also applicable to firms that manufacture APIs, intermediates, or dosage forms.

In this situation, the current quality system audit approach will evaluate the processes and procedures for identifying and reporting OOS results, their investigation, invalidation of results where appropriate, retesting, and resampling. Procedures also address how the OOS result is considered in lot release decision making.  Nominally, an audit of this area will include review of OOS results within a particular time period or for a particular product, their investigations and outcome, and any corrective and preventive actions and their effectiveness. This often results in determinations that investigations were not adequate for any number of reasons, including but not limited to: staff failed to follow procedures, results were invalidated for reasons not scientifically supported, or the number of samples retested was not justified. 

However, the initiating action of OOS review is the identification and reporting of a suspected OOS event. What the common quality system approach fails to address are the situations where electronic data are not reviewed and OOS events may not be formally identified, or, even worse, are hidden or intentionally obscured or deleted. For example, data could be inappropriately manipulated by repeated manual reintegration until the desired result is obtained.  Alternatively, the failing data could be simply deleted and not addressed at all.  If the electronic system has the critical metadata (audit trails) enabled, data reviewers will become aware of these actions if and only if they review the electronic data.  If OOS events occur where staff share logins and passwords, it is impossible to assign responsibility to a unique individual and to have documented evidence of who to query during a suspected OOS investigation. Even in instances where the final test result is “meets specifications,” it does not ensure that the procedures and processes used to generate the results are acceptable or in compliance with regulatory agency requirements.  Moreover, it does not ensure that all data generated in the process of achieving the passing result was considered in the lot release decision-making process.  FDA cites 211.194 in these instances, when not all laboratory records are considered in reaching a disposition decision.

Thus, a classical quality system audit is unlikely to evaluate OOS results that may be found only in the electronic data, where they may have been obscured, ignored, or deleted.   

Why Audits Should Begin With Data Management And Data Integrity

To address this problem, it may be time to consider a dramatic change in approach to some types of GMP auditing, driven by the premise that if we cannot trust the data generated by a firm, what can we trust?  It doesn’t much matter to me, as an auditor, how well written the governing procedures and processes are if the underlying data is not completely trustworthy.  Like any evaluation, this approach must focus on serious deficiencies — those that can have potential impact on product quality and patient safety — and not just the occasional minor error in documentation.

The proposed approach focuses on identifying the data management and data integrity status at the site before proceeding to an evaluation of any specific system.  In this approach, auditors evaluate how data is controlled as it is collected, recorded, processed, reviewed, approved, and archived throughout its lifecycle.  The auditor is tasked with determining whether the data can be or has been altered, modified, or deleted.  Data considered for evaluation first would be those associated with product release, critical in-process determinations, or release of critical raw materials.

Also included in this evaluation would be the computer validation status of the various systems.  This evaluation would NOT require review of reams of validation documents but rather would focus on the general principles and how requirements were developed, incorporated, and tested as part of the validation process.  It would include a review of documented evidence that the validation met the requirements of Part 11 and Annex 11.  This approach may result in an audit limited only to data management and data integrity when serious deficiencies are identified early on.  When data is not trustworthy, evaluation of several quality systems is unnecessary to draw a conclusion regarding the firm’s GMP compliance status.

This approach will likely result in the need to provide additional training for auditors in requirements and expectations for data management and data integrity — and how to perform these evaluations.  Among the best training materials are the publicly available from 483s and warning letters from FDA.  Data management- and data integrity-focused audits are not rocket science, but they do require a knowledgeable audit group with deep expertise in GMPs, critical thinking skills (to know which questions to ask and how to connect the dots), and where to look for deficiencies.

Another reason to apply this new approach is that it addresses the need to quickly identify serious deficiencies with potential impact on patient safety and product quality.  This is particularly important for visits that focus on vendor qualification, due diligence, and contract manufacture and contract laboratory qualification. Time is generally limited for each of these activities. Between the conference room presentations and limited tour of the contract manufacturing facility, there is precious little time to evaluate meaningful raw data and original records.  The alternative approach could provide a more efficient use of limited resources and more accurately establish the overall validity of data and reports generated by the firm in question.

In these audits, confidentiality agreements may preclude the evaluation of raw data.  In such cases, the focus should be on the controls that are in place to ensure the trustworthiness of both electronic and paper records.

Perhaps the most important justification for this approach is the increasing percentage of FDA warning letters over the last four fiscal years citing data integrity deficiencies.  In FY2016, 80 percent of warning letters (excluding those issued to compounding pharmacies) cited deficiencies in data management/integrity. (For further discussion of this recent trend, see my previous article An Analysis Of FDA FY2016 Drug GMP Warning Letters.) This was true for warning letters issued both to sites inside and outside the United States. Thus, it seems reasonable that if the FDA is placing such emphasis in this area, we outside of the agency would be prudent in doing the same.

Key Components Of A Data Governance And Data Integrity Audit

This section describes some evaluations that can be included in audits of data governance and data integrity. The examples focus on the QC laboratory, but the concepts are generally applicable to any GXP computer system.  Firms must recognize that 21 CFR 11 requirements apply whenever electronic records and/or electronic signatures are used in GXP processes and activities.  Firms that maintain they operate primarily paper-based systems should consider that their laboratories depend largely on instrument-associated computer systems.  The raw data from the dozen HPLC and GC instruments are not paper based.  Further, a firm cannot write an SOP that exempts itself from compliance with this regulation. It is useful to read the preamble accompanying the 21 CFR Part 11 final rule published in 1997 to more fully understand intent and its applicability.  

In general, firms should have a data governance program, even though they may call it something different.  The UK’s Medicines & Healthcare Products Regulatory Agency (MHRA) defines data governance as “The sum total of arrangements to ensure that data, irrespective of the format in which it is generated, is recorded, processed, retained and used to ensure a complete, consistent and accurate record throughout the data lifecycle.”  Further, MHRA stated in 2013 that firms should ensure their contractors and suppliers have an adequate data governance system.  This is a program that should cover all GMP areas and perhaps include GXP efforts.  Firms should determine for themselves what the appropriate components of such a program should include and how this should be applied to audits they conduct. 

Most data integrity audits focus on QC laboratories, and this is where most, but not all, items cited in inspection reports are found.  The list provided below identifies an approach to audits of QC laboratories focusing on data governance and data integrity and should be within the skill set of all firms' GMP audit staff.  These items do not represent forensic auditing of laboratory systems that necessarily require additional skills. They represent the areas that have been cited in both FDA form 483s and warning letters for the past 15 years. Specifics that may be addressed by auditors in the QC laboratories include, but are not limited to the following:

• A key expectation is that firms have a list of all GMP or GXP computer systems for review by regulatory authorities.  This includes systems in areas other than the QC laboratories.
• Laboratory instrument-associated computer systems and other computer systems should be identified and assessed for their risk to product quality and patient safety, and requirements should be defined and validated appropriately.  Periodic evaluations should be performed and documented to ensure the systems remain in a validated state.
• Laboratory instrument-associated computer systems and other GXP computer systems should be assessed for compliance with 21 CFR Part 11 - Annex 11 and the MHRA guidance on data integrity.  Gaps should be identified and supported by a remediation plan that includes a timeline.
• Changes to computer system software and hardware should be appropriately assessed and should not be made outside of the quality system.  For example, an outsourced help desk function should not make changes to GXP systems unless staff has the appropriate training and qualification.  These changes should be documented within the quality system process, not exclusively in a help deck ticket.
• The following limited list of activities to evaluate in the QC laboratory includes items from warning letters and forms 483 made available by FDA, as well as those described in regulations and guidelines:
  • Is configuration of the instrument-associated software qualified and tested appropriately to meet predefined requirements?  Where is this documented?
  • Are passwords and log-ins shared or are they unique to each individual?  Shared passwords prevent attributing specific actions to specific individuals.  This includes actions such as logging into the system, collection of data, processing data, and modifying or deleting data.
  • Are access privileges assigned appropriately?  Is there a listing of who has what privilege and actions that may be taken by each?
  • Are time/date stamps fixed, or can individuals alter them?
  • Is electronic data, including critical metadata (audit trails) reviewed as part of laboratory data review, lot release, or OOS investigations?  In the absence of audit trails and their review, it is impossible for the reviewer to determine whether data has been altered or deleted.  Of particular importance is whether data was modified or deleted because it was  an OOS result.
  • Is the review of electronic data described in an SOP, and are reviewers appropriately trained in what they are to evaluate?  How is the review of the electronic data documented?
  • How quickly can the audit trails be provided to an auditor? When it takes four staff members a half hour to locate them, it suggests the audit trails are not routinely evaluated.
  • Is data periodically backed up to a secure server, or is it deleted to make space on existing hard drives?  Is the backup automatic or manual?  If the transfer is manual, how does the firm ensure that the transfer is complete and that data is not inadvertently deleted or altered in the process? Are these backups conducted according to a predefined schedule? If using automatic backup, has the process been validated, and is it routinely successful?  If not, why not?
  • Is data archived?  Is the meta-data associated, or able to be associated, with the archived electronic record?  Are the archived records protected against environmental factors such as fire and flood?
  • Equally import to the laboratory instrument-associated computer systems are computerized controls applied on the floor in the manufacturing equipment.  The manufacturing floor has received less attention from regulators in the past, but that is changing rapidly.

Benefits Of Checking The Trustworthiness Of The Data First

Quality system auditing does not generally include detailed review of the raw data underlying the governing processes and procedures.  Thus, it can fall short in detecting instances of serious data integrity failures that may impact product quality and patient safety.  The failure becomes more pronounced when time for audits and assessments is limited to a day or two, specifically for due-diligence evaluations, vendor qualification, and periodic auditing of critical suppliers, contract manufacturers, and contract laboratories. 

To make the best use of limited resources and time allocated to audits and assessments, I propose auditors focus first on data management and data integrity.  Audits and assessments should establish that the auditee produces trustworthy and valid data before pursuing more broad assessments of other aspects of the pharmaceutical quality system.  An evaluation of the facility design and visual evaluation of facility and equipment maintenance should be included based on facility walkthroughs.  Advantages of this approach include that it:

• Reflects the FDA-identified seriousness of this issue, with approximately 80 percent of drug GMP warning letters in FY2016, both domestic and abroad, citing data governance and data integrity concerns.  Other global authorities have identified deficiencies in this area, but the data is not available to establish percentages.
• Requires the auditors to take a new approach and not permit themselves to be isolated in conference rooms evaluating SOPs that cannot provide documented evidence of the trustworthiness of data generated by the auditee.
• Results are fact-based and conclusions regarding GMP compliance status are data-driven. 
• Can be performed with a limited number of auditors in a limited amount of time.
• Permits rapid determination of the validity and trustworthiness of data generated by the auditee, and equates this with their overall GMP compliance.
• Provides a means of effectively qualifying and ensuring effective ongoing evaluations of key players in the drug supply chain based on documented evidence supported by trustworthy data.
• Evaluation of facility and equipment maintenance is completed based on a facility walkthrough with minimal extra effort. This, in addition to evaluation of data governance and data integrity, adds another important dimension to audits when time is limited. 

The focused auditing approach described herein provides a means to more quickly determine the trustworthiness of GMP data that is necessary to ensure patient safety and product quality.   It also supports an efficient use of limited resources.  After the trustworthiness of the data is established, it is appropriate and necessary to move onto a more detailed evaluation of the quality systems.  If data is determined to not be trustworthy, it suggests that the additional effort of evaluating the quality system may not be appropriate or add value until the data management issues are remediated.