Proficiency Testing (PT) Part 3: Quality Indicators

Last month we discussed the rules associated with evaluating your PT results, and how to investigate any unsuccessful surveys. In the last of this 3-part series we’ll review ways to utilize your PT reports to check for trending in your patient values – shifts, trends and bias. Your PT results can help show you developing problems and allow you to correct them, before they become failures or begin to affect patient care. Before declaring a failure as a ‘random error’, be sure that it truly is.

Accuracy & Systematic Errors

Accuracy describes how close your measured value is to the reference value – did you obtain the correct result? This will be affected by systematic errors, such as using expired or degraded reagents, changes in lot numbers or calibration values, or instruments with analytical lamps or lasers near the end of their use life. Systematic errors are reproducible inaccuracies that occur in the same direction; all results will be falsely low or all results will be falsely high. If systematic errors are present, all results will show similar deviations from the true value. Bias is a measure of how far off your results are from their true intended value.

Precision and Random Errors

Precision on the other hand refers to the overall agreement of results upon replicate testing – will you get the same value if you repeat the test? Precision is affected by random errors, such as incomplete aspiration of a sample or reagent due to fibrin clots or air bubbles, operator variability in pipetting technique, or temperature fluctuations. Random errors are statistical fluctuations in the measured data due to the limitations of the assay in use. These errors will occur in either direction from the mean, unlike systematic errors that will be on the same side. Imprecision can be measured and monitored by evaluating the standard deviation (SD) and coefficient of variance (CV) for an assay.

Let’s look at some example PT results from CAP, and see what hints these reports reveal to us.

  • Albumin: Although all results passed and were graded as ‘acceptable’, there are still issues that should be looked into. For the last 3 surveys in a row, the plot shows that nearly all samples have been on the same right side of the mean. When comparing the value of the % relative distance from the first survey to the most recent one, you can see that the values are trending worse and getting closer to being unacceptable if the pattern continues. Additionally, be mindful of the standard deviation index (SDI) value reported. This is a measure of your bias, and how far off your values are from the mean. It should be defined within your Quality System Manual (QSM) the values which should trigger an investigation, but as a general rule, anything >±2.0 indicates a potential issue. (
  • Alkaline Phosphatase: Again all results passed, but 3/5 samples have SDI values >±2.0. The first survey had all values to the right of the mean, the second survey was a nice tight even mix of +/- bias, and now with the most recent survey all values are appearing to the left of the mean. If this shift coincides with a change in lot number, a calibration may be necessary to get results back on target to help lower the SDI values.
  • GGT: Although only 1 sample was graded as unacceptable, all of the results for this recent survey were at risk of being failures due to how close they were to the upper limit of acceptability. Results like this should be very carefully evaluated to ensure that there is no impact on patient care. Provided the sample stability has not been exceeded, all 5 samples should be repeated. If the repeat values are closer to the target mean, you will need to identify what went wrong on the day the samples were originally tested. If the repeat values are still grossly far from their intended target, a full patient lookback would need to be performed from the time the samples were originally tested until the day they were repeated, as there is a systemic problem that has now continued for weeks or longer.  
  • Vancomycin: Similar to the albumin example above, these results show a trend occurring between the first survey and the most recent; however unlike albumin these are moving in the correct direction. Values are getting closer to the target mean, and SDI values are decreasing, suggesting that any corrective actions implemented after the last survey were successful.
  • Lithium: This shows a good example of what you hope all of your quantitative proficiency results will look like. There is a nice distribution of results on both sides of the mean, and SDI values are all relatively low. Values such as these allow you to have complete confidence in the accuracy of your patient results.
  • MCH: Focus on sample #2, with an SDI of -1.9. The other samples within this survey all appear fine, but it looks as though there was truly a random error with sample #2. When we look at the affiliated analytes we see a similar issue with the RBC count of sample #2, which coincides with our decreased MCH (a reminder for our non-hematology readers, MCH = (Hgb x 10)/RBC). For any calculated values, be sure to evaluate the all parameters together as well as individually to serve as a common sense check that your results are appropriate and truly make sense.

It is important to have a robust quality assurance program that outlines what to monitor, key decision points for when to take action, and guidance on what those actions should include. Your proficiency testing results can provide you with a ton of useful information to evaluate the overall quality of laboratory, and help provide confidence in the patient values being reported out as well.

-Kyle Nevins, MS, MLS(ASCP)CM is one of ASCP’s 2018 Top 5 in the 40 Under Forty recognition program. She has worked in the medical laboratory profession for over 18 years. In her current position, she transitions between performing laboratory audits across the entire Northwell Health System on Long Island, NY, consulting for at-risk laboratories outside of Northwell Health, bringing laboratories up to regulatory standards, and acting as supervisor and mentor in labs with management gaps.

Proficiency Testing (PT) Part 2: Investigating Failures

Last month we discussed the rules and requirements for how to properly perform proficiency testing (PT) within your laboratory. In part 2 of this 3-part series we’ll review the rules associated with evaluating your results, and how to investigate any unsuccessful surveys. Still to come in part 3 we will look into how to utilize your PT results to monitor for trends and shifts in your values.

The rules:

  • Performance Review: Laboratories must initiate and document a review of their PT performance evaluations within 2 weeks of notification that results are available. This includes a review of both graded and non-graded/educational analytes and events as well.

Key things to note: Even though educational samples are not formally graded, you should still verify the accuracy of your results, with appropriate follow-up for any failures. CAP specifically requires you to evaluate these educational challenges as well. Whether the sample is graded or not does not change the fact that you had an incorrect result.

  • Unsatisfactory Performance: For any unsatisfactory results, you are required to perform a root cause analysis to determine why (see below for guidance). This also includes any clerical errors – you need to evaluate your process and find ways to prevent these simple errors from happening again. If they are happening with PT samples, it is possible they are happening with patient samples as well.
  • Cessation of Patient Testing: Unsatisfactory events indicate that there was a problem with that particular survey; whereas unsuccessful events indicate there has been a pattern of unsatisfactory events/samples and a larger problem exists. If a pattern of poor performance is detected, you may be asked by your local state department of health to cease all testing for a particular analyte.

Key things to note: This also applies to clerical errors. Even if there was no technical problem with the accuracy of your results, failure to submit results on time or clerical errors made while submitting can also have severe impacts on your ability to continue offering that test.

  • Remedial Action: If you’ve been notified by your PT provider or state DOH to cease testing, there are extensive steps that must be completed to prove that the problem was correctly identified and corrected. You must also identify where samples will be referred to for tests you are unable to perform in-house.

Key things to note: If testing has been removed from your laboratory, you will be required to demonstrate successful performance in 2 consecutive PT survey events for the analyte(s) in question before being granted permission to resume patient testing. This can cause significant delays and financial impact for your organization.

Root Cause Analysis: Investigate to determine who, what, why, when, and how the event occurred. Be sure to evaluate all phases of testing to ensure you identify all potential causes.

  • Pre-Examination:
    • Human Resources – evaluate the training and competency records for staff involved in the handling and testing of samples.
    • Facilities – reagent inventory control & storage temperatures, equipment maintenance and function checks
    • Standard Operating Procedures (SOPs) – staff compliance with written policies, bench excerpts are current and valid, document version control up to date
    • Specimen –test requisition/order entry (was the correct test code ordered/performed?), labeling (were aliquot/pour off tubes properly labeled?), transport (was appropriate temperature requirements maintained until testing performed), quality (was there visible deterioration with the sample prior to testing or cracked/damaged tubes received?), quantity (was the original sample spilled or leaking causing an incomplete aspiration of sample by your instrument?)
  • Examination:
    • Method Validations – were instruments current with calibration requirements, any bias noted during instrument correlation studies, values being reported within the verified AMR
    • Environmental Controls – temperatures/humidity within tolerance limits, for light sensitive studies (bilirubin) was there excessive exposure of the samples to light prior to testing, excessive vibrations occurring that may have affected results (nearby construction or a running centrifuge on a shared work bench)
    • Quality Control – did QC pass on the day of testing, was QC trending or shifts noted that month
    • Analytical Records (worksheets) – were sample results transcribed correctly between the analyzer and worksheet, between the worksheet and LIS
    • Instrument Errors – were any corrective actions or problems noted for the days before, during, or immediately after testing of PT occurred
    • Testing Delay, Testing Errors – were samples prepared and not tested immediately leaving them exposed to light or air which may affect results (blood gas samples), any errors or problems noted during testing that may have caused a delay or affected accuracy of results
  • Post-Examination:
    • Data & Results Review – check for clerical errors, was data trasmitted correctly from the instrument into LIS, was data entered correctly on your PT provider entry submission forms
    • Verification of Transmission – did your results correctly upload to the PT provider website, was there an error or failure with submission
    • Review of LIS – are your autoverification rules set up correctly, is the autoverification validation current with no known issues
    • Patient Impact – perhaps the most important step to take when reviewing PT failures, you need to determine what impact your failure had on your patient results. Depending upon the identified root cause and how different your values were from the intended response, this can potentially pose a severe impact on your patient values tested at the same time as the PT samples.

Involve your medical director to determine if the discrepancy in results is clinically significant. Perform a patient look-back to review patient values for the same analyte with the failure during the time period in question. Evaluate the bias that was present, and if deemed to be clinically significant then corrected patient reports will need to be issued with a letter from the medical director explaining why. If it was decided that the discrepancy is not clinically significant, document this in writing and keep on record with your complete investigation response.

Corrective Actions/Preventative Actions – use the following set of questions to help guide you in ensuring that the problem identified during your root cause analysis will not occur again:

  • What changes to policies, procedures, and/or processes will you implement to ensure there will not be a repeat of this problem?
  • Do any processes need to be simplified or standardized?
  • Is additional training or competency assessment needed? If so, identify specific team members to be trained, and who will be accountable for performing and documenting this training.
  • Is additional supervisory oversight needed for a particular area or step?
  • Are current staffing levels adequate to handle testing volumes?
  • Would revision or additional verification of the LIS rules address or prevent this problem?
  • How can the communication between laboratory, nursing, and medical staff be improved to reduce errors in the future?

Continuous Process Improvement – after identifying the true root cause(s) for the failure and implementing corrective/preventative actions, you need to evaluate the effectiveness of those improvements. Have they been sustained? Are they working to correct the original problem? Have you created new problems by changing the previous process?

  • Quality Management Meetings – if necessary, increase the frequency of these meetings during the evaluation period for timely feedback to management and staff
  • Implement internal audits and quality indicators to check for potential issues
  • Access the specimen transport conditions to ensure they meet test requirements
  • Evaluate and monitor your turnaround time metrics to track problem specimens and impact of testing delays
  • If necessary, increase the frequency when QC is performed or calibration frequency if stability issues are identified

Performing a thorough root cause analysis for any failures will allow you to implement appropriate corrective actions that will address the true issues. Having a robust quality management program will help ensure these issues are identified and corrected in a timely manner, and reduce the potential for the dreaded Cessation of Patient Testing letter from your local DOH.

Coming up in the final installment of this series on PT testing, we’ll review all of the quality indicators and data that can be found in your PT evaluation reports to help ensure you’re on track for accurate patient values.

-Kyle Nevins, MS, MLS(ASCP)CM is one of ASCP’s 2018 Top 5 in the 40 Under Forty recognition program. She has worked in the medical laboratory profession for over 18 years. In her current position, she transitions between performing laboratory audits across the entire Northwell Health System on Long Island, NY, consulting for at-risk laboratories outside of Northwell Health, bringing laboratories up to regulatory standards, and acting as supervisor and mentor in labs with management gaps.

Proficiency Testing (PT)Part 1: Are You Doing it Right?

Every laboratory knows that they must participate in proficiency testing (PT) for all of the regulated analytes they report. But did you know that there is more to it than simply checking your overall score in each survey you participate in? Whether you utilize samples from the CAP, API, or have developed your own in-house blind sample testing algorithm, there is a lot of data available to help you assess the quality of your laboratory program. In the first of this 3-part series, we’ll review why PT testing is important and the rules that must be followed. In part 2 we’ll discuss how to properly perform an investigation when scores are <100%. Lastly, in part 3 we’ll look at how to review your results so that you get the most out of them for a successful quality laboratory.

Why participate? Well frankly, because you have to. It is a CLIA/CMS requirement, and if your lab has additional accreditations, those agencies will have their own rules and requirements as well (we’ll get to the rules in a little bit). But outside of the regulations stating you must participate; all labs should want to participate. It’s an opportunity to check your accuracy against peers who are using the same instrumentation as you. Similar to utilizing an affiliated QC report, this is a way to see what the “real” value is supposed to be (despite what a manufacturer may claim it to be), and how close/far off your lab is to that true value. It can help you identify potential problems before they become huge problems with patient values being affected, and it’s also a great way to satisfy competency requirements for your staff.

The rules:

  • Participation: For every regulated analyte being tested under your laboratory permit1, you must participate in a CMS-approved PT program2.

Key things to note: This only applies to testing performed using non-waived methodologies. Waived testing is exempt from PT requirements; although it is still recommended that participation occur if an evaluation program is available. Additionally, this only applies to your primary instrumentation. For example, if you have an automated urinalysis reader and your backup methodology is to read dipsticks manually, you are only required to participate in PT for the primary methodology. (Your backup method would then be evaluated for accuracy through semi-annual correlation studies.)

  • Routine Analysis: Unless otherwise instructed by the provider of your PT samples, PT samples are to be treated the same as patient samples. Meaning they are handled, prepared, processed, examined, tested and reported the same way you would perform patient testing; AND by the same staff who would handle patient testing.

Key things to note: If nursing staff perform a particular test within their unit (for example, ACT testing in the cardiac cath lab), it is those nursing staff members who must run the PT samples. You cannot have the laboratory perform PT testing unless the laboratory also performs the patient testing. Additionally, PT samples should be rotated among all staff members who perform patient testing. Meaning all shifts, and all days of the week that the test is performed – don’t let the day shift get all the fun.

  • Repeated Analysis: Similar to rule #2, unless you routinely perform duplicate testing on your patient samples, you cannot perform duplicate or repeat testing on your PT samples. You cannot run a PT sample in duplicate “just to make sure.” Patient samples are just as important to be accurate as a PT sample, which is why we participate in a PT program in the first place.

Key things to note: After the date that laboratories are required to report results back to the PT provider, you are then allowed to use the samples for repeat testing. This can be used to check for uniformity in grading of reactions among staff members, and to assess annual competency. But only after the submission date has passed.

  • Interlaboratory Communication: You cannot discuss the results or samples from a PT survey with any other laboratory (or Facebook user group) until after the results submission deadline has passed. Doing so before that time would be considered cheating. The point of PT testing is not to see how good your networking skills are, but to ensure accuracy of your own results. Plus, the other lab may not be as good as you think they are.

Key things to note: If your laboratory is part of a larger integrated health system, be careful that you have separate designated staff assigned to enter results from each location. Entering results for more than one permit number by the same person would be considered a violation of the interlaboratory communication rule as they could compare results from Lab A to Lab B prior to submitting. Also, be mindful of what you put on social media. User groups are a great networking resource and learning tool, but you still need to follow the rules. Violating them in a public arena such as Facebook for all the world to see would put yourself and your organization in great jeopardy if you were caught. 

  • Referral of Samples: You are not permitted to forward or share your PT samples with any other laboratory until after the result submission deadline has passed. Similarly, if your laboratory has received PT samples from another lab, state regulations may require you to notify your local Department of Health to inform them of the violation.

Key things to note: The intended purpose of performing PT testing is to verify the accuracy of your own laboratory testing. If you would routinely send a positive sample to a reference lab for additional confirmation testing, you would not do so in this case. Simply report out the values for the tests that your laboratory performs only. The reference laboratory will have their own PT samples to check accuracy for the confirmation testing they perform for you. Ensure your testing menu is up to date and accurate so that your PT provider is not expecting values for a confirmatory test if you do not physically perform it in-house.

  • Records Retention: Ensure that all records and documents related to the testing of PT samples are saved for the amount of time required by your regulatory agencies (typically 2-5 years). This includes instrument print outs, LIS chart copies of the filed results, QC records for the day of testing, and any associated worksheets used to document your results.

Key things to note: Retaining a copy of the instrument maintenance logs and QC records along with the actual PT results will help you investigate any scores that are less than 100%.

  • Attestation: Both the laboratory director and all personnel performing testing must sign the included attestation statement. This is not just a way to track who performed the test, but is a legal binding document assuring that testing was carried out appropriately as per the rules defined above.

The penalties for labs that are caught violating the rules (whether intentionally or not) can be quite severe. These penalties can include the revocation of your CLIA permit; a ban for the laboratory owner and laboratory director; as well as possible financial penalties and fines.

Coming up in the next blog we’ll review the rest of the rules related to evaluation of your scored PT results, and how to perform a thorough investigation into any unsuccessful survey events.



-Kyle Nevins, MS, MLS(ASCP)CM is one of ASCP’s 2018 Top 5 in the 40 Under Forty recognition program. She has worked in the medical laboratory profession for over 18 years. In her current position, she transitions between performing laboratory audits across the entire Northwell Health System on Long Island, NY, consulting for at-risk laboratories outside of Northwell Health, bringing laboratories up to regulatory standards, and acting as supervisor and mentor in labs with management gaps.

Are “Clerical Errors” Acceptable in Proficiency Testing?

Laboratories across the United States must complete Proficiency Testing (PT) and score a minimum of 80% score for all specialties (the exception is Transfusion Services, which requires a score of 100 percent). Occasionally, errors on PT could be attributed to “clerical error.” However, that should not be confused with “human unavoidable error.” Clerical errors in the clinical setting could bring serious harm to the patients and therefore, should be avoided and mitigated. Risk of patient safety within the three month period of the failed test event period should also be assessed after each unsuccessful PT event and education for personnel need to be conducted.

Once a supervisor or manager discovers their department has missed a PT survey because of clerical error, sometimes the only corrective action is to counsel the individual technologist, but no preventative action is taken. When that happens, there are no safeguards in place to keep the error from happening again; after all, these types of errors can happen to anyone, not just a specific employee. There should be an investigation for the root cause of the event. Review and assessment of the root cause analysis for the event could reveal that the technical personnel might have been rushed and didn’t double check the PT sample identity or result entry for the test. Could the clerical error been caused by unmanageable workloads, or sleep deprivation? Maybe the error was caused by general inattentiveness, such as failing to read instructions or mismanaging the samples. All of these examples highlight problems that are a great scope than “clerical error.”

Institutions must ensure their tests result are accurate and are free of any clerical/ transcription errors. These types of events should be reviewed by management with an appropriate preventive action/ mitigation to prevent it from occurring in the future.

Information on policies or practices are solely from my personal experience ONLY and have NO relation to my  affiliation with any regulatory or government agency.


-Caroline Satyadi, MT(ASCP), SM, DLM, SLS, MBA, MS, CQA (ASQ) has been a laboratory management professional for over 25 years. She has worked with several different medical industries for CLIA/CMS, FDA/ICH/ISO, TJC/CAP/COLA/HFAP accreditation survey readiness.