The U.S. Government increasingly uses statistical analysis and data mining to identify fraud and abuse in the healthcare market. A key tool in the government’s arsenal is RAT-STATS software, used to sample and quantify improper claims, which then become the basis for damages and monetary penalties. As a result, healthcare providers, compliance officers and legal professionals must grasp the government’s methodologies for performing this analysis. Understanding potential weaknesses in the government’s analysis is also imperative to a sound defense.
RAT-STATS is a free statistical software package the government offers to assist with statistical analysis. In turn, the government often relies on RAT-STATS when investigating vendors for enforcement or fraud matters. While a user guide exists, the 394-page document and 245-page companion manual are overwhelmingly technical and often dispirit those who most need to understand them. This article will demystify RAT-STATS and clearly articulate what the software can and cannot do.
Hypothetical
To help illustrate topics throughout this article, we will refer to a hypothetical scenario which embodies a common use of RAT-STATS. In this, the Government has filed suit against HealthAid, a fictional national nursing home network. The government’s complaint alleges a significant portion of HealthAid’s 92,000 mental health treatments were medically unnecessary. The government’s analysis, performed using RAT-STATS, concluded over 70 percent of treatments during the past four years were unneeded, resulting in damages to the government of $88 million.
RAT-STATS capabilities
Designed to assist in performing random samples and interpreting the results, many consider RAT-STATS to be a black-box, but the reality is remarkably less complex. Consider your standard calculator, which performs discrete functions (add, subtract, etc.). RAT-STATS is not so very different.
RAT-STATS performs three main functions:
1. Determine sample size
The purpose of this function is to select a sufficient sample to yield reliable results. With limited input, RAT-STATS calculates the number of instances required to reliably infer performance for the full population.
In our example, this would denote a subset of HealthAid’s 92,000 treatments. Assuming the government requires 95 percent confidence with +/-3 percent precision for the final results, RAT-STATS would calculate a required sample size of 1,085 treatments.
2. Generate random numbers to select the sample
The purpose of this function is to ensure a sample’s randomness. RAT-STATS generates a set of random numbers used to select the sample from the total population.
In our example, these random numbers would dictate exactly which of the 92,000 treatments should be included in the sample of 1,085.
3. Infer the performance of the total population, based on the results from the sample
After selecting and analyzing the sample, this function allows inference of how the sample results relate to the total population (e.g. frequency, size, etc.).
For HealthAid, the government performs this function after analyzing the sample of 1,085 treatments. This demonstrates how the cost and frequency of unnecessary treatments translate to the total population, which is commonly called extrapolation.
RAT-STATS’ functionality is reasonably simple to understand and rather limited in its scope.
But don’t be misled: Using a calculator does not guarantee conclusions will be reliable, and using RAT-STATS is no different. A broader framework is required to render reliable conclusions.
The big picture
Overreliance on RAT-STATS without a grasp of the underlying data can lead to inaccurate conclusions. This is due to a variety of assumptions and decisions that exist outside of the software itself; these comprise the broader statistical framework and include a variety of pitfalls that may result in sub-standard analysis.
The following framework offers a comprehensive approach and helps to better achieve reliable results when conducting analysis with RAT-STATS. Rather than a black box approach, consider this framework when confronted with compliance or legal defense issues.
1. Define the question to be answered
A seemingly simple concept, this is often the first misstep. Ensure the goals precisely identify the objectives of the analysis.
In our example, the government presumably sought to determine “the cost of HealthAid treatments determined to be medically unnecessary.” This objective may miss the mark given industry standards-of-care often change over time, which can affect if and when certain procedures are determined to be unnecessary. A better question may seek procedures that were unnecessary at the time they were performed. This seemingly minor nuance can dramatically change the design of the analysis.
After defining the question you are trying to answer, it is imperative to revisit that question throughout the analysis. Too often, we see good analysis that answers the wrong question.
2. Define and evaluate the total population
The total population should directly relate to the defined objectives. Including all relevant data is important, but excluding irrelevant data is equally critical.
In our example, the government included all HealthAid claims for mental health treatment over the previous four years. HealthAid should be wary of such a broad population, since portions may be irrelevant to the analysis. For example, HealthAid operates six specialty homes for military veterans that provide extensive mental health services for PTSD treatment. These unique claims are exempt from medical necessity requirements and could skew the results if included.
When evaluating data in the total population, consider the following:
a. Are certain subsets of the data not applicable? (e.g. unique attributes, dates of service, etc.)
b. Do outliers exist that misrepresent the total population?
3. Determine sample methodology and design
Consider that a random sample should reasonably represent the total population. Sampling methodologies should account for unique data attributes that distinguish subsets of the population.
For HealthAid, consider attributes that may bias the results. Could one facility be causing a greater proportion of medically unnecessary treatments than another? If so, sampling design should reflect the disparity and allow you to prove or disprove such a hypothesis. Rogue providers may be to blame for such a result, which can dramatically reduce the government’s broad claims.
The practical design of such a sampling methodology — otherwise known as stratification — can be complex. Rather than delve into the minutia of such a design, it is more important to stay focused on the tenets of sample design:
a. Random samples should represent the population
b. Unique attributes should be reflected in the sample design
Evaluating sample design is a common approach for refuting analysis. Providers are often able to identify factors that improperly skew the results of government analysis.
4. Determine sample size
a. Use RAT-STATS software
5. Select the sample
a. Use RAT-STATS software
b. Verify the sample is reasonably representative of the population. If not, re-evaluate the sample design (Step 3)
6. Analyze the sample
This is the test-work portion of analysis, which often requires an impartial perspective.
Objectives include reliable, relevant and repeatable assessments of the data. The government regularly scrutinizes this aspect of a client’s analysis, so it is worthwhile to invest appropriate resources to ensure it is well designed and executed.
In the HealthAid example, a written standard identifying the criteria of medical necessity should be developed by clinical experts. Impartial reviewers then evaluate each claim in the sample using that as a guide. The results of the analysis should be clearly recorded and capture both the quantity and value of unnecessary treatments.
7. Extrapolate the results
- a. Use RAT-STATS software
- b. Additional analysis may be desired to aid compliance initiatives or further test hypotheses about data attributes. This additional analysis is highly encouraged.
As you can imagine, minor missteps during this analysis can have a dramatic impact once the results are extrapolated across the full population. Using this framework, companies can scrutinize the government’s analysis and identify potential missteps in an effort to minimize damages.
Conclusion
Hopefully, you no longer consider RAT-STATS a statistical “black box,” but rather a calculator with three distinct functions. Understanding when and how to properly use that calculator involves the use of a broader framework. A thoughtful and comprehensive application of that framework will lead to more reliable conclusions, and help to identify weaknesses in sub-standard analysis. Providers, compliance officers and legal professionals need to understand this framework to fully implement their own compliance analyses, and to shape a sound defense against government enforcement.
Chris Haney, CPA, CFE is a Director with Duff & Phelps and specializes in forensic accounting and healthcare fraud and abuse. Chris is a senior member of the firm’s Integrated Healthcare Group. Previously, Chris was a member of the FBI’s Forensic Accounting Unit specializing in complex white collar and healthcare violations.Prior to joining the FBI, Chris spent five years at General Electric focused on internal investigations and M&A due diligence and was a Medical Services Officer in the U.S. Army.
Related articles:
The great EHR market shakeout — it's pending, but when?
