NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Comprehensive Case Management for Substance Abuse Treatment [Internet]. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2000. (Treatment Improvement Protocol (TIP) Series, No. 27.)
Substance abuse treatment programs, including those that receive public assistance, are increasingly operating in a managed care environment. Policymaking and clinical decisionmaking in a managed care environment depend on outcome data that have traditionally described the impact of case management and substance abuse treatment interventions in terms of services used and money spent. (See Chapter 6 for more on implementing case management in a managed care setting.) An additional demand for data comes from public and private payers who want services linked to specific outcomes.
In the past, public sector substance abuse programs were not paid to collect such data and were discouraged from using funds designated for service delivery to conduct evaluations. Consequently, evaluation services often were available only through demonstration grants or through the efforts of university-based evaluators. Today, however, many providers plan, fund, and perform their own evaluations. This reflects both the mandates of funding organizations and agencies' desire to refine or improve their services. To prepare treatment programs to get involved in these efforts, this chapter first presents findings from previous evaluation efforts and then proposes a framework for facilitating quality improvement and other evaluative efforts that consider multiple stakeholders and focus on myriad outcomes and data sources.
Researchers only recently have begun to assess the effectiveness of case management. Studies conducted thus far have suffered from significant methodological problems that include small sample sizes, poorly defined or implemented case management interventions, problems in evaluation design and measurement, lack of distinction between case management and comparison interventions, poor timing, and unaccounted-for contextual factors in communities where case management was studied (Orwin et al., 1994). Problems in research design are more than an academic concern—they render results that may be misleading, difficult to interpret, and unreliable for use in developing case management programs or policy.
Although problems in research design affect other kinds of addiction treatment research, case management is especially difficult to evaluate because contextual factors play a critical role in program operations. Case management programs do not function in isolation. A key component of a successful case management intervention is the establishment of linkages to other agencies in a service network. Some researchers have suggested that the effectiveness of case management may have more to do with the environment in which it functions than with the functions of the program per se (Ridgely and Willenbring, 1992; Morlock et al., 1988). However, in spite of these difficulties, some useful findings have emerged from work in the mental health and substance abuse fields.
Much of the research on case management has been conducted in the mental health field. Reviews of its effectiveness are mixed (Bond et al., 1995; Chamberlain and Rapp, 1991; Rubin, 1992; Soloman, 1995), revealing the need to identify specific program models and expectations about which type of case management works for particular populations and at what cost (Bond et al., 1995). The Assertive Community Treatment (ACT) model currently appears to have the strongest research base for persons with initially high rates of psychiatric hospitalization, both in terms of increased retention in community based treatment programs and in reduced psychiatric in-patient days (Stein and Test, 1980). This model includes a team of case managers who work with clients in an intensive manner to address problems of daily living and who have a long-term commitment to providing services to clients as long as their needs exist (McGrew and Bond, 1995). While the model appears to be effective in reducing psychiatric hospitalization, there is little evidence that the approach results in improved quality of life or level of functioning for the client (Bond et al., 1995; McGrew and Bond, 1995; Olfson, 1990; Soloman, 1992; Test, 1992).
Evaluation of so-called administrative models in which case managers coordinate services but provide little specific clinical care is inconclusive. Some of these programs improved clients' quality of life but did not interrupt patterns of rehospitalization. However, at least one study revealed that administrative case management both increased the use of services and increased costs for clients without a concomitant measure of improvement in clients' lives (Willenbring et al., 1991).
Few studies have been undertaken on case management in the substance abuse field, and it is difficult to generalize the findings of those studies that have. One study in Canada found results similar to those in mental health studies: There are positive, measurable effects of case management, especially for clients with poor prognostic indicators at admission (such as heavy consumption of alcohol and other drugs, previous treatment failures, and lack of social support) (Lightfoot et al., 1982).
Other studies of case management in the substance abuse field have reported few or no differences for case managed clients compared to those in treatment who do not receive case management services (Inciardi et al., 1994; Falck et al., 1994; Hasson et al., 1994). The authors of those studies, however, speculate that implementation and population issues may have affected outcome. Other studies attribute some of these negative findings not to poor case management interventions, but rather to methodological problems in the evaluations (Orwin et al., 1994).
Even in light of the implementation and methodological concerns about case management research, all the studies together with the findings of other addiction research suggest that case management can be an effective enhancement to intervention in and treatment of substance abuse. This is especially true for clients with other disorders, who may not benefit from traditional substance abuse treatments, who require multiple services over extended periods of time, and who face difficulty gaining access to those services.
In addition, research suggests two reasons why case management may be effective as an adjunct to substance abuse treatment. First, treatment may be more likely to succeed when “drug use is treated as a complex of symptom patterns involving various dimensions of the individual's life” (Inciardi et al., 1994, p. 146). Case management focuses on the whole individual and stresses comprehensive assessment, service planning, and service coordination to address multiple aspects of a client's life. Second, retention in treatment is associated with better outcomes, and a principal goal of case management is to keep clients engaged in treatment and moving toward recovery and independence (Institute of Medicine, 1990). Studies looking at treatment retention and case management posit a positive relationship between the two (Siegal, 1997; Rapp et al., in press).
Case management's ambitious scope is one of the reasons its effectiveness is difficult to measure. Ashery and others have recommended that practitioners in the field maintain reasonable expectations for case management, pay attention to the implementation of programs, and understand the enhancing or limiting factors of the particular service context in which the case management programs are implemented (Ashery, 1994). The field should consider not only how to best research case management but what to expect from it.
In order for substance abuse programs to ascertain if case management works, the program and its various stakeholders (including funding and regulatory agencies) must specify and measure outcomes they regard as indicators of success.
This section presents options for basic evaluative methods, including documentation of the case management program's progress and measurement of system and individual client outcomes. It concludes by identifying the data needs of various stakeholders. Whether an evaluation is conducted internally by agency personnel, or by experts hired from outside, front-line case managers are the key source of information.
In documenting a case management effort, it is important to start with benchmarks—expectations that are made concrete as measurable statements (e.g., “case managers spend 60 percent of their time in face-to-face contact with their clients”). Some of the sources that programs can use to establish benchmarks include
Policy and procedure manuals
Federal, State, and local case management standards
Agency case management program descriptions and mission statements
Literature on program models (if the program under evaluation is a replication)
If no written manuals or protocols are available, or if it is clear that the program has drifted from its original design, the program managers and staff may use a consensus-development process to arrive at benchmarks.
Once the process benchmarks are defined in measurable terms, the next step is to develop and implement a method for measuring practice—to answer the question, “What are case managers doing and how does their practice conform to the benchmarks?” One approach is to maintain a simple staff log that measures case managers' activities by contact. The information should be comparable to the benchmarks and brief enough to ensure compliance and quality of data. Staff log instruments such as the one used by John Brekke and his colleagues (Brekke, 1987) have been widely adapted and used in the mental health field. They usually record the client's name, location of the contact, duration of the contact, activity, and whether other individuals participated (e.g., staff of other agencies or family members). The brevity and frequency of case managers' contacts with clients makes this measure extremely burdensome, and as a result many programs use time-limited or sampling measures (for example, over a two-week period) to get a “snapshot” of activities.
If time and resources permit, it may be valuable to use several methods of documentation to compare their usefulness and sensitivity. Other methods and purposes include
Reviews of case manager client records (to evaluate how service planning and referrals adhere to protocols and procedural expectations)
Interviews or surveys of case managers or clients and their family members (to collect information on activities in which case managers engage, to gauge how clients' and case managers' views of those activities differ)
Analysis of data from the agency's management information system (to examine patterns on type, number, and duration of case manager contacts with different target populations)
In addition to using multiple methods of documentation, it is important to review case manager activities over time because programs may drift from innovative to familiar patterns of service delivery. In addition, the timing of data collection is crucial. New programs need time to stabilize, and new staff members need a period of orientation before a true picture of program activities can be established.
Evaluators can use a key informant survey to examine the operations of a program's case management activities. The survey is a fixed series of questions about the functioning of both the case management program and the system of care and is administered to a variety of stakeholders in the community. Different stakeholders are identified by each agency, depending on its particular case management model and the system of care within which it works. Appropriate stakeholders may include, but are certainly not limited to
Staff from other substance abuse and human service agencies, homeless shelters, and hospital emergency rooms
Clients and their family members
Criminal justice and law enforcement personnel
Survey participants might be asked about their awareness of case management services, their use of these services, types of ongoing contact with the case management program, and their perception of the impact of these services on the community. To ensure a cross section of informed opinion at various points in time, all stakeholders are asked the same questions, and the survey is repeated at several intervals. Such surveys have been used to evaluate systems change in the mental health field (Morrissey et al., 1994) and could be adapted for use in case management programs.
Knowing how clients perceive the services they receive is essential to evaluative activities. One can argue that satisfaction with service is related to treatment retention. It is also important to know whether the service provider—in this instance the case manager—and client share a common view of the services being offered and their benefits. For example, did the client feel that the case management services actually led to needed resources? Other questions might focus on client perceptions about those providing the service: Did the case manager understand their needs and have the skills and experience necessary to help them accomplish their goals?
Such process data have direct utility for program management and development. They may help programs with defining staff training needs and assuring that the needs of the population they are working with are being addressed. Such data are also quite useful for those who have the responsibility for funding programs.
Many programs in the managed care environment control access to services through what is called “case management,” in which gatekeeping procedures are used to limit clients' use of expensive services such as hospitalization and residential treatment. These programs may be particularly interested in measuring system-level outcomes to see whether case management has a systemic effect on the delivery of substance abuse and allied services (e.g., change in patterns of service utilization or costs). Thus, a net reduction in the number of inpatient admissions for substance abuse treatment would, by itself, be defined as a positive outcome. This, of course, may not reflect the needs of all clients.
If the goal is preventing clients from “falling through the cracks” between discharge from detoxification and entry into outpatient substance abuse treatment, a system-level outcome might be measured by continuity of care. Greater continuity could be defined as fewer clients with no outpatient treatment episode after a detoxification discharge, patterns showing shorter periods of time between detoxification discharge and outpatient treatment admission, and fewer people with “revolving door” detoxification admissions. Another case management program may aim for increased access to care for certain target populations (for example, cocaine-abusing pregnant women). In this instance, it would be useful to compare the number of admissions in the target population to all admissions during a specified time period.
In order to measure most system outcomes, it is necessary to track clients within a comprehensive service agency and, if a program's mandate includes managing care across a network of agencies, to gather data on encounters and costs and analyze them. Access to a computerized management information system (MIS) is essential for complete analyses. Although these systems vary widely in their level of sophistication, for this purpose, one must be able to document more than units of service information and should be able to link encounter, claims, and cost data and produce information quickly and easily. Over a period of time, a comprehensive MIS tracks changes in patterns of service utilization and changes in costs, which gives the agency information crucial to management and planning. For example, an MIS that combines utilization and cost data could help identify high utilizers for a program that focuses on clients who use numerous or expensive services. A later section in this chapter describes how a program can evaluate and enhance its MIS system.
While most would agree that “evaluation” is generally worthwhile, there is considerably less agreement about the measurement and documentation of specific outcomes for individual clients. When trying to evaluate case management in an ongoing service agency setting, additional challenges—conceptual, methodological, and ethical—are posed. The field has seen a long-standing and often strident debate about what kinds of outcomes should be measured. Some claim a single measure such as sobriety or complete abstinence from any drug use is the only meaningful measure of treatment success. Others assert that treatment success is most appropriately measured by a constellation of factors, including diminished alcohol and/or other drug use, improved family functioning, improved occupational functioning, less deviant and/or criminal activity, fewer contacts with the criminal justice system, and improvement on a range of psychological variables. The debate will continue. In the meantime, programs should carefully consider treatment objectives to articulate and then operationalize those outcome variables they want to measure.
Another significant complication arises when trying to evaluate case management activities and client outcomes. A program must be able to articulate the role of case management and how it meshes with other program activities. However, when “standard” client outcomes—such as reduced substance use or fewer contacts with the criminal justice system—are measured, it is very difficult to separate the effects of substance abuse treatment activities from the effects of case management activities.
Finally, conducting research in community-based treatment/service organizations presents significant challenges. Experimentation, that is, comparison and control, is at the heart of any scientific research study. One group—typically defined as the “experimental group”—receives one kind of treatment and the control group does not. The two groups are then compared, and conclusions can be reached about the efficacy of the treatment. However, in the context of community-based treatment, a potentially beneficial service like case management cannot be withheld from some clients. This makes it extremely difficult to definitively attribute specific client outcomes to case management or some other service.
The types of data required for an evaluation of case management, how the data are collected, and the manner in which data are put to use vary among different stakeholders. It is important to understand the types of data that various stakeholders need to evaluate the program. Structured feedback loops should be established to ensure that the data gathered are returned to various stakeholders in some meaningful way so that they have an impact on shaping future program development (and future data needs). One of the benefits of the case management approach is that it can be adapted to meet the sometimes contradictory needs of the various stakeholders.
Although the data needs of case managers may vary from agency to agency, rapid access to data in three particular areas is critical:
Information about clients currently on the caseload (roster management), including outcome data so case managers have feedback on their performance
Data that allow case managers to track clients through various services
Data that produce “flags” for follow-up letters, aftercare, and other time-sensitive functions
In addition to these elements, case managers with gatekeeping or budgeting responsibility need overall service utilization and cost figures by client in order to manage services within a budget. To evaluate process, case managers need access (preferably computerized) to referral networks, bed allocation systems, progress notes, and data related to the daily conduct of their jobs. In terms of outcome data, case managers may want rapid access to client status, especially if it would prompt additional efforts.
Program managers must ensure that the data collected reflect the program mission and facilitate the program's management. While the case manager focuses on individual clients, the program manager analyzes data elements to see patterns and to flag and investigate “outliers”—those who deviate drastically from the statistical norms of the population.
The initial data needs of program managers reflect concerns with concrete aspects of program operation. To program managers, case management essentially begins when the phone rings, and therefore, their data needs are filled by asking the following basic questions:
How many inquiries are we getting about services?
Are we getting clients?
From what area are our clients?
Are clients entering care once they make contact?
Are we responsive to clients' needs from first contact forward?
Is the type of client changing?
In addition to collecting these initial data, program managers must be able to track clients through their services so they can decide how to alter service provision. Important questions include
Who is in what level of care at what time?
How does the service fit with their treatment plans?
Is the program meeting clients' different cultural needs?
Who is dropping out, and why?
What service not currently provided is requested most frequently?
How much money is being spent on a particular service?
Other questions relate to the program manager's administrative functions, including
What are the case managers doing? What are their caseloads?
What are the results of internal monitoring?
Are we reaching the target populations?
Are clients retained at the appropriate level of care?
Community policymakers may be local government officials, members of community coalitions, representatives of local law enforcement agencies, school board members, or other interested community-based stakeholders. Since they are not often directly associated with treatment programs, they may not have a very sophisticated understanding of program goals and may think of outcomes in terms of questions like “Is the client sober or not?” or “Is there less crime?” They tend to be less interested in improved scores on standardized measures of client functioning than in easily defined and observable outcomes that affect the community, principally
Taxes—Reducing costs to taxpayers in the areas of incarceration, unemployment, and welfare enrollment and reducing costs of case management and substance abuse treatment by substituting a costly treatment with a less expensive one
Safety—Reducing neighborhood crime and the number of homeless persons loitering in business districts
Social costs—Increasing the number of substance abusers who are working and improving care for children of substance abusers
Directors of State substance abuse agencies value data elements that describe the overall accessibility, quality, and cost of the substance abuse treatment system. In addition, these directors require data to track and contain the growth of Medicaid and public sector behavioral health care expenditures, to put managed care systems in place, and to evaluate the effect of managed care (including the provision of case management) on the delivery of behavioral health care services.
Key data elements that State directors often want to see in evaluation efforts include
Patterns of service utilization and costs, including the use of public hospital and residential treatment centers
Numbers of clients working and withdrawing from welfare and Medicaid
Numbers of clients avoiding prison, reducing child welfare cases and costs, and reducing food stamp usage
Numbers of appeals and grievances by clients
Number and characteristics of substance abuse patients accessing other publicly funded social services
Increasingly, State directors of substance abuse agencies are becoming less isolated and are beginning to look for opportunities to exchange data among previously independent departments (e.g., mental health departments, Medicaid offices, and criminal justice offices). Some State agencies share access to statewide data sets. In addition, the movement toward managed behavioral health care has prompted more integration of data between State Medicaid offices and State substance abuse and mental health authorities.
Third party payers such as insurance companies need data that justify case management as a cost above and beyond the direct costs of treatment services (see Chapter 6). In addition, when case management is used to coordinate care, third party payers want to know whether clients are receiving the right services, at the right level of care, and in the right sequence, and to ensure that clients who are no longer in need are no longer receiving services. To that end, important data elements include
The severity of the client's illness
Assignment to levels of care
Patterns of service utilization
Use of free self-help or volunteer organization services
Urinalysis results, use of other drugs, and scores on standardized outcome indicators
Clients and family members may serve on advisory or governing boards of local programs or may be involved in family or peer support groups within the community. They may use outcome data, especially results of client satisfaction surveys, to change programs and policies or to choose services and providers. They may be less interested in patterns of service utilization or standardized scores on outcome evaluations than in how the system functions from the user's perspective. In fact, clients might consider a program successful if it is supportive, reliable, and easily accessible, as opposed to “efficient.”
Data elements important to clients and family members include
The availability and accessibility of services
The freedom of choice (of services and providers) that the system allows
The use and effectiveness of the appeals and grievance process
The influence of input from consumers and family members
Effectiveness of treatment
Acceptability of treatment among the targeted populations
Specifically, clients seek answers to the questions
Am I getting the right services, in the right setting?
Are there systems I can access myself?
How appropriate is my care?
The management information system contains all this information and allows stakeholders to use it. Managed care has provided the behavioral health care field with an example of how to manage far-flung data on clients.
One evaluation task for local programs is determining how to use data already routinely collected by a statewide MIS or managed care company-based MIS, saving the program from duplicating primary data collection. Another important task is to develop or enhance program-level MIS that track data the program needs locally, integrate with other computer-based or paper-based systems, and supply data required by third party payer and governmental bodies. All staff members of a specific program should be stakeholders in the MIS, which increases both system accuracy and the likelihood that a broad array of staff members will use it. If an agency does not have the resources to develop a sophisticated system, it should be able to automate at least a minimum amount of client information through commercially available software.
Local programs that are part of a managed care network undoubtedly will be included in a larger MIS sponsored by the umbrella provider. Providers who are not part of these networks may need to assess their readiness to take on managed care activities by evaluating their current MIS capabilities. Today, it is critical that an MIS be designed with the data requirements of managed care organizations in mind. The following guidelines, adapted from a Federal technical assistance publication, may help a program determine whether its existing MIS is sophisticated enough to support managed care operations. A program's MIS will suffice if it does each of the following:
Retrieves patient information online or in less than an hour
Cross-matches client records, use of services, and financial and insurance information
Permits individual inquiries from managed care organizations
Produces information that is used by clinicians, supervisors, and managers
Integrates information from other programs and sites
Allows client and service information to be reported to all major payers
Generates patient invoices (CSAT, 1995d)
An existing MIS that can perform all of the above functions will likely support managed care and program demands; if it cannot, the program needs to strengthen deficient areas. Changes and advancements in data collection and access to patient information must be accompanied by appropriate protections for client confidentiality.
Research focused on case management in the substance abuse field is limited and offers many opportunities for local substance abuse programs to make significant contributions to the field. Suggested directions for future research include the following:
Key ingredients of successful programs, especially for hard-to-reach populations
Relative cost-effectiveness of particular case management models, including cost outcome results within systems incorporating full parity of substance abuse with other health care, outcome results when a full continuum of care is available to patients, and outcome results associated with use of standardized guidelines for placement, continued stay, and discharge for substance abuse patients
Improved methodology to investigate research questions in “real world” settings
Development of brief versions of valid and reliable research outcome instrumentation
The effect of particular forms of case management on societal costs of substance abuse and its treatment
Cost shifting among health, behavioral health, criminal justice, and other systems that can be accessed by the target population
Creative ways to use secondary data sets (such as Medicaid and Medicare) to determine trends and patterns of care
Research questions from broader sociological or multi-disciplinary perspectives
This is an open-access report distributed under the terms of the Creative Commons Public Domain License. You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.
