Evaluation Activities
The goal of the ODS waiver evaluation is to be as comprehensive and useful as possible within practical constraints by following several principles:
- Analyze existing state administrative datasets where possible.
- Align measures with existing or expected future data requirements.
- Where necessary, collect new data while minimizing the burden on stakeholders.
- Provide results to stakeholders quickly to inform ongoing implementation efforts.
Quantitative methods will be used to analyze trends and the degree of changes over time, while qualitative methods will be used to help interpret quantitative data within the broader context of stakeholder perceptions.
Evaluation Plan
- UCLA Evaluation Plan (approved 06/20/2016)
Administrative Data
The evaluation will make use of various data sources including the California Outcomes Measurement System, Treatment (CalOMS-Tx), Drug Medi-Cal Claims, Medi-Cal Managed Care, Fee-For-Service (FFS) data, and client level-of-care data, as become available to researchers.
County Administrator Survey
UCLA conducts a survey of county substance use disorder (SUD) program administrators on an annual basis to obtain information and insights from all SUD administrators in the state. The survey addresses the following topics: access to care; screening and placement practices; services and training; quality of care; collaboration, coordination, and integration of services; and waiver implementation preparation/status, among others.
Provider Survey
UCLA is conducting surveys of providers in each waiver county throughout the state. Provider surveys are conducted at the care delivery unit level, referring to a treatment modality (e.g., inpatient, outpatient, methadone maintenance) at a specific site. Clinical directors are asked questions related to access (e.g., treatment capacity), quality (e.g., ASAM criteria, electronic health records) and coordination of care (e.g., partnerships with other treatment and recovery support providers, levels of integration with physical and mental health scare systems) in their treatment programs.
Client Satisfaction Surveys (Treatment Perceptions Survey; TPS)
The Treatment Perceptions Survey (TPS) is used to measure client satisfaction under the DMC-ODS waiver. As part of the waiver evaluation, counties are required to have their network of providers administer the client Treatment Perceptions Survey (TPS). The information collected is used to measure adult clients' perceptions of access to services and the quality of care, and may also be used by counties (and service providers) to evaluate and improve the quality of care and client experience.
County Beneficiary Access Line Secret Shopper
UCLA conducts secret shopper calls to evaluate access to counties' beneficiary access lines (BALs). The purpose of these calls is to verify that the requirement of having a phone number available to beneficiaries is being met by counties that have started providing DMC-ODS services. Initiation of these "secret shopper calls" occurs soon after the county's contract with DHCS is executed.
Stakeholder Interviews
Interviews are conducted with administrators from selected counties that are participating in the DMC-ODS waiver. The purpose of these key informant interviews is to continue to compile in-depth information on selected waiver topics, implementation lessons learned, promising strategies, and recommendations for improvement. The findings help to inform other counties' and the State's implementation efforts and to aid in the interpretation of the quantitative survey results and administrative data.