Development of the First Draft
Building upon the robust groundwork of the original OPOC-MHA for Registered Clients (Rush et al., 2014), the development of this new instrument prioritized comparability. The team aimed to maintain consistency in key constructs, terminology, and even specific items where appropriate, ensuring a familiar framework across versions. Mirroring the success of the main tool, the goal was to create a multi-dimensional instrument. This approach focuses on distinct sub-domains of the program experience, balancing thoroughness with practical completion time and minimizing respondent burden. This design choice differentiates the Ontario Perception Of Care Tool For Mental Health And Addiction from simpler, shorter tools—often developed locally by individual mental health and addiction organizations—that may lack broader applicability (Jiang et al., 2019). Instead, it aligns more closely with comprehensive instruments like the multi-dimensional Brazilian Mental Health Services Family Satisfaction Scale (Bandeira et al., 2002, 2011). A key objective was to ensure the tool’s relevance and applicability across both mental health and substance use service settings.
Table 1: Pilot sites and programs involved in testing the Ontario Perception of Care Tool for Mental Health and Addiction, showcasing the diversity in service types and client populations.
In June 2017, the project team collaborated with the Ontario Family Caregiver Advisory Network (OFCAN) to assess the viability of a new tool tailored to the needs of families and caregivers. The vision was to empower them to contribute feedback on the treatment and support their loved ones received. This collaboration solidified a commitment to create a standardized method for gathering family and caregiver feedback across Ontario’s mental health and addiction system. The initial step involved collecting a range of family caregiver feedback surveys currently utilized within Ontario’s mental health and addiction programs, many of which were represented by OFCAN. These surveys, alongside the registered client version of the OPOC-MHA, were cross-referenced to identify recurring themes. These common themes became the foundation for defining domains within the new tool – the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers. Through iterative discussions, the following core sub-domains were established for the initial draft:
- Caregiver Involvement and Engagement
- Caregiver Education and Support
- Caregiver Personal Support
- Environment
- Perception of Loved One’s Care
- Overall Experience
Individual questions within these domains were formulated based on the compiled sample of family caregiver feedback surveys and the OPOC-MHA. Particular emphasis was placed on: (1) item focus and wording to specifically capture perception of care rather than general client satisfaction, (2) the practical utility of responses for informing quality improvement initiatives, (3) the tool’s broad applicability across diverse treatment contexts within the mental health and addiction sector, and (4) ensuring comparability with the original OPOC-MHA where appropriate.
This process yielded a comprehensive set of 38 items, excluding demographic questions designed to categorize and compare responses from different caregiver subgroups. The selection of demographic items was intentionally postponed to a later phase to ensure close alignment with those used in the OPOC-MHA for Registered Clients, facilitating comparative analysis.
Stakeholder Consultation
To ensure the tool’s relevance and practicality, three focus group sessions and four key informant interviews were conducted with stakeholders deeply involved in Ontario’s mental health and addiction landscape. The focus groups included representatives from hospital-based service providers (n=5 organizations), community-based service providers (n=4 organizations), and OFCAN. Participants in these sessions and interviews provided invaluable feedback on the relevance and applicability of questions within each domain of the Ontario Perception of Care Tool for Mental Health and Addiction. They also assessed question readability and offered crucial insights into the tool’s potential implementation. Feedback from these consultations directly informed adjustments to the survey language, ensuring it better resonated with the perspectives of family members and caregivers. For instance, language concerning education and support was refined to more accurately reflect caregiver experiences, and descriptive explanations for each domain were incorporated into subsequent versions of the tool. Consultations also reinforced the importance of including questions addressing the process of obtaining consent for caregiver involvement in client care.
Pilot Testing
A three-month pilot phase, spanning June to September 2018, was crucial for gathering further feedback and understanding the initial successes and challenges of implementing the new Ontario Perception of Care Tool for Mental Health and Addiction across a diverse array of publicly funded mental health and addiction agencies in Ontario. Specific feedback was collected from both caregivers and service providers regarding the survey questions themselves, as well as the support service providers might need for widespread implementation. Insights from the pilot directly shaped final modifications to the tool and guided the implementation team in developing a targeted implementation strategy specifically for the OPOC-MHA for Caregivers.
Table 2: Key feedback themes from pilot site leads on the Ontario Perception of Care Tool for Mental Health and Addiction, highlighting both strengths and areas for improvement in the tool’s design and implementation.
A geographically diverse group of agencies, offering a wide spectrum of services, participated as pilot test sites. Each site designated an OPOC-MHA lead to serve as a point of contact with the project team throughout the pilot. These leads were responsible for identifying suitable programs within their organizations for pilot participation and recruiting 20 to 30 caregivers, supporters, or family members of clients. The focus was on caregivers not formally enrolled as service recipients themselves. Participating sites administered the survey over a 4- to 6-week period. Following the pilot, site leads shared their experiences with the tool, focusing on language clarity, readability, caregiver experience, and staff experiences in supporting survey implementation. A concise 10-item feedback questionnaire was also administered directly to participating caregivers immediately after they completed the tool. This questionnaire assessed the clarity of question wording, ease of understanding, and how well items reflected their service experiences. Although all pilot sites had prior experience with the standard OPOC-MHA for Registered Clients, site leads also attended a virtual introductory session about the caregiver tool and the pilot process. They were also provided with ongoing support from an implementation specialist from the Provincial System Support Program (PSSP) at CAMH. The implementation specialist assisted site leads in creating an implementation action plan for the caregiver version, designed to complement their existing approach for the OPOC-MHA. Pilot site leads were informed that their anonymized, program-level data would be summarized and returned to them at the pilot’s conclusion.
Twelve distinct programs across seven agencies agreed to participate in the pilot. At the pilot’s conclusion, nine programs from five agencies submitted completed surveys. Two community-based mental health agencies did not submit data, though one participated in a debrief interview to discuss implementation barriers and survey feedback. As detailed in Table 1, the pilot sites represented a diverse cross-section of Ontario’s mental health and addiction agencies, encompassing various program types (hospital, community-based, crisis, supportive housing) and client populations (youth, family, seniors, inpatient/outpatient).
Data collection occurred over a six-week period in June/July 2018. The method and timing of tool administration varied across agencies. Approaches included offering the survey at program completion, retrospectively contacting caregivers via email, and conducting a “blitz” where caregivers of all clients seen during the pilot period were offered a survey. The blitz approach allowed engagement of participants at different stages of their program involvement. One pilot site scheduled time during the final session of a family member and caregiver group program for on-site questionnaire completion. Depending on agency resources, sites could offer either paper or web-based versions of the questionnaire (using www.surveymonkey.com). Some sites offered both options. Paper survey responses were subsequently entered into Survey Monkey by agency staff. No incentives were offered to survey respondents.
Pilot Site Lead Debrief Interviews
Following the pilot testing phase, pilot site leads participated in 30-minute telephone interviews to share their experiences with the questionnaire and on-site implementation processes. Five pilot sites participated in phone interviews, and one provided written feedback. Debrief interviews took place in the two months following pilot testing. Some site leads included additional staff in these calls to provide staff perspectives. Using a semi-structured interview guide, participants were asked about the tool itself (e.g., language, readability, length, suitability for their caregiver population) and the implementation process (e.g., differences from their OPOC-MHA for Registered Clients implementation, survey introduction and facilitation methods, successful aspects of their approach). Sites were also asked about factors influencing caregiver participation rates.
As outlined in Table 2, debrief interviews with pilot sites revealed both strengths and challenges associated with the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers. A recurring concern, noted in both debrief interviews and stakeholder consultations, was the use of the term “loved one.” Tool developers explored alternative terms to replace “loved one,” but a consensus on a more universally accurate and inclusive term for the caregiver-client relationship was not reached. Consequently, organizations using this version of the OPOC-MHA need to be prepared to address potential caregiver questions regarding the term “loved one.” Multiple sites commented on the perceived irrelevance of certain items. However, a review of pilot data with one site during the debrief interview revealed that the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers had been administered in a program not ideally suited for the survey, as respondents were primarily involved in family member programming. This misapplication resulted in a high number of “Not Applicable” responses. Crucially, when these responses were excluded from the data, the frequency of “Not Applicable” responses significantly decreased. This finding suggests that when implemented with the appropriate population, the survey questions were highly relevant to caregiver experiences. This valuable feedback underscored the need to refine implementation strategies to ensure appropriate tool application across the province.
Pilot Results
A total of 84 participants completed the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers questionnaire across the five pilot agencies. Table 3 provides respondent demographic information. Pilot agencies employed various administration methods, including paper and electronic formats via Survey Monkey. Of these 84 responses, 57 participants also completed the 10-item feedback survey about the OPOC-MHA for Caregivers tool. Data from the feedback survey were analyzed using Microsoft Excel. Qualitative data were analyzed using open-coding techniques by study authors to identify strengths and areas for improvement for the OPOC-MHA for Caregivers. The following results encompass both quantitative and qualitative feedback on the survey. Regarding the extent to which the survey covered important areas of their caregiver experience, 11 (22.5%) strongly agreed, 31 (63.3%) agreed, five (10.2%) disagreed, and two (4.1%) strongly disagreed. Written feedback concerning missing items included suggestions for an “I Don’t Know” option (e.g., “Could have had a category: Not Sure”) for situations where caregivers lacked direct knowledge or their loved one’s treatment details were confidential. Three respondents highlighted the need to assess difficulties in finding appropriate treatment (e.g., “The most difficult aspect is finding the right treatment”; “I did not know where to get help for my son for almost a year”).
Table 3: Demographic characteristics of respondents who participated in the pilot testing of the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers, providing insights into the caregiver population surveyed.
According to respondents, 47 (94.0%) agreed or strongly agreed that the survey questions were easy to understand and clearly worded. When asked about confusing questions, two respondents suggested explanations for sexual orientation in the demographic section. Another respondent noted that while questions weren’t confusing, they desired an opportunity to explain “disagree” responses. Among respondents with survey questions, all received clarification from agency staff. Regarding survey administration preferences, 18 (45.0%) preferred online completion, while 22 (55.0%) preferred paper.
A primary theme emerging from qualitative feedback was caregivers’ lack of awareness or involvement in their loved one’s service experience. Consequently, many items seemed inapplicable, reinforcing the need for an “I Don’t Know” response option. Consistent with pilot site debrief feedback, many respondents indicated that numerous questions were irrelevant because their loved one was not currently receiving treatment at the agency where they completed the survey. This highlighted the critical need to improve the program selection process to ensure appropriate tool application prior to widespread implementation of the Ontario Perception of Care Tool for Mental Health and Addiction.
Final Tool Revisions
Based on pilot testing results and feedback survey themes, the tool underwent a final set of revisions. A significant change was the addition of an “I Don’t Know” response option, prompted by caregiver and agency staff feedback concerning items that might be relevant but unanswerable due to lack of caregiver involvement or awareness (e.g., “You need to add “I don’t know” as an option”). This option was added to items where respondents might reasonably lack knowledge. These items were then grouped into a new domain titled “Perceptions of My Loved One’s Care.” Furthermore, an “I Don’t Know” option was included in the residential/inpatient domain, acknowledging caregiver uncertainty about their loved one’s inpatient/residential experiences, especially if they were not directly involved in these aspects of care. All items retained a “Not Applicable” option. The project team determined that differentiating between “I Don’t Know” and “Not Applicable” could provide valuable insights into caregiver involvement, useful for quality improvement efforts related to the Ontario Perception of Care Tool for Mental Health and Addiction.
In response to debrief interview feedback indicating tool administration to ineligible respondents at one site (e.g., “The survey did not really apply to me because my loved one did not receive treatment here”; “The majority of this survey does not apply to caregivers with loved ones who are not receiving treatment”; “Loved one did not attend this facility”), further analyses focused on survey results from that specific program. As this program administered a high volume of surveys to ineligible participants, the number of “Not Applicable” responses was significantly inflated but decreased dramatically when analyzing data solely from eligible respondents. Therefore, the OPOC-MHA development team decided to retain all items despite feedback regarding inapplicability. See Fig. 1 for a sample item based on eligible respondents. Finally, in response to pilot site and caregiver feedback, a substantial portion of the demographic questions were revised for the final version of the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers. The updated demographic questions align with other OPOC-MHA versions and were adapted from the We ask because we care: The Tri-Hospital + TPH health equity data collection research project (Wray et al., 2013), which aimed to establish a consistent, evidence-based approach for collecting socio-demographic information in healthcare settings. The questionnaire continues to include demographic items about both the caregiver and their loved one.
Figure 1: Example survey item from the Ontario Perception of Care Tool for Mental Health and Addiction, demonstrating the question format and response options available to caregivers.
Given that mental health and addiction service needs and outcomes can vary based on social and demographic indicators such as gender, age, racialized status, and sexual orientation, the demographic items aim to identify potential inequities in access, quality, and satisfaction of services for both caregivers and their loved ones. Consequently, data from the Ontario Perception of Care Tool for Mental Health and Addiction for Caregivers can be used to inform and address barriers specific to population groups, guide quality improvement interventions, and track progress over time.
Final Version
The final version of the Ontario Perception of Care Tool for Mental Health and Addiction contains 41 items, including six specific to inpatient or residential programming, and 16 demographic items pertaining to both the caregiver and their loved one receiving services. These items are organized into seven quality domains: Caregiver Involvement and Engagement, Caregiver Education and Support, Caregiver Personal Support, Environment, Perception of My Loved One’s Care, Overall Experience, and Residential/Inpatient.
