Abstract
Introduction Ensuring the well-being of residents in long-term care facilities requires sustained health initiatives, particularly in areas like smoking cessation. While numerous frameworks address program sustainability, practical assessment tools, especially those tailored for specific health challenges like smoking in long-term care, remain scarce. This article introduces the Program Sustainability Assessment Tool (PSAT) as a reliable instrument adaptable for evaluating and strengthening the capacity for sustainability in smoking cessation programs within long-term care settings. This is crucial as effective smoking cessation programs can only deliver long-term benefits if they are successfully maintained over time. Therefore, a robust Smoking Assessment Tool For Long Term Care programs must also consider the program’s long-term viability.
Methods This study, originally designed for broader public health program assessment, employed measurement development to validate the PSAT’s reliability. Data from 592 program managers and staff across 252 public health programs were analyzed. These programs, spanning state and community levels and focusing on chronic diseases including tobacco control, provided a diverse dataset to assess the PSAT’s applicability and adaptability to various program contexts, including the potential for smoking cessation programs in long-term care.
Results The refined PSAT comprises 40 items across 8 sustainability domains, each with 5 items. Confirmatory factor analysis affirmed the robust fit of the 8-domain structure. Subscales demonstrated excellent internal consistency, with an average Cronbach’s α of 0.88 (range: 0.79–0.92). Initial validation analyses suggest PSAT scores correlate with significant program and organizational characteristics, indicating its potential utility for diverse applications, including assessing the sustainability of smoking assessment tool for long term care programs.
Conclusion The PSAT is a reliable assessment instrument adaptable for measuring the sustainability capacity of health programs, including smoking cessation initiatives relevant to long-term care. Its structured approach can aid researchers, evaluators, and long-term care facility managers in planning, strengthening, and sustaining effective smoking cessation programs. By considering sustainability from the outset, facilities can ensure that residents benefit from continuous and effective support in quitting smoking, underscoring the importance of incorporating sustainability assessments into the implementation of any smoking assessment tool for long term care.
Introduction
The field of dissemination and implementation science emphasizes translating research into effective programs and practices (1). While implementation is crucial, the long-term maintenance of these programs, particularly in the context of public health and vulnerable populations like those in long-term care, is equally vital. Smoking remains a significant health concern in long-term care facilities, necessitating effective and, crucially, sustainable cessation programs. The benefit of any smoking assessment tool for long term care hinges on the ability of the programs it informs to endure and deliver lasting impact. Rapid shifts in funding, policy priorities, and organizational resources can challenge the longevity of even the most effective interventions. For residents of long-term care facilities, consistent access to smoking cessation support is paramount for their health and well-being. Therefore, understanding and fostering program sustainability is essential to maximize the return on investment in public health research and program development (3).
The concept of program sustainability has been explored across various sectors, including business, healthcare administration, and public health (4). However, a unified definition and consensus on key determinants of sustainability remain elusive (5–7). While conceptual frameworks for sustainability exist, the translation of these frameworks into practical, validated assessment tools is lacking. Specifically, in the context of smoking assessment tool for long term care and related cessation programs, the need for robust sustainability measures is pressing. Existing literature highlights numerous frameworks, but few have yielded reliable and valid tools for assessing sustainability in public health settings (5). Indeed, many identified frameworks lack psychometric validation, underscoring the gap between conceptual understanding and practical measurement. The absence of such tools hinders the ability of long-term care facilities to proactively plan for and ensure the endurance of their smoking cessation programs.
This article introduces the Program Sustainability Assessment Tool (PSAT), grounded in a comprehensive program sustainability framework (4). The PSAT enables programs to evaluate their sustainability capacity across eight key domains. Developed and tested across diverse public health programs at community and state levels, the PSAT offers a robust and adaptable tool for various settings, including long-term care facilities aiming to implement and sustain effective smoking assessment tool for long term care and cessation programs. A reliable tool like the PSAT empowers facilities to engage in proactive program planning and improvement, enhancing the likelihood of long-term program success. Furthermore, dissemination and implementation scientists can utilize the PSAT to investigate the factors that contribute to the sustained impact of evidence-based programs in real-world settings, including the crucial area of smoking cessation in long-term care.
Methods
This measurement development study aimed to create a user-friendly, reliable instrument for assessing program sustainability capacity. The PSAT’s structure was derived from an established sustainability framework (4), guided by four core design principles: 1) brevity and ease of use; 2) applicability to programs of varying sizes (especially community and state-level initiatives); 3) relevance across diverse public health program types, including potential applications in clinical and social service settings like long-term care; and 4) utility as a scientific, evaluation, and program planning tool. These principles are particularly pertinent when considering the practical needs of long-term care facilities seeking to implement sustainable smoking assessment tool for long term care programs.
Initial Instrument Development
Building on prior literature reviews and concept mapping studies (4), a pilot version of the PSAT was developed. Concept mapping, a structured conceptualization technique, facilitates group-based development of frameworks for evaluation and planning (10). The pilot PSAT included 63 items across 9 sustainability domains: Political Support (later Environmental Support), Funding Stability, Partnerships, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, Public Health Impacts, and Strategic Planning. Each item assessed an element identified as relevant to sustainability through literature and concept mapping. A 7-point Likert scale (1 = “Little or no extent” to 7 = “A very great extent”) was used for responses.
The pilot study focused on instrument development rather than in-depth validation. However, limited data were collected for preliminary validation analyses. The PSAT’s primary goal is to accurately assess program sustainability capacity. Therefore, a valid instrument should yield scores reflecting program, agency, or organizational characteristics, rather than individual respondent traits. This is crucial when applying the PSAT to assess the sustainability of programs incorporating a smoking assessment tool for long term care, as the assessment should reflect programmatic factors, not just individual perceptions.
Data Collection
Data were collected using paper surveys at training events and online surveys via Qualtrics. Both survey formats were identical, comprising 63 items within the original 9 domains.
Participating Programs and Respondents
The project advisory group identified diverse initiatives and programs for PSAT pilot testing (Table 1). To ensure sample diversity, programs were selected from four chronic disease areas and two program sizes (state and community levels). Respondents from 252 programs completed the initial PSAT between October 2010 and October 2011. The dataset (n = 592; 386 complete surveys, 206 with minor omissions) included 494 online and 98 paper surveys. Ethics review board approvals were obtained. Participant numbers per program ranged from 1 to 15. Program foci included tobacco control (49.2%), obesity prevention (37.3%), diabetes (7.1%), and oral health (5.9%), with a small fraction covering multiple areas. The majority (63.3%) operated at the state level, 35.8% at the community level, and a small percentage at the national level. This diverse sample, while not specifically focused on long-term care, provides a robust foundation for assessing the PSAT’s generalizability and potential adaptation for evaluating programs such as those incorporating a smoking assessment tool for long term care.
Covariates
Program-descriptive variables were used as covariates in validation analyses: program level (state/community), program type (obesity prevention, diabetes, oral health, tobacco, multiple health topics), respondent role (manager/director, evaluator, etc.), years in program, and perceived sustainability. Perceived sustainability was measured using a 7-point Likert scale (“How would you rate the overall sustainability of your program?”). These covariates aimed to capture potential influences on program sustainability capacity, such as program level affecting resource access or partnerships. For programs utilizing a smoking assessment tool for long term care, factors like program type and level might similarly influence sustainability.
Analyses
Confirmatory factor analysis and psychometric analyses were conducted using the lavaan package in R. Confirmatory factor analysis is appropriate for testing hypothesized subscale structures (12,13). Initial analyses identified poorly performing items and tested the hypothesized domain structure. Poor items were characterized by low variability or poor subscale fit. After refinement, multiple-group confirmatory factor analysis tested for factorial invariance across program level and type (14), assessing whether the PSAT’s subscale structure remained consistent across program variations. Model fit was assessed using CFI, RMSEA, and SRMR (15). These rigorous analyses ensure the PSAT’s robustness and applicability across different program contexts, increasing confidence in its potential use for evaluating the sustainability of programs incorporating a smoking assessment tool for long term care.
Results
Instrument Improvement and Domain Structure
Item and confirmatory factor analyses led to the final PSAT structure. Table 2 shows improved psychometrics throughout development. The baseline model (no subscale structure) served as a comparison for pilot and final models. The pilot model (63 items, 9 subscales) was refined by removing 23 items and the Public Health Impacts subscale due to low factor loadings, poor variance, excessive missing data, and high subscale intercorrelation. Public Health Impacts was deemed a sustainability outcome rather than a capacity domain. The final PSAT has 40 items across 8 domains, each with 5 items, simplifying training and scoring.
Table 2 shows good fit for the 8-domain confirmatory factor analysis model, indicating the 40-item PSAT effectively measures 8 key sustainability domains. The final model had the lowest AIC, suggesting better data fit. RMSEA (0.066) was between good and acceptable, and SRMR was below 0.08, indicating good fit (16,17). Table 3 lists the 40 items in the 8 subscales with item-factor loadings. This refined structure enhances the PSAT’s usability and interpretability, making it a practical tool for assessing the sustainability capacity of diverse programs, including those incorporating a smoking assessment tool for long term care.
Multiple-group confirmatory factor analysis tested factorial invariance across program level and type. Results indicated significant differences between community and state programs (χ2 = 78.0, df = 32, P < .001) and marginally significant differences between program types (χ2 = 46.0, df = 32, P = .052). While not identical, item-factor loadings were similar across groups (Table 3). Average item-loading difference between program levels was 0.05, with only 2 of 40 items differing by ≥0.20. Large sample size may have detected small, statistically significant differences. Overall, factor loading patterns suggest the PSAT has a similar structure across different public health programs, reinforcing its adaptability for assessing the sustainability of programs in various contexts, including those centered around a smoking assessment tool for long term care.
Subscale Reliability
Subscale reliabilities (internal consistency) were excellent, especially given the small subscale size (5 items) (18) (Table 4). Average internal consistency was 0.88 (range: 0.79–0.92). Item loadings were consistently high within subscales, though Funding Stability had slightly lower loadings (Table 3). These high reliability scores indicate the PSAT’s consistent and dependable measurement of sustainability domains, crucial for its application in evaluating programs, such as smoking cessation initiatives in long-term care, and any associated smoking assessment tool for long term care.
Preliminary PSAT Results and Validation
Pilot data from 592 participants were used for exploratory descriptive and validation analyses. Average total sustainability score across 252 programs was 4.84 (range: 1.32–7.00, interquartile range: 4.17–5.58). Scores showed good range coverage, though very low scores were less frequent (Figure 1).
Figure 1. Density plot (frequency) of the variability of Program Sustainability Assessment Tool (PSAT) scores across 252 public health programs participating in tests of the PSAT. [A text description of this figure is also available.]
Analyses of relationships between overall sustainability scores and covariates showed significant associations with program type (F4,587 = 3.33, P = .01) and level (F1,590 = 70.6, P < .001), but not respondent role (F3,589 = 0.09, not significant). This provides discriminant validation evidence. Subscale scores varied by program level and type (Figure 2). These preliminary analyses suggest the PSAT can differentiate sustainability levels based on program characteristics, such as level or focus, indicating its potential to distinguish sustainability capacity among different types of programs, including those employing a smoking assessment tool for long term care.
Figure 2. Program Sustainability Assessment Tool domain scores by level of program and type of program among programs participating in tests of the tool. [A text description of this figure is also available.]
A construct validity analysis correlated PSAT scores with perceived program sustainability (n=205). PSAT scores showed moderate positive correlations with perceived sustainability: overall = 0.68, Political Support = 0.48, Funding Stability = 0.67, Partnerships = 0.44, Organizational Capacity = 0.58, Program Evaluation = 0.45, Program Adaptation = 0.32, Communications = 0.55, Strategic Planning = 0.63. Funding Stability and Strategic Planning domains were most strongly associated with perceived sustainability. This correlation supports the PSAT’s validity in measuring program sustainability as perceived by program staff, further suggesting its utility in assessing the sustainability of programs, such as those using a smoking assessment tool for long term care.
Discussion
The psychometric study results indicate that the PSAT is a reliable tool ready for use in assessing program sustainability capacity by researchers, evaluators, program managers, and staff. Confirmatory factor analyses demonstrate the PSAT’s ability to capture distinct program sustainability elements with just 40 items, aligning with prior conceptual work (4). High internal consistency scores for all 8 subscales, despite their small size, are noteworthy. Pilot data suggest no range restriction issues, with sustainability scores varying across the instrument’s intended range. This study advances dissemination and implementation science by providing a validated tool to explore factors influencing the maintenance of public health programs, including crucial areas like smoking cessation in vulnerable populations within long-term care.
The PSAT’s design prioritizes ease of use across diverse public health and social service programs, including long-term care settings implementing smoking assessment tool for long term care programs. Its brevity, consistent structure, and versatility enable use for program monitoring, evaluation, and strategic planning. A companion article (19) further details PSAT applications in community and public health programs.
Future research should explore the PSAT’s utility across broader contexts. While tested with diverse chronic disease programs, its application beyond this area, including in diverse long-term care settings, warrants investigation. Dropping the Public Health Impacts domain enhances the tool’s applicability to social service and clinical care programs, including health systems (20). A modified PSAT version with Environmental Support replacing Political Support was introduced in 2013. Further research should assess the PSAT’s utility across diverse fields and intervention types (21), particularly its effectiveness in evaluating the sustainability of programs incorporating a smoking assessment tool for long term care.
Further PSAT validation is also needed. While preliminary data link PSAT scores to organizational and program characteristics, deeper exploration is necessary. For instance, the finding that state programs scored lower than community programs on sustainability requires further investigation into underlying causes. Understanding sustainability differences across program types, including those focused on smoking assessment tool for long term care within long-term care facilities, is crucial.
The ultimate validation challenge is using the PSAT to predict and understand long-term sustainability outcomes (3). Program sustainability is vital for realizing the benefits of public health investments. The PSAT provides a reliable tool to advance understanding of factors enabling programs to sustain their impact, including the sustained effectiveness of smoking cessation programs and associated smoking assessment tool for long term care initiatives in long-term care settings.
Acknowledgments
This project was funded by the Centers for Disease Control and Prevention (CDC), Office on Smoking and Health, contract no. 200-2011-42065. The authors thank Monica Eischen and Shawna Shields of CDC’s Office on Smoking and Health and the project’s advisory committee for their guidance.
Author Information
Corresponding Author: Douglas A. Luke, PhD, Center for Public Health Systems Science, George Warren Brown School of Social Work, Washington University in St Louis, 700 Rosedale Ave, Campus Box 1009, St Louis, MO 63112. Telephone: 314-935-3794. E-mail: [email protected].
Author Affiliations: Annaliese Calhoun, Christopher B. Robichaux, Sarah Moreland-Russell, Washington University in St Louis, Missouri; Michael B. Elliott, Saint Louis University, St Louis, Missouri.
References
Tables
Table 1. Characteristics of Participating Programs in Tests of the Program Sustainability Assessment Tool
| Participating Initiatives and Programs | Program Level | Program Focus | No. of Programs | No. of Participants |
|---|---|---|---|---|
| Missouri Healthy and Active Communities grantees | Community | Obesity Prevention | 47 | 99 |
| Appalachia Diabetes Coalitions | Community | Diabetes | 31 | 31 |
| Missouri Tobacco Prevention and Cessation Initiative grantees | Community | Tobacco | 31 | 82 |
| Centers for Disease Control and Prevention (CDC), Division of Nutrition, Physical Activity, and Obesity grantees | State | Obesity Prevention | 50 | 114 |
| Missouri Council for Activity and Nutrition Coalition | State | Obesity Prevention | 1 | 8 |
| CDC, Fall Institute workshop grantees | State | Diabetes, tobacco | 21 | 24 |
| Missouri Tobacco Control Program | State | Tobacco | 1 | 11 |
| CDC, Office on Smoking and Health grantees | State | Tobacco | 53 | 142 |
| CDC, Office on Smoking and Health Sustaining States grantees | State | Tobacco | 4 | 46 |
| CDC, Division of Oral Health grantees | State | Oral Health | 13 | 35 |
| Total | 252 | 592 |
Table 2. Confirmatory Factor Analysis Results of Baseline, Pilot, and Final Program Sustainability Assessment Tool Instruments
| Phase | Subscales | Items | χ2/df | CFI | RMSEA | SRMR | AIC |
|---|---|---|---|---|---|---|---|
| Baseline | 1 | 63 | 15.3 | 0.58 | 0.102 | 0.087 | 114,884 |
| Pilot | 9 | 63 | 3.7 | 0.82 | 0.067 | 0.063 | 108,194 |
| Final | 8 | 40 | 3.6 | 0.89 | 0.066 | 0.055 | 69,518 |
Abbreviations: CFI, comparative fit index; RMSEA, root mean square error of approximation; SRMR, standardized root mean residual; AIC, Akaike Information Criterion.
Table 3. Item-Factor Loadings for Final Itemized Subscales of the Program Sustainability Assessment Tool (PSAT) and Confirmatory Factor Analysis for Program Level and Program Type for Programs Participating in Tests of the PSAT
| Subscale Definition and Items | Total Sample (n = 592) | Confirmatory Factor Analysis |
|—|—|—|—|—|
| | | Program Level | Program Type |
| | | Community (n = 212) | State (n = 380) | Tobacco (n = 301) | Nontobacco (n = 291) |
| Political Supporta: Internal and external political environments that support your program | | | | | |
| 1. Political champions advocate for the program. | 0.84 | 0.84 | 0.84 | 0.82 | 0.84 |
| 2. The program has strong champions with the ability to garner resources. | 0.81 | 0.74 | 0.82 | 0.82 | 0.80 |
| 3. The program has political support within the larger organization. | 0.72 | 0.77 | 0.68 | 0.69 | 0.75 |
| 4. The program has political support from outside of the organization. | 0.84 | 0.87 | 0.82 | 0.81 | 0.86 |
| 5. The program has strong advocacy support. | 0.74 | 0.68 | 0.75 | 0.66 | 0.82 |
| Funding Stability: Establishing a consistent financial base for your program | | | | | |
| 6. The program exists in a supportive state economic climate. | 0.61 | 0.44 | 0.66 | 0.57 | 0.66 |
| 7. The program implements policies to help ensure sustained funding. | 0.66 | 0.66 | 0.64 | 0.64 | 0.68 |
| 8. The program is funded through a variety of sources. | 0.61 | 0.59 | 0.63 | 0.62 | 0.60 |
| 9. The program has a combination of stable and flexible funding. | 0.77 | 0.72 | 0.80 | 0.80 | 0.75 |
| 10. The program has sustained funding. | 0.75 | 0.79 | 0.73 | 0.74 | 0.76 |
| Partnerships: Cultivating connections between your program and its stakeholders | | | | | |
| 11. Diverse community organizations are invested in the success of the program. | 0.77 | 0.76 | 0.77 | 0.77 | 0.77 |
| 12. The program communicates with community leaders. | 0.85 | 0.82 | 0.86 | 0.86 | 0.85 |
| 13. Community leaders are involved with the program. | 0.85 | 0.81 | 0.88 | 0.86 | 0.85 |
| 14. Community members are passionately committed to the program. | 0.78 | 0.74 | 0.78 | 0.78 | 0.78 |
| 15. The community is engaged in the development of program goals. | 0.78 | 0.77 | 0.78 | 0.77 | 0.79 |
| Organizational Capacity: Having the internal support and resources needed to effectively manage your program | | | | | |
| 16. The program is well integrated into the operations of the organization. | 0.77 | 0.68 | 0.80 | 0.75 | 0.79 |
| 17. Organizational systems are in place to support the various program needs. | 0.84 | 0.73 | 0.86 | 0.85 | 0.83 |
| 18. Leadership effectively articulates the vision of the program to external partners. | 0.81 | 0.78 | 0.78 | 0.84 | 0.79 |
| 19. Leadership efficiently manages staff and other resources. | 0.84 | 0.85 | 0.80 | 0.85 | 0.85 |
| 20. The program has adequate staff to complete the program’s goals. | 0.60 | 0.60 | 0.52 | 0.65 | 0.54 |
| Program Evaluation: Assessing your program to inform planning and document results | | | | | |
| 21. The program has the capacity for quality program evaluation. | 0.78 | 0.74 | 0.78 | 0.78 | 0.78 |
| 22. The program reports short-term and intermediate outcomes. | 0.82 | 0.78 | 0.83 | 0.82 | 0.81 |
| 23. Evaluation results inform program planning and implementation. | 0.89 | 0.90 | 0.89 | 0.93 | 0.86 |
| 24. Program evaluation results are used to demonstrate successes to funders and other key stakeholders. | 0.84 | 0.84 | 0.84 | 0.86 | 0.83 |
| 25. The program provides strong evidence to the public that the program works. | 0.80 | 0.72 | 0.80 | 0.76 | 0.82 |
| Program Adaptation: Taking actions that adapt your program to ensure its ongoing effectiveness | | | | | |
| 26. The program periodically reviews the evidence base. | 0.78 | 0.78 | 0.81 | 0.81 | 0.74 |
| 27. The program adapts strategies as needed. | 0.89 | 0.87 | 0.89 | 0.92 | 0.85 |
| 28. The program adapts to new science. | 0.86 | 0.84 | 0.88 | 0.90 | 0.81 |
| 29. The program proactively adapts to changes in the environment. | 0.89 | 0.89 | 0.88 | 0.90 | 0.88 |
| 30. The program makes decisions about which components are ineffective and should not continue. | 0.75 | 0.73 | 0.74 | 0.77 | 0.74 |
| Communications: Strategic communication with stakeholders and the public about your program | | | | | |
| 31. The program has communication strategies to secure and maintain public support. | 0.89 | 0.89 | 0.88 | 0.91 | 0.88 |
| 32. Program staff members communicate the need for the program to the public. | 0.86 | 0.85 | 0.84 | 0.85 | 0.88 |
| 33. The program is marketed in a way that generates interest. | 0.85 | 0.81 | 0.83 | 0.86 | 0.84 |
| 34. The program increases community awareness of the issue. | 0.83 | 0.74 | 0.81 | 0.85 | 0.81 |
| 35. The program demonstrates its value to the public. | 0.81 | 0.62 | 0.82 | 0.85 | 0.77 |
| Strategic Planning: Using processes that guide your program’s directions, goals, and strategies | | | | | |
| 36. The program plans for future resource needs. | 0.81 | 0.80 | 0.82 | 0.82 | 0.80 |
| 37. The program has a long-term financial plan. | 0.83 | 0.85 | 0.84 | 0.80 | 0.84 |
| 38. The program has a sustainability plan. | 0.82 | 0.83 | 0.81 | 0.77 | 0.84 |
| 39. The program’s goals are understood by all stakeholders. | 0.74 | 0.64 | 0.74 | 0.82 | 0.67 |
| 40. The program clearly outlines roles and responsibilities for all stakeholders. | 0.78 | 0.65 | 0.79 | 0.85 | 0.71 |
a This domain is now called Environmental Support.
Table 4. Subscale Reliabilities (Internal Consistency) for the Program Sustainability Assessment Tool
| Subscale | Cronbach’s α |
|---|---|
| Political Supporta | 0.88 |
| Funding Stability | 0.79 |
| Partnerships | 0.90 |
| Organizational Capacity | 0.87 |
| Program Evaluation | 0.90 |
| Program Adaptation | 0.91 |
| Communications | 0.92 |
| Strategic Planning | 0.88 |
a This domain is now called Environmental Support.
