Development and validation of the Out-of-Hospital Adherence Questionnaire for Stroke Patients (OHAQ-SP)

Abstract

Background

The extent of rehabilitation of stroke patients outside the hospital is contingent upon their dedication to rehabilitative exercises, adherence to a suitable diet, and medication protocols. Therefore, multiple aspects of compliance must be assessed. Questionnaires are indirect measurements that are economical and easy to use. However, questionnaires should meet specific criteria to minimize errors and ensure reprehensibility of the results.

Objective

To develop a new questionnaire-based assessment tool for out-of-hospital treatment adherence in stroke patients and to evaluate the psychometric properties and feasibility of the questionnaire by conducting preliminary measurements in the stroke population. The questionnaire should have at least one validity measure and one reliability measure.

Methods

A preliminary "OHAQ-SP" was developed based on a literature review and two rounds of Delphi expert consultation. Between April 1 and July 2022, 308 stroke patients revisited the neurology departments of four Grade A tertiary hospitals in Shanghai using the convenience sampling approach. Various statistical methods, such as item analysis, correlation analysis, exploratory factor analysis, confirmatory factor analysis, and the internal consistency measure Cronbach’s α coefficient, were employed to assess the reliability and validity of the questionnaire.

Results

A total of 290 questionnaires were collected; the Cronbach’s α coefficients were 0.936, 0.862, 0.916, and 0.928 for the different sections, and the overall Cronbach’s α coefficient for the entire scale was 0.902. The results of the confirmatory factor analysis were CMIN/DF = 2.490, RMR = 0.05, GFI = 0.901, NFI = 0.927, IFI = 0.955, and RMSEA = 0.054. The standardized loading values varied between 0.717 and 0.916. The combined reliabilities of the four dimensions were 0.936, 0.864, 0.917, and 0.929, respectively. The average variance extraction values for the four dimensions are 0.676, 0.614, 0.611, and 0.652, respectively.

Conclusion

The OHAQ-SP is a reliable and accurate instrument for evaluating medication use, nutritional status, and behavior throughout the out-of-hospital rehabilitation phase.

Introduction

Stroke is a prevalent and severe neurological disorder. According to a report published by the World Health Organization in 2021, approximately 130 million individuals experience a stroke annually, leading to over 116,000 fatalities each year [1]. It ranks as the third leading cause of disability-adjusted life years globally, thereby imposing a substantial burden on healthcare systems [2]. Approximately 35% of stroke survivors do not achieve complete recovery, and 20% to 25% of these persons necessitate substantial support for mobility [3, 4, 5, 6]. Consequently, stroke patients require prolonged rehabilitation and care during the stable phase of their disease.

The Chinese Guidelines for Stroke Prevention and Control (2021 edition) outline a comprehensive framework for implementing standardized measures for primary and secondary prevention [5]. The guidelines underscore the significance of enhancing patient adherence through health promotion programs involving the patient's family. Adherence, as defined by the World Health Organization (WHO), refers to the extent to which an individual follows medical recommendations provided by healthcare professionals regarding diet, medication, and lifestyle changes [7]. Poor adherence is often linked to various factors including socioeconomic, treatment-related, individual patient-related, environmental, and healthcare system-related issues [8, 9]. Furthermore, multiple studies have demonstrated a clear and significant correlation between adherence and prognosis in stroke survivors [10, 11]. Therefore, it is essential to promptly and accurately assess the adherence of patients with stroke during the rehabilitation phase.

Numerous methodologies exist for assessing patient compliance, such as the universal compliance questionnaire, pharmaceutical compliance questionnaire, and functional exercise compliance questionnaire, among others [12, 13]. The more renowned questionnaires include the General Adherence Scale (GAS) and the Morisky Self-Report Adherence Questionnaire (MAQ), which are frequently utilized to evaluate the adherence of stroke patients. The GAS comprises five items intended to evaluate overall adherence in individuals with chronic conditions over a four-week duration [14], demonstrating strong stability, internal consistency (α coefficient = 0.81), and reliability (0.4) [15]. The scale is extensively utilized globally [16, 17, 18, 19]. The MAQ, comprising four measures, evaluates individuals' likelihood of discontinuing medication under specific conditions and their ability to adhere to specified treatment regimens [19]. This scale is user friendly, making it accessible to patients with varying levels of literacy. Its applicability is also versatile, and it has demonstrated strong reliability and validity (Cronbach's α = 0.76) when used among individuals with hypertension in China [20, 21, 22, 23].

At present, no approach exists specifically for evaluating adherence in stroke patients. In addition, the degree of recovery for stroke patients out of hospital is contingent upon their dedication to rehabilitation exercises, adherence to an appropriate diet, and compliance with medication protocols. Consequently, it is inadequate to evaluate out-of-hospital adherence in stroke patients using a singular assessment criterion. This study aimed to assess the accuracy of a non-hospital adherence questionnaire designed for Chinese stroke patients. The capability to measure identical constructs as other recognized assessments was evaluated, and confirmatory factor analysis was employed to validate the questionnaire's structure and accuracy.

Materials and methods

Study design and process

The questionnaire development process consisted of the following stages.

Establishment of a questionnaire development working group

The working group for this study included decision-makers, researchers, and nursing management staff from various institutions, including Fudan University, Shanghai Traditional Chinese Medicine University, and Shanghai University of Medicine and Health Science. In addition, experts from multiple disciplines, such as epidemiology, evidence-based medicine, evidence-based social sciences, public health management, health policy, and health education, were recruited to ensure that a wide range of perspectives was represented during the development of the questionnaire.

Conceptual framework and main concepts

The prevailing concept of patient adherence refers to a patient's adherence to prescribed behaviors, including medication intake, food regulation, lifestyle adjustments, and regular attendance at medical visits, thereby establishing a compliant rapport between the patient and healthcare provider.

The theory of planned behavior (TPB) is a theoretical framework that elucidates and forecasts individual behavior. The theoretical model exerts an impact on real behavior through behavioral attitudes, norms, and perceptions [23]. The idea of planned behavior has been widely used across various disciplines and has strong explanatory power. This study could offer a theoretical framework for administering an OHAQ-SP (Fig. 1).

Fig. 1

Theoretical framework

Determination of the content of the questionnaire items

This study relied on the definition of adherence, theory of planned behavior, Medication Adherence Self-efficacy Scale [24], Chinese guidelines for stroke rehabilitation treatment, and Recommendations for Exercise for Stroke Patients issued by the American Heart and Stroke Association ( AHA) in 2014. Following the group discussion, a 28-item questionnaire was created to assess adherence in stroke patients. The questionnaire consisted of 8 items related to medication adherence, 5 items related to self-monitoring adherence, 8 items related to rehabilitation adherence, and 7 items related to adherence to lifestyle and behavioral modifications.

Experts’ consultation

Experts from pertinent disciplines were invited to assess the initial version of the questionnaire in two rounds of consultation with the aim of enhancing its precision. In the initial phase, specialists were asked to evaluate the essentiality, significance, achievability, and comprehensibility of each element in the survey using a five-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = undecided, 4 = agree, and 5 = strongly agree). Additionally, they had the opportunity to offer recommendations for altering particular elements or for the questionnaire as a whole. Items with mean scores of 3.5 or higher and full score ratios of 0.2 or higher were retained, while items with lower scores were either adjusted or removed based on the findings of this round.

In the second round, experts were tasked with assessing the pertinence of dimensions to subjects and things using a four-level Likert scale ( scores ranging from 1 to 4 indicate varying degrees of irrelevance to high relevance).The scale was adjusted for expert probabilistic consistency (Pc), and the validity of the questionnaire was evaluated using modified kappa statistics (K ∗). K ∗ value ranging from 0.60 to 0.74 were considered to indicate strong content validity. A K ∗ value over 0.74 indicated exceptional validity [25, 26].

Validation

The validation study was conducted in Shanghai, China between April 1 and July 1, 2022. To achieve a heterogeneous sample, individuals were selected from multiple medical establishments, including the Shanghai Hospital of Traditional Chinese Medicine, Yueyang Hospital of Integrated Traditional Chinese and Western Medicine, Shanghai University of Traditional Chinese Medicine, Shuguang Hospital, Longhua Hospital, and Huashan Hospital Fudan University. Participants were recruited via convenience sampling, considering factors such as age, sex, and other relevant characteristics. First, the target sample size of 280 participants was determined, approximately ten times the number of questionnaire items. To compensate for potential omissions in interviews and unreliable surveys, the sample size was increased by 20%, yielding the desired sample size of 308 participants. The inclusion criteria for patients were as follows: (1) history of transient ischemic attack (TIA) or stroke, (2)1 to 6 months post-illness and stable, (3) proficiency in language and effective communication, and (4) willingness to participate in this research. The exclusion criteria for patients were as follows: (1) severe complications, cognitive impairment, or mental disorders; (2) visual or mental impairments that hindered their ability to complete the assessment; (3) employment in medical, nursing, or other healthcare-related fields; and (4) voluntary withdrawal from the study for any reason.

Survey methods and quality control

The six subject investigators were divided into three pairs and underwent relevant training in their respective areas of expertise. Before administering the survey, the goals and substances of the study were comprehensively elucidated to the participants. All participants provided written informed consent. After providing consent from the participants, the researchers proceeded to ask the questions individually, giving the participants the option to answer independently or provide verbal guidance to the researchers while recording their responses. Upon completion, all questionnaires were promptly collected and the data were input into the study database using MS Office Excel on the same day. The research group subsequently conducted a questionnaire validation on the day of data collection to identify and eliminate any replies that were deemed invalid or incomplete to ensure the general validity of the data.

Statistical analysis

The level of agreement across expert viewpoints was measured using the coefficient of variation (CV) and the Kendall harmonic coefficient (KW). The significance of the Kendall harmonic coefficient was assessed using the χ2 test. The total Kendall coordination coefficient had a significance level of P < 0.01, suggesting a strong level of coordination. The expert authority coefficient (Cr) was calculated as the average of the judgement basis coefficient (Ca) and the familiarity coefficient (Cs). The data obtained from the validation research were analyzed using the statistical software SPSS 26.0. Counts were represented by their frequency and percentage, whereas quantitative data were represented by the mean value plus or minus the standard deviation. Person correlation analysis, critical ratio, exploratory factor analysis, confirmatory factor analysis, and Cronbach's alpha coefficient were employed to evaluate content validity, internal consistency, and test reliability, respectively.

Ethics approval

The study was carried out in strict adherence to the principles of voluntariness, secrecy, and respect for human subjects, ensuring the protection of respondents' legitimate rights and interests. This trial was approved by the Institutional Ethics Committee of the Shanghai Municipal Hospital of TCM (IRB; reference number 2020SHL-KY-16; IRB at the Shanghai Municipal Hospital of TCM Ethics Committee).

Results

Questionnaire development

A total of 2152 documents were obtained, and 763 documents were selected, including 345 in Chinese and 418 in English. The entry pool consists of 46 entries. Following group discussions and expert consultations, a thorough review resulted in manual removal of 18 duplicate and similar items. Consequently, an initial OHAQ-SP consisting of 28 items was developed. This questionnaire included 8 items related to medication adherence(A), 5 items related to self-monitoring adherence(B), 8 items related to rehabilitation adherence(C), and 7 items related to adherence to lifestyle behavior changes(D). Each question was formatted as a single-choice one (Fig. 2).

Fig. 2

Procedure for developing the questionnaire items

Expert consultation and content validity

A total of 15 experts in related fields were consulted in the first round, including experts in health communication, behavioral science, public health policy, health promotion, stroke disease prevention and control, health career management, basic medical education research, and nutrition and health. The average number of years of work experience 24.2 ± 6.89 (Table 1).

Table 1 Characteristics of the experts (n = 15)

The Ca values in the two rounds of correspondence in this study were 0.85 and 0.87, respectively. The experts' familiarity with the Cs values were 0.88 and 0.89, respectively. The experts' Cr values were 0.87 and 0.88, respectively. All 15 experts were found to have high Cr values based on their self-identification. The Ca value ranged from 0.7 to 1.0, while the Cs value ranged from 0.8 1.0. The level of expert authority was deemed high.

KW values were calculated for each factor, yielding KW(A) = 0.344, KW(B) = 0.525, KW(C) = 0.299, and KW(D) = 0.282. The results were statistically significant (P < 0.01), indicating a high level of agreement among experts. The level of consensus among the expert viewpoints was deemed satisfactory, as all the scores were less than 0.25. Furthermore, the questionnaire underwent a second round of modifications in which expert viewpoints were incorporated to make the necessary deletions.

The second round of the expert consultation form involved 27 items across four dimensions. The KW values for each dimension were as follows: KW(A) = 0.533, KW(B) = 0.573, KW(C) = 0.575, and KW(D) = 0.672 (P < 0.05). One entry from the life behavior change adherence dimension that did not match the criteria was ultimately removed, resulting in 26 entries in the preliminary questionnaire (Table 2).

Table 2 The degree of coordination between the two rounds of expert opinions

Validation of the questionnaire

Questionnaire population validation analysis

To refine the questionnaire, 20 stroke patients who underwent a follow-up assessment in the neurology department of the previously mentioned hospital were randomly chosen for the preliminary survey. The questionnaire has a validity rating of 100%. A total of 12 males and 8 females were included, and the mean age was 61.38 ± 5.83 years. The questions were answered within approximately 7 min. The surveys have been revised and modified accordingly. The questionnaire consisted of 26 items distributed across four dimensions, with eight, four, seven, and seven items.

A standardized questionnaire was used to assess patients with stroke who underwent a follow-up assessment in the neurology department of the previously mentioned hospital. A total of 308 participants were included, 290 of whom were considered legitimate. This resulted in a valid recovery rate of 94.16%. A total of 170 males and 120 females participated in this study. The mean age was 59.03 ± 6.11 years. The majority of the patients had literacy levels equivalent to elementary school or junior high school. Married patients constituted 89.66% of the total patient population. Patients’ medical expenses were predominantly reimbursed by China's healthcare insurance (Table 3).

Table 3 General characteristics of stroke patients (n = 290)

Questionnaire discrimination and homogeneity analysis

The critical ratio t-test method was used, with the total score before and after 27% as the grouping standard. There were 51 people in the high-risk group and 49 people in the low-risk group. Two independent sample t-tests were performed to determine whether there were significant differences between the high-risk group and the low-risk group. Typically, the disparity in scores between the two groups is not statistically significant, or the t-value falls below 3, which is the threshold for exclusion. The results indicated that there were no statistically significant differences between the A3, A8, B1, and C5 scores. Consequently, the research team opted to exclude A8 from the dataset while retaining the remaining three entries. All remaining entries had a t value greater than 3, indicating that the questionnaire exhibited a high level of differentiation. All the Pearson correlation coefficients for the total correlation question were above 0.4, suggesting that each item had a high level of internal consistency and that the questionnaire demonstrated good homogeneity [27] (Table 4).

Table 4 Analysis of the total correlation of the questions

Questionnaire validity analysis

Exploratory factor analysis of the questionnaire yielded a KMO value of 0.923 and an approximate chi-square value of 8964.138, with a significance level of P < 0.001. These results indicate that the OHAQ-SP was successful after factor analysis and variable extraction. The questionnaire identified four factors that did not have multivariate loading. The dimensions had high loadings on their respective factors, suggesting that the results of the exploratory factor analysis were consistent with the original design of the scale. This indicates that the questionnaire had good structural validity (Table 5).

Table 5 Analysis of exploratory factor

Furthermore, the results of the factor analysis model validation for the questionnaire indicated that the CMIN/DF value was 2.490, RMR was 0.05, GFI was 0.901, NFI was 0.927, and IFI was 0.955. All of these values met the criteria for an excellent model fit. Additionally, the RMSEA value was less than 0.08, which is considered a good standard for a model fit. These findings suggest that the factor analysis model used in this study is well suited. The questionnaire demonstrated strong aggregate validity, as indicated by standardized loading values ranging from 0.717 to 0.916 across the four dimensions (Table 6 and Fig. 3).

Table 6 Analysis model of adaptation of validating factor

Fig. 3

Verifying factor model

Convergent validity measures the degree of agreement between the different aspects of the underlying variable. When examining the four dimensions, the standardized loadings for the measurement items varied from 0.717 to 0.916, all of which were greater than 0.7. The combined reliabilities of the four dimensions were 0.936, 0.864, 0.917, and 0.929, all greater than 0.7. Additionally, the average variance extracted (AVE) values for the four dimensions were 0.676, 0.614, 0.611, and 0.652, all of which exceeded 0.5. Hence, the overall reliability of the questionnaire was satisfactory (Table 7).

Table 7 Analysis of aggregation validity

Discriminant validity analysis revealed that the bold numbers represent the square root of the mean variance recovered for each dimension, while the other numbers indicate the correlation coefficients between the different dimensions. The square root of the mean variance extracted for all four dimensions exceeded the correlation coefficient between dimensions. This suggested that there was a distinction between the four dimensions and that the scale demonstrated high discriminant validity (Table 8).

Table 8 Analysis of discriminant validity

Questionnaire reliability analysis

The study revealed that the Cronbach's alpha coefficients for dimensions A, B, C, and D were 0.936, 0.862, 0.916, and 0.928, respectively. Cronbach's alpha coefficient for the total scale was 0.902. Importantly, all Cronbach's alpha coefficients exceeded the threshold of 0.7, indicating a high level of reliability for measuring stroke adherence. The formal questionnaire demonstrated a high internal consistency, reliability, and stability [28] (Table 9).

Table 9 Analysis of Cronbach's alpha coefficient for four dimensions

Instructions for utilizing the questionnaire

The stroke patient compliance questionnaire encompasses four dimensions: medication adherence, self-monitoring adherence, rehabilitation adherence, and life behaviors change adherence, comprising a total of 25 items. The Likert 5-point scale is employed, with "strongly disagree," "disagree," "neutral," "agree," and "strongly agree" receiving 1 to 5 points respectively, resulting in a cumulative score ranging from 25 to 125 points.

Discussion

This study aims to create a dependable measurement instrument to assess the out-of-hospital compliance of stroke patients, grounded in the Theory of Harmed Behavior. OHAQ-SP encompasses four dimensions: medication adherence, self-monitoring, rehabilitation, and adjustments in daily life behavior. It may precisely and thoroughly evaluate the out-of-hospital adherence of stroke patients, hence assisting clinical personnel in performing focused evaluations of patients' situations.

The scale was developed by a meticulous process. A thorough literature study of local and foreign studies on stroke patient compliance was done to establish a preliminary item pool, subsequently enhanced by qualitative interview findings. The 15 polled experts were drawn from three provinces and cities in China, encompassing medical professionals, nurse administrators, traditional Chinese medicine practitioners, and stroke nursing specialists, all of whom were regionally representative and authoritative in their respective domains. Following the development of the scale, 20 patients were chosen for a preliminary survey to guarantee a good level of readability. During the reliability testing phase of the scale, this study employed a sample size from several hospitals in Shanghai to validate the data's legitimacy. Consequently, the scale can precisely evaluate the adherence of stroke patients in non-hospital settings and demonstrates good validity.

The content validity, structural validity, discriminant validity, and convergent validity of the validity scale constitute its overall validity. The critical ratio t-test t-values for the validity assessment in this study exceeded 3, and the Pearson correlation coefficients were all over 0.4. This indicates that the content and distribution of the scale are fairly rational. In the enrollment analysis, four methodologies—literature review, Delphi expert consultation, small sample pre-experiment, and reliability and validity analysis—were employed for enrollment screening, thereby affirming the scale's homogeneity. Exploratory factor analysis was employed to ascertain the optimal structure of the scale, and the "question-by-question deletion procedure" was utilized to reduce the number of items. Ultimately, 25 items were preserved, and the shared components with eigenvalues over 1 were utilized as the four dimensions of the scale, accompanied with a KMO value of 0.923. Factor 1 pertains to medication adherence, factor 2 relates to self-monitoring adherence, factor 3 concerns rehabilitation adherence, and factor 4 involves lifestyle behavior change adherence. The outcomes of the exploratory factor analysis align with the original structure of the questionnaire and affirm its robust structural validity. The standardized loading values between the dimensions and measurement items of OHAQ-SP ranged from 0.717 to 0.916, all exceeding 0.7, signifying that the four dimensions of OHAQ-SP were appropriately grouped. A confirmatory factor analysis was performed on OHAQ-SP, yielding an RMSEA value below 0.08, which meets the criteria for a strong model fit. The Cronbach’s α coefficients for the questionnaire were 0.936, 0.862, 0.916, and 0.928, respectively, while the overall scale's Cronbach’s α coefficient was 0.902, demonstrating strong internal consistency reliability of the questionnaire.

Besides aspects like stroke type and the timing of rehabilitation initiation post-stroke, adherence levels can profoundly influence the overall rehabilitation outcomes for stroke patients [29]. The World Health Organization defines adherence as the extent to which a patient's actions align with the recommendations of a healthcare practitioner about personal conduct, drug usage, dietary adherence, or lifestyle modifications [30, 31]. Instead of concentrating on a singular facet of compliance in post-stroke patients. Consequently, a thorough evaluation of stroke patients' adherence is a crucial component of their rehabilitation. Currently, numerous approaches exist for evaluating patient compliance; however, most are not used to stroke patients, and certain questionnaires raise concerns regarding their reliability and validity [32, 33]. This study established the questionnaire's dimensions by exploratory factor analysis and presented succinct and pertinent items that can effectively assess the out-of-hospital compliance of stroke patients. The OHAQ-SP developed in this study employs a scientific and rigorous methodology, emphasizes patients' subjective experiences, contains the fewest items, offers the most comprehensive content, demonstrates excellent sensitivity and specificity, and possesses substantial practicality and therapeutic relevance. It assists medical personnel in accurately assessing the overall compliance of stroke patients during rehabilitation, facilitates self-evaluation for patients, and establishes a platform for future study.

Limitations of questionnaire development

A few limitations should be taken into consideration in interpreting the results of this study. The reliability and validity of the questionnaire were established using data from the Shanghai stroke population. Hence, the generalizability of the results to the overall Chinese population cannot be assured. Furthermore, the sample size of 290 participants may be considered insufficient. However, employing a questionnaire to assess out-of-hospital adherence in patients with stroke is currently the best approach. The research group intended to regularly revise the questionnaire at intervals of approximately 3–5 to years. Additionally, we aimed to broaden the scope of the survey by including a larger sample size to enhance its rigor.

Conclusion

In conclusion, the OHAQ-SP is a reliable and accurate instrument for evaluating stroke patients in routine clinical settings and in population-based research.