Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT (original) (raw)

Abstract

Background

The positive effects of the Dutch Community Occupational Therapy in Dementia programme on patients' daily functioning were not found in a multicentre randomised controlled trial (RCT) in Germany.

Objectives

To evaluate possible effect modification on the primary outcome within the German RCT with regard to (1) participant characteristics, (2) treatment performance and (3) healthcare service utilisation; and (4) to compare the design and primary outcome between the German and the original Dutch study.

Methods

(1) The impact of participant baseline data on the primary outcome was analysed in exploratory ANCOVA and regression analyses. (2) Therapists completed questionnaires on context and performance problems. The main problems were identified by a qualitative content analysis and focus-group discussion. Associations of the primary outcome with scores of participant adherence and treatment performance were evaluated by regression analysis. (3) Utilisation rates of healthcare services were controlled for significant group differences. (4) Differences in the Dutch and German study design were identified, and the primary outcome was contrasted at the item level.

Results

(1) Participant characteristics could not explain more than 5% of outcome variance. (2) The treatment performance of some active intervention components was poor but not significantly associated with the primary outcome. (3) There were no significant group differences in the utilisation of healthcare resources. (4) In contrast to the Dutch waiting-control group, the active intervention in the German control group may have reduced group differences in the current RCT. The German patients demonstrated a higher independence at baseline and less improvement in instrumental activities of daily living.

Conclusion

The differences in outcome may be explained by a more active control treatment, partially poor experimental treatment and less room for improvement in the German sample. Future cross-national transfers should be prepared by pilot studies assessing the applicability of the intervention and patient needs specific to the target country.

Trial registration

International Clinical Trials Registry Platform, DRKS00000053.

Keywords: RCT—Alzheimer's disease—complex intervention—process evaluation

Article summary

Article focus

Key messages

Strengths and limitations of this study

Introduction

New guidance from the British Medical Research Council states that developing and evaluating complex interventions can be a lengthy process. All steps should be sufficiently addressed. These steps include (1) the development of the intervention, (2) a pilot study on feasibility, (3) a randomised controlled trial (RCT) on effectiveness and (4) an evaluation of implementation in healthcare practice.1 Cross-national transfer of complex intervention can speed up the uptake of innovative and effective programmes from one country to another. Time and resources might be saved when an intervention programme that has already been developed, piloted and evaluated on effectiveness in one country can be directly proven regarding effectiveness in the healthcare context of another country. We followed this approach by transferring the Dutch evidence-based Community Occupational Therapy in Dementia Programme (COTiD)2 to the German healthcare system and testing its effectiveness in a seven-centre RCT.3 However, the highly positive effects of the Dutch COTiD on patients' daily functioning could not be found. Process evaluation is recommended as being highly valuable in RCTs to provide an insight into unexpected intervention failure.4 Differences in participants as well as aspects of treatment performance and contextual factors should be assessed with regard to their associations with the primary outcome.5–7 Based on these recommendations, our process evaluation investigated four research questions. We evaluated both possible bias within the German study (question 1–3) and differences between the Dutch and German RCT (question 4).

  1. Did specific patient or carer characteristics influence a patient's outcome after the intervention?
  2. What problems and variations in experimental treatment performance could be identified in the study context, and did they influence the daily functioning of patients?
  3. What differences in the utilisation of further healthcare resources during the treatment period could be identified, and did they influence the daily functioning of patients?
  4. What differences between the Dutch and the German study could be identified in terms of design and primary outcome?

Methods

Specific participant characteristics of the German sample

The outcome of interest was daily functioning, indicated by two measurement instruments, the Interview for Deterioration in Daily Living Activities in Dementia (IDDD) and the Perceive, Recall, Plan and Perform System of Task Analysis (PRPP). The IDDD performance scale records the patients' need for assistance in 11 basic and instrumental activities of daily living.8 In the PRPP, the number of errors occurring during the performance of a self-chosen daily living task is measured.9 An ANCOVA was used to investigate the mean changes from baseline in the IDDD and PRPP between the COTiD and control group controlling for (1) the patient's age, gender, education and financial limitation; and the daily activities, mood and cognition at baseline; and (2) the carer's gender, education, relationship to the patient; and the sense of competence and mood at baseline. Data were collected with standardised measurement instruments as described in the study protocol and were in line with a recent health-technology assessment of risk or protective factors for Alzheimer's disease.10 11 The percentage variance explained in mean changes from baseline in the IDDD and PRPP was assessed using multiple regression.

COTiD performance in the German experimental group

Therapists completed semistructured questionnaires during and after the treatment period. (Questionnaires are available in German from the corresponding author.) During the treatment phase, they reported reasons for a problematic performance in 20 subprocesses for each experimental case. The subprocesses were defined according to the study protocol (table 1 and 4). After the treatment period, therapists described and rated their professional experience in the field and their valuation of introduction, pilot phase and supervision, as well as inhibiting and facilitating processes at the study site. A qualitative content analysis with inductive category development was used to identify the main performance problems from the comments given in the questionnaires.12–16 A focus-group discussion served as a member check, in order to achieve consensus among the therapists about the main performance problems.17–21

Table 1.

Statements by therapists stating main performance problems within the therapeutic subprocesses

Setting therapy goals ‘Priorisation by the patient was difficult, because he was very uncritical.’ ‘The carer wants immediately to talk about problem solving. I again and again had to suggest the procedure [of systematic shared goal setting].’
Educating patient in new skills ‘Patient needs much guidance. Concentration and endurance [are] very limited. Assistance for simple tasks [is needed].’ ‘Activities agreed on could not be carried out twice due to apathy and depressive mood.’ ‘In addition, patient had dyspraxia, which made training difficult.’ ‘[There was a] lack of training due to the negative attitude of the carer.’ ‘It is difficult for the patient to accept the disease. Therefore a high degree of convincing is needed in each session.’
Adapting physical or social environment ‘The carer is the house owner and refuses any adaptation.’ ‘[Adapting physical environment] does not succeed because the carer is ostensibly open for intervention, but in reflective talks reluctant and negative.’ ‘An adaptation [of the physical environment] seems not reasonable to the patient, although [it is] necessary.’ ‘[Adapting physical environment] is possible only step by step, because the patient reacts on it with reluctance.’ ‘The patient lives rather reclusively, wishes no changes [in the social environment].’ ‘The patient is very anxious and avoiding [change]’. ‘The son strongly adheres to old patterns of interaction’. ‘The family dynamic is very fixed. Both daughters seem to have difficulty in just letting the mother [patient] simply do … Changes take place, but very slowly. [It is] questionable, whether there will be work on the goals after the intervention is finished.’ ‘In the community, there is no day care and no care centre for people with dementia.’
Training of carer's competence ‘The son [is] often not or only temporary present at the sessions.’ ‘[The carer is] many a time overstrained and tries to give away [the responsibility] to the therapist.’ ‘It is difficult for the carer to get used to something new. He quickly falls back into old patterns [of behaviour] without being aware of it.’ ‘[The carer] seems to be very overstrained and burdened by the disease. He needs additional professional support, for example, from a psychologist’. ‘The carer has need for support, but refuses any offer of support for himself.’ ‘The carer mostly sees only his own problems. He cannot or only very rarely empathises with the patient. Offers of support are refused.’ ‘There are difficulties in the interaction between the family and the patient. The patient plays off the caring family members against each other.’

Furthermore, the therapists dichotomously scored the 20 treatment subprocesses as performed either with or without problems. These scores were used to operationalise the quality of performance for each case. The best quality was indicated by 100% when all subprocesses were performed without any problems.

The therapists also rated patient adherence regarding the cooperation during the interview, the goal setting and the training; as well as regarding the patient's daily changing mental capacity, their collaboration with the carer and regarding the acceptance of innovations. Additionally, the carer adherence was assessed with regard to their cooperation during the scheduling, the interview, the goal setting and the training of supervision, as well as with regard to their encouragement of the patient, the acceptance of support service and the implementation of innovations. Therapists rated these indicators for adherence on a five-point-Likert scale ranging from ‘very facilitating for the treatment performance’ (=1) to ‘very hindering for the treatment performance’ (=5).

Correlations between the mean changes from baseline in the IDDD and PRPP and scores of the performance quality and the participant adherence were calculated (Pearson coefficient). An exploratory regression analysis was deemed to be appropriate for smaller samples, and this was to evaluate whether such scores could explain variance in the mean changes from baseline in the IDDD and PRPP.22

Utilisation of healthcare resources in the German study

The Resource Utilisation in Dementia23 was applied to collect patient data during the treatment period on (1) the number of consultations with general practitioners, neurologists/psychiatrists and other medical specialists; (2) the time for individual therapy such as physio-, speech- or psychotherapy; (3) the time for group therapy such as cognitive stimulation or exercise groups; (4) times of receiving nursing or domestic home care; (5) the number of technical aids implemented within the patient's home; and (6) increasing, decreasing, constant or no intake of acetyl cholinesterase inhibitors. Furthermore, comorbidity indicating a possible need for further healthcare services was rated using the Cumulative Illness Rating Scale.24 These data were tested for the significance of group differences between the experimental and the control arm (non-parametric two-tailed Mann–Whitney U test owing to the negative skewness of the data distribution).

Comparison between the Dutch and the German study

Regarding the patient characteristics and change from baseline to follow-up on the primary outcome, we compared the single IDDD items of the Dutch and the German sample within the identical measurement period of 5 weeks from baseline to the first follow-up measurement at week 6. This was in order to assess whether both samples had the same room for improvement in items which indicate the need for assistance in daily activities. Furthermore, we compared the expertise of the Dutch and German therapists in terms of pre-experience with the experimental intervention and intensity of treatment delivery (patients per therapist) and the study designs regarding the control-group intervention.2 3

Results

Specific participant characteristics of the German sample

The mean changes to baseline in the IDDD and the PRPP were associated neither with carers' socio-demographic or baseline assessment data nor with patients' socio-demographic data or baseline mini mental state (table 2). We found a minor correlation of mean changes to baseline in the IDDD with patients' mood at baseline (Cornell Scale for Depression in Dementia (CSDD)25; r=0.21; p=0.044). A stepwise regression analysis using patient and carer characteristics as listed in table 2 could not explain more than 5% of variance in change over time in the patient's daily functioning. An adjusted ANCOVA using the patient's baseline values of the CSDD, the PRPP and the IDDD as independent variables did not yield any significant group differences in the dependent variables, which were the mean changes to baseline in the IDDD and the PRPP (results not shown). This indicated that after correction for baseline scores of mood and daily functioning, there were still no significant differences on the primary outcome in the German sample with moderate to good daily functioning at baseline.

Table 2.

Pearson correlation coefficient of specific participant characteristics and mean changes to baseline in the Interview for Deterioration in Daily Living Activities in Dementia and the Perceive, Recall, Plan and Perform System of Task Analysis (German completers of the Community Occupational Therapy in Dementia Programme and control group)

N Perceive, Recall, Plan and Perform System of Task Analysis24 change to baseline Interview for Deterioration in Daily Living Activities in Dementia23 change to baseline
Patient
Age 104 −0.02 0.11
Gender 104 −0.16 −0.11
Education 104 −0.13 −0.02
Financial limitation 93 0.07 0.14
Mood, Cornell Scale for Depression in Dementia25 baseline 95 0.16 0.21*
Cognition, Mini-Mental State Examination baseline 104 0.02 0.10
Carer
Gender 104 0.08 −0.11
Education 104 −0.12 0.11
Relationship to patient 104 0.15 0.09
Sense of competence, Sense of Competence Questionnaire26 baseline 103 −0.06 −0.02
Mood, Center for Epidemiologic Depression Scale27 baseline 103 0.09 0.09

COTiD performance in the German experimental group

Eleven therapists from seven study sites delivered the COTiD to 54 patients. The therapists' characteristics (table 3) varied in previous years in dementia care from 1 year part time to 11 years full time, in perceived facilitators from quite facilitating to slightly hindering and in the quality of treatment performance from 52 to 90% of optimal performance. The data did not provide stable patterns in the sense that many previous years in dementia care and high values of perceived facilitators did lead to a high quality of treatment performance or vice versa.

Table 3.

Characteristics of the 11 therapists who delivered Community Occupational Therapy in Dementia Programme to 54 patients with Alzheimer's (German completers of the experimental intervention)

Basic data Perceived facilitators† Treatment performance
Age Gender Years in occupational therapy Years in dementia care Pre-existing knowledge Study preparation Site support Total Cases Quality (%)‡
27 Male 3 3 2.2 2.0 1.1 1.8 3 86
31 Female 8 5 1.6 2.0 1.2 1.6 3 89
45 Female 9 7 1.8 2.5 2.2 2.2 5 94
44 Female 7 1* 3.2 1.7 1.8 2.2 2 64
40 Male 13 13* 3.8 3.0 2.4 3.1 10 81
34 Female 5 3 3.8 2.5 2.8 3.0 10 90
54 Male 11 11 2.0 3.5 2.7 2.7 4 73
36 Female 11 9 2.3 2.7 2.6 2.5 1 52
39 Female 18 12* 2.4 2.5 2.9 2.6 2 74
40 Female 6 3* 2.4 3.3 2.9 2.9 2 59
32 Female 9 9* 4.2 3.3 3.4 3.4 12 84

The quality of the subprocess performance (table 4) did also vary from receiving full medical information in 52 of 54 cases (96%) to successfully adapting physical environment in 24 cases (44%). Subprocesses relating to therapeutic active agents as identified by Graff et al15 were performed with no problems at only a low frequency with 76% for setting therapy goals, 46% for training of patient's skills, 44 and 46% for adapting physical and social environment, and 59 and 54% for training of carer's competence in instruction and problem-solving. In the questionnaires, the therapists commented on the main performance problems in these subprocesses (table 4).

Table 4.

Quality of subprocesses of Community Occupational Therapy in Dementia Programme performance in 54 Alzheimer's disease patients (German completers of the experimental intervention)

Subprocesses Performance* Main problems
Good (%) Poor
01 Receiving medical information 52 (96) 2 Received wrong phone number or no detailed medical information
02 Making appointments with participants 49 (91) 5 Participants had other appointments
03 Travelling to participants 46 (85) 8 Long travel to patient's home (some >40 km)
04 Meeting the participants 50 (93) 4 Participants forgot to cancel the date and were late or not at home
05 Contacting and providing confidence 50 (93) 4 Patient was sceptic or abrasive
06 Informing about the procedure 50 (93) 4 Patient could not understand procedure, misunderstood procedure as test for nursing home placement
07 Observing the time frame 42 (78) 12 Participants (mainly carer) had a great need to tell and talk
08 Explaining clearly, responding to questions 50 (93) 4 Patient could not understand the explanations, owing to communication deficits or mood swings
09 Mastering conflicts and problematic situations 39 (72) 15 Patient had severe mood swings or additional cognitive deficits or was not aware of deficits; carer was overstrained, abrasive or placed sole responsibility on therapist; family conflicts existed for a long time
10 Interviewing patient with OPHI 38 (70) 16 Patient was unable or hardly able to tell, had anomia or severe deficits in biographic memory or was disorientated
11 Observing patient activity with Volitional Questionnaire, if OPHI not done 5 (71*) 2 Patient not motivated to demonstrate activities; *Volitional Questionnaire not necessary in 47 cases, because OPHI was done
12 Interviewing carer with Ethnographic Interview 47 (87) 7 Carer had only little understanding of dementia or felt very burdened
13 Observing activities of patient and carer 43 (80) 11 Patient did activity incompletely, was very passive or was fraught when being observed; carer was demanding or impatient
14 Setting therapy goals with patient and carer 41 (76) 13 Participants negated need for change or could not specify goals
15 Defining occupational therapy problems 43 (80) 11 Patient had no activity limitations; participants could not understand the relevance of problems; problems were very complex or became clearer only later during intervention or were related not to dementia but to depression or physical limitations
16 Educating patient in new skills and compensation capability 25 (46) 29 Patient was not or hardly motivated in training, additional symptoms such as dyspraxia, depression, apathy, attention deficit disorder hampered the training; carer or family were not supportive
17 Adapting physical environment 24 (44) 30 Participants refused or hesitantly accepted necessary adaptations
18 Adapting social environment 25 (46) 29 Participants were reluctant to change social environment; informal social support or care services were lacking
19 Training of carer's competence in instruction and interaction 32 (59) 22 Carer could not change behaviour as being very burdened or impatient or bound in firm habits; was not willing to take responsibility or was missing sessions
20 Training of carer's competence in problem solving 29 (54) 25 Carer was not willing to undertake the responsibility of problem solving or not able to do so owing to high burden; carer would have needed more time or further support to undertake the responsibility for independent problem-solving

Association between COTiD performance and primary outcome

We found no significant associations between the scores of COTiD performance and changes to baseline in the IDDD and in the PRPP (detailed data not shown). Since there was a poor performance in those subprocesses which were related to active therapeutic agents (nos 14 to 20; table 4), we further analysed the association between the performance score of these subprocesses and the changes to baseline in the IDDD and in the PRPP. A minimal correlation was found, r=0.268 (p=0.05) only with the PRPP. No association was found between carer adherence and the changes to baseline in the IDDD and in the PRPP. The score of patient adherence and the change to baseline in the PRPP demonstrated a moderate correlation of r=−0.317 (p=0.02). The subsequent regression analysis revealed that patient adherence could explain 10% of the variance (p=0.02) in the PRPP change to baseline. The IDDD change to baseline could not be explained by patient or carer adherence, or by the quality of treatment performance.

Utilisation of healthcare resources in the German study

The COTiD group had a somewhat higher comorbidity index, slightly more visits to general practitioners and fewer hours for nursing or domestic home care. Negative skewness in the data distribution indicated that many participants had a low utilisation rate, and only a few participants had a high intensity of resource utilisation (table 5). However, we found no significant differences on group level within the German trial in any resource utilisation or comorbidity. The subgroups of patients with decreasing or increasing acetylcholinesterase-inhibitor medication were too small to detect any significant group differences. However, the daily functioning in the COTiD group was not better than in the control group, although more COTiD patients received acetylcholinesterase inhibitors at a constant level (COTiD, 63% vs control, 52%).

Table 5.

Utilisation of further healthcare resources of patients with Alzheimer's during the intervention period of intense occupational therapy compared with a single session control intervention (German completers of the Community Occupational Therapy in Dementia Programme and control group)

Healthcare resources Community Occupational Therapy in Dementia Programme (n=54) Control (n=50)
Mean (SD) Range Skewness Mean (SD) Range Skewness
Medical consultations per week
General practitioner 0.28 (0.40) 0–2.33 3.211 0.18 (0.17) 0–0.67 0.828
Neurologist or psychiatrist 0.04 (0.09) 0–0.33 2.092 0.03 (0.09) 0–0.50 3.508
Other medical expert 0.14 (0.26) 0–1.33 2.778 0.15 (0.24) 0–1.00 1.825
Hours for therapy per week
Individual therapy 0.14 (0.32) 0–1.00 2.029 0.11 (0.29) 0–1.00 2.597
Group therapy 1.05 (2.92) 0–14.00 3.064 0.88 (3.32) 0–16.00 4.163
Hours for nursing or domestic home care per week 1.33 (3.23) 0–15.08 3.074 1.87 (5.32) 0–25.54 3.173
No of technical aids provided at home 0.15 (0.49) 0–3.00 4.306 0.06 (0.24) 0–3.00 3.821
Comorbidity (Cumulative Illness Rating Scale24) 3.15 (3.20) 0–13 1.101 2.42 (2.60) 0–11 1.374
No of patients with acetylcholinesterase-inhibitor medication
De-novo treatment or increased dose 4 (7%) 3 (6%)
Decreased dose or medication ceased 2 (4%) 1 (2%)
Constant level 34 (63%) 26 (52%)
No acetylcholinesterase-inhibitors 14 (26%) 20 (40%)

Comparison between the Dutch and the German study

Differences in the room for improvement in the IDDD

Table 6 shows that the COTiD group in the Dutch sample did improve notably in household instrumental activities of daily living (IADL), only marginally in basic activities of daily living and not at all in handling finances. Graff et al2 defined 20% improvement as being clinically relevant, which is indicated by a pre–post-treatment difference of 0.8 on item-level. The household IADL items demonstrated such differences and, therefore, a high responsiveness to the COTiD programme. Thus, the household IADL items can be presumed to be a therapeutic window basically providing room for improvement given a sufficient need for assistance in these items at baseline. Comparing the Dutch and German COTiD groups, the baseline values in these IADL items differed considerably more than in the other IDDD items.

Table 6.

Responsiveness of specific activities of daily living after an identical treatment period of 5 weeks in the Dutch and the German sample

Dutch Community Occupational Therapy in Dementia Programme Dutch control German Community Occupational Therapy in Dementia Programme German control
N Mean T0–T1 SD N Mean T0–T1 SD N Mean T0–T1 SD N Mean T0–T1 SD
IDDD items with low responsiveness in the Dutch sample
Washing oneself T0 55 1.42 0.67 1.37 57 2.04 −0.07 1.63 54 0.74 0.15 1.20 50 0.78 −0.12 1.27
T1 55 0.75 0.95 57 2.11 1.58 54 0.59 1.00 50 0.90 1.30
Dressing T0 55 1.73 0.71 1.47 57 2.09 −0.17 1.49 54 0.98 0.07 1.34 50 1.06 −0.06 1.32
T1 55 1.02 1.24 57 2.26 1.51 54 0.91 1.26 50 1.12 1.44
Combing one's hair and brushing one's teeth T0 55 0.67 0.29 1.11 57 1.04 −0.05 1.40 54 0.70 0.00 1.16 50 0.60 −0.06 1.14
T1 55 0.38 0.87 57 1.09 1.43 54 0.70 1.19 50 0.66 1.15
Eating T0 55 0.18 0.03 0.67 58 0.43 0.03 0.98 54 0.28 0.02 0.74 50 0.40 0.04 1.07
T1 55 0.15 0.45 58 0.40 0.95 54 0.26 0.78 50 0.36 0.92
Using the toilet T0 55 0.42 0.17 0.88 57 0.96 0.12 1.48 54 0.37 0.07 0.85 50 0.58 0.22 1.09
T1 55 0.25 0.65 57 0.84 1.39 54 0.30 0.74 50 0.36 0.94
Handling finances T0 47 3.89 0.10 0.60 55 3.87 0.00 0.39 54 3.09 0.00 1.35 50 2.78 0.12 1.66
T1 47 3.79 0.69 55 3.87 0.61 54 3.09 1.35 50 2.66 1.60
Overall mean T0 55 1.33 0.34 0.73 57 1.73 −0.01 0.92 54 1.03 0.05 0.77 50 1.03 0.02 0.85
T1 55 0.99 0.59 57 1.74 0.88 54 0.98 0.78 50 1.01 0.84
IDDD items with high responsiveness in the Dutch sample
Making tea or coffee T0 55 2.05 1.14 1.39 57 2.18 −0.26 1.45 54 1.33 0.13 1.66 49 1.04 0.20 1.46
T1 55 0.91 0.91 57 2.44 1.31 54 1.20 1.59 49 0.84 1.21
Shopping T0 53 3.62 1.19 0.71 57 3.33 −0.06 1.19 54 2.00 −0.15 1.70 50 2.08 0.08 1.64
T1 53 2.43 1.17 57 3.39 1.18 54 2.15 1.71 50 2.00 1.60
Using the phone T0 55 2.04 1.04 1.53 54 2.28 −0.26 1.46 54 1.43 0.26 1.51 50 1.18 −0.24 1.29
T1 55 1.00 1.23 54 2.54 1.53 54 1.17 1.34 50 1.42 1.54
Preparing a meal T0 54 3.22 1.26 1.16 55 3.09 −0.18 1.44 54 2.15 0.11 1.80 50 1.92 0.30 1.77
T1 54 1.96 1.32 55 3.27 1.37 54 2.04 1.74 50 1.62 1.68
Cleaning the house or doing minor repair work T0 54 3.15 1.43 1.24 56 3.18 −0.36 1.22 54 2.31 0.43 1.65 50 1.62 0.06 1.50
T1 54 1.72 1.27 56 3.54 0.93 54 1.89 1.64 50 1.56 1.59
Overall mean T0 55 2.81 1.21 0.78 57 2.81 −0.22 0.96 54 1.84 0.16 1.29 50 1.56 0.08 1.56
T1 55 1.61 0.74 57 3.04 0.86 54 1.69 1.24 50 1.50 1.19

The German patients showed much less need for assistance in this area. The limited room for improvement in the German sample is obvious when regarding the baseline differences between the Dutch and the German sample. Analysis of a German subsample matched to the Dutch sample with a comparable need for assistance in these household IADL items at baseline was not possible owing to the low number of German patients with such baseline values.

Differences in design

The German trial design included a comprehensive consultation as active control intervention which approximately represents the non-pharmacological standard care in Germany. This was in order to evaluate the possible benefits of COTiD in addition to standard care. Compared with the waiting-control-group design of the Dutch original trial, the German active control intervention may have reduced the group differences in daily functioning after the treatment. Compared with the Dutch therapists, the therapists in Germany had less experience with COTiD before their study involvement (NL: 240 h vs GER: 0 h), less seminar and training time in the study preparation phase (80 h vs 40 h) and fewer COTiD patients per therapist during the treatment period (34 vs 5).

Discussion

The process evaluation of our multicentre RCT on community occupational therapy in Alzheimer's disease revealed that the characteristics of the German participants at baseline did not mediate patients' daily functioning after treatment as indicated by the mean change to baseline in the IDDD and PRPP. Some subprocesses, which were deemed to be active components of the applied complex psycho-social experimental intervention, were performed poorly. However, variances in the performance were not associated with patients' mean change to baseline in the IDDD and PRPP. The utilisation of further healthcare resources was equal in the experimental and control groups. Based on exploratory analyses of process data, we can reject the hypothesis that group differences in participant characteristics and variances in the treatment performance or the utilisation of further healthcare resources had a confounding influence on the primary outcome within the German study sample. The analyses were limited by the restriction of range in the IDDD baseline data within the German sample. However, the variance of the IDDD baseline data in the German sample was higher than in the Dutch sample (German sample: experimental group, mean 15.4 (SD 9.9); control group, 14.1 (10.1); Dutch sample: experimental group 23.5 (7.9); control group, 24.5 (8.7)).

Using the same eligibility criteria, the German sample showed much less room for improvement in daily functioning than the Dutch sample. In Germany, patients' daily functioning at baseline was much better. Most German patients still performed better at the end of the study irrespective of group assignment than the successfully treated Dutch group, in which patients had lower baseline scores and improved significantly. This underlines the importance to pay attention to the needs of the patients and care givers specific to the target country.

In the German study, the self-reported performance of active intervention components was not associated with the primary outcome. The small sample size and the method of self-rating are limitations for detecting such associations. Although an exploratory regression analysis is vulnerable to misinterpretation,22 we also performed this type of analysis, in order to detect any signs of an influence of treatment performance on the primary outcome. However, we found only minor rates of correlation and explanation, which makes any meaningful association between variances in the treatment performance and the primary outcome unlikely. Self-rating can be a feasible approach for evaluating adherence in dementia research,28 in order to deal with limited resources.29 However, therapists tend to overestimate their own performance.30 Although the therapists were explicitly asked to be critical when judging their own performance, for further studies it is recommended that there be an additional external monitoring of treatment performance. This may reduce possible bias introduced by overestimation or overcriticism in self-rating. Furthermore, it might help to find appropriate onsite coaching strategies. These strategies should aim at high-quality treatment performance, even though the complexity of psycho-social interventions induces variance—especially in multicentre RCTs.

Data on the association of treatment performance and primary patient outcomes, although encouraged,5 31 are scarce in RCTs studying complex interventions. Teri et al28 implemented an external video rating of therapists' adherence to protocol but found no associations between this rating and any outcome variable. Similar studies in the field32 33 did not operationalise the impact of treatment performance on the primary outcome as the British Medical Research Council had strongly recommended.1 Within the original Dutch RCT, the large number of patients treated by two highly motivated therapists from the same study site suggests an excellent quality of treatment performance.2 However, its association with the patients' outcome was not quantified. The experiences with the subsequent Dutch implementation of the guidelines revealed that novices had difficulties in adapting to this highly complex intervention in a concrete treatment setting. Prior to cross-national implementation of complex interventions, a successful national transfer from a single-centre setting with highly motivated specialists to a multicentre routine setting with therapists varying in competence and motivation seems to be appropriate.

Conclusion

Our process evaluation revealed that the participants in the study may have had insufficient need for the applied treatment and that active components of the complex psycho-social intervention were poorly performed. Also, an interaction can be considered in the sense that little need for assistance can make a less intensive, one-session treatment appropriate, as applied to the control group. These recent experiences suggest that cross-national transfers are best prepared by a pilot study in the target country exploring specific patient needs, the feasibility of inclusion criteria, the usability of measurement instruments and the applicability of the complex intervention by therapists in routine care settings.

Supplementary Material

Author's manuscript

Reviewer comments

Acknowledgments

We thank all participants and therapists for their contribution. We acknowledge C Mayers, Research Fellow, Faculty of Health & Life Sciences, York St John University, UK, for critical reading and English correction.

Footnotes

To cite: Voigt-Radloff S, Graff M, Leonhart R, et al. Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT. BMJ Open 2011;1:e000094. doi:10.1136/bmjopen-2011-000094

Funding: German Federal Ministry of Health, Reference Number: IIA5-2508FSB111/44-004.

Competing interests: MOR: consultancy for Jansen-Cilag and Numico. MH: grants from Wyeth, Pfizer and Medivation; payment for lectures from Wyeth, Pfizer and Merz.

Ethics approval: Ethics approval was provided by the Medical Ethics Committee of the University Hospital Freiburg (no 110/08).

Contributors: SVR, MG, RL, MH and MVD contributed to the study conception and design. SVR conducted the data and study management, and prepared the statistical analysis. RL performed the statistical analysis. SVR drafted the manuscript. MG, RL, MH, MOR and MVD revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data sharing statement: Data sets can be provided on request for fellow researchers in the context of collaborative projects and publications.

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Author's manuscript

Reviewer comments