HELENA DAVIES | The Open University (original) (raw)

Papers by HELENA DAVIES

Medical Education, 2009

Objectives This study represents an initial evaluation of the first year (F1) of the Foundation ... more Objectives This study represents an initial evaluation of the first year (F1) of the Foundation Assessment Programme (FAP), in line with Postgraduate Medical Education and Training Board (PMETB) assessment principles.Methods Descriptive analyses were undertaken for total number of encounters, assessors and trainees, mean number of assessments per trainee, mean number of assessments per assessor, time taken for the assessments, mean score and standard deviation for each method. Reliability was estimated using generalisability coefficients. Pearson correlations were used to explore relationships between instruments. The study sample included 3640 F1 trainees from 10 English deaneries.Results A total of 2929 trainees submitted at least one of all four methods. A mean of 16.6 case-focused assessments were submitted per F1 trainee. Based on a return per trainee of six of each of the case-focused assessments, and eight assessors for multi-source feedback, 95% confidence intervals (CIs) ranged between 0.4 and 0.48. The estimated time required for this is 9 hours per trainee per year. Scores increased over time for all instruments and correlations between methods were in keeping with their intended focus of assessment, providing evidence of validity.Conclusions The FAP is feasible and achieves acceptable reliability. There is some evidence to support its validity. Collated assessment data should form part of the evidence considered for selection and career progression decisions although work is needed to further develop the FAP. It is in any case of critical importance for the profession’s accountability to the public.

Advances in Health Sciences Education, 2008

Purpose To design, implement and evaluate a multisource feedback instrument to assess Foundation ... more Purpose To design, implement and evaluate a multisource feedback instrument to assess Foundation trainees across the UK. Methods mini-PAT (Peer Assessment Tool) was modified from SPRAT (Sheffield Peer Review Assessment Tool), an established multisource feedback (360°) instrument to assess more senior doctors, as part of a blueprinting exercise of instruments suitable for assessment in Foundation programmes (first 2 years postgraduation). mini-PAT’s content validity was assured by a mapping exercise against the Foundation Curriculum. Trainees’ clinical performance was then assessed using 16 questions rated against a six-point scale on two occasions in the pilot period. Responses were analysed to determine internal structure, potential sources of bias and measurement characteristics. Results Six hundred and ninety-three mini-PAT assessments were undertaken for 553 trainees across 12 Deaneries in England, Wales and Northern Ireland. Two hundred and nineteen trainees were F1s or PRHOs and 334 were F2s. Trainees identified 5544 assessors of whom 67% responded. The mean score for F2 trainees was 4.61 (SD = 0.43) and for F1s was 4.44 (SD = 0.56). An independent t test showed that the mean scores of these 2 groups were significantly different (t = −4.59, df 390, p < 0.001). 43 F1s (19.6%) and 19 F2s (5.6%) were assessed as being below expectations for F2 completion. The factor analysis produced 2 main factors, one concerned clinical performance, the other humanistic qualities. Seventy-four percent of F2 trainees could have been assessed by as few as 8 assessors (95% CI ±0.6) as they either scored an overall mean of 4.4 or above or 3.6 and below. Fifty-three percent of F1 trainees could have been assessed by as few as 8 assessors (95% CI ±0.5) as they scored an overall mean of 4.5 or above or 3.5 and below. The hierarchical regression when controlling for the grade of trainee showed that bias related to the length of the working relationship, occupation of the assessor and the working environment explained 7% of the variation in mean scores when controlling for the year of the Foundation Programme (R squared change = 0.06, F change = 8.5, significant F change <0.001). Conclusions As part of an assessment programme, mini-PAT appears to provide a valid way of collating colleague opinions to help reliably assess Foundation trainees.

European Journal of Cancer, 2006

Introduction: Follow-up for cancer survivors is recommended to detect recurrence; monitor late-ef... more Introduction: Follow-up for cancer survivors is recommended to detect recurrence; monitor late-effects; record toxicity and provide care and education. We describe our experience with a three-level model developed to guide decisions about intensity and frequency of follow-up [Wallace WHB, Blacklay A, Eiser C, et al. Developing strategies for the long term follow-up of survivors of childhood cancer. BMJ 2001;323:271-274]. Procedure: One hundred and ninety eight survivors (52% male) recruited over 12-months: (mean age = 23.8 years, range = 16-39 years; mean time since diagnosis = 16.2 years, range 2.4-32.7 years) reported their number of symptoms and late-effects. Information was taken from the medical records to assign each survivor to the appropriate levels by six clinic staff independently.

Human Fertility, 2003

This review discusses the most commonly observed problems in young people who have survived child... more This review discusses the most commonly observed problems in young people who have survived childhood cancer, with particular emphasis on the issues of fertility. It also describes the role of the late effects clinic in transitional care and the multidisciplinarity of the clinicians required to meet the needs of these young people. There is a need for evidence-based guidelines for management of long-term survivors.

Pediatric Blood & Cancer, 2004

BackgroundChildhood cancer is rare, but there are now good survival prospects and in the UK appro... more BackgroundChildhood cancer is rare, but there are now good survival prospects and in the UK approximately 1 in 1,000 young adults is a survivor of childhood cancer. There are many adverse health outcomes associated with the treatment of childhood cancer often arising several years after completion of treatment. The aim of this study was to quantify the long-term clinical follow-up practices concerning survivors of childhood cancer.Childhood cancer is rare, but there are now good survival prospects and in the UK approximately 1 in 1,000 young adults is a survivor of childhood cancer. There are many adverse health outcomes associated with the treatment of childhood cancer often arising several years after completion of treatment. The aim of this study was to quantify the long-term clinical follow-up practices concerning survivors of childhood cancer.ProcedureA cross-sectional postal survey of 22 treatment centres of the United Kingdom Children's Cancer Study Group (UKCCSG) clinicians was carried out as well as a cross-sectional postal survey of general practitioners of most adult survivors of childhood cancer in Britain.A cross-sectional postal survey of 22 treatment centres of the United Kingdom Children's Cancer Study Group (UKCCSG) clinicians was carried out as well as a cross-sectional postal survey of general practitioners of most adult survivors of childhood cancer in Britain.ResultsSubsequent to 5 years after the end of treatment: 52% of UKCCSG clinicians follow-up all survivors for life, while 45% discharge some patients. Of those clinicians discharging: over 50% discharged benign, stage I or tumors treated with surgery alone, in contrast 16% reported discharging all or most patients; almost all (97%) clinicians discharged to a general practitioner. Only 14% of clinicians reported nurses undertook a specialist role. Sixty-five percent of the 10,979 general practitioners reported that their patient was not on regular hospital follow-up.Subsequent to 5 years after the end of treatment: 52% of UKCCSG clinicians follow-up all survivors for life, while 45% discharge some patients. Of those clinicians discharging: over 50% discharged benign, stage I or tumors treated with surgery alone, in contrast 16% reported discharging all or most patients; almost all (97%) clinicians discharged to a general practitioner. Only 14% of clinicians reported nurses undertook a specialist role. Sixty-five percent of the 10,979 general practitioners reported that their patient was not on regular hospital follow-up.ConclusionsThere are wide variations in the extent to which survivors of childhood cancer are discharged from hospital follow-up. There is a need for regularly updated national guidelines concerning the levels of follow-up required for specific groups of survivors defined principally by the treatment they received. Pediatr Blood Cancer 2004;42:161–168. © 2003 Wiley-Liss, Inc.There are wide variations in the extent to which survivors of childhood cancer are discharged from hospital follow-up. There is a need for regularly updated national guidelines concerning the levels of follow-up required for specific groups of survivors defined principally by the treatment they received. Pediatr Blood Cancer 2004;42:161–168. © 2003 Wiley-Liss, Inc.

BACKGROUND: There is a lack of evidence regarding current screening practices for incipient ovari... more BACKGROUND: There is a lack of evidence regarding current screening practices for incipient ovarian failure in young women following gonadotoxic therapy and the most appropriate form of estrogen replacement. This study examined the (i) prevalence and management of ovarian failure and (ii) quality-of-life implications of early menopause (EM). METHODS: A medical case note audit for 288 women with a history of gonadotoxic therapy (aged 18-50 years) was conducted. Self-reported quality-of-life data were obtained from 178 (62%). RESULTS: Ovarian screening was recorded in 44% of medical case notes, and ovarian failure was documented for 35%. From the self-reported data, 89/178 (50%) women reported experiencing an EM/ovarian failure. Worse menopausal symptoms were negatively associated with both sexual activity [pleasure (r 5 0.29, P < 0.01), discomfort (r 5 0.50, P < 0.001) and habit (r 5 0.22, P < 0.05)] and general quality of life (P 5 0.01). Hormone replacement therapy is the most commonly prescribed estrogen preparation; however, 34% of women with EM/ovarian failure reported not taking any replacement therapy. CONCLUSIONS: Given the extent and impact of menopausal symptoms, further work is needed to establish systems for screening ovarian function and to determine appropriate and effective management of ovarian failure.

European Journal of Cancer, 2006

Follow-up Clinic Satisfaction A B S T R A C T Childhood cancer survivors experience a wide range ... more Follow-up Clinic Satisfaction A B S T R A C T Childhood cancer survivors experience a wide range of late-effects. As survival rates improve, follow-up in paediatric clinics becomes less feasible, and alternative models of care have been proposed. In this study, satisfaction among those attending a traditional paediatric late-effects clinic was compared with a multi-disciplinary clinic in an adult setting. Survivors (adult clinic n = 93, paediatric clinic n = 105, age 16-39 years) completed measures of symptoms, understanding of vulnerability to late-effects, purpose of followup, satisfaction and number of topics discussed. Predictors of satisfaction were: number of topics discussed, greater understanding of the purpose of follow-up and sex. Females, and those reporting longer waiting time were less satisfied. Aspects of clinic organisation, including shorter waiting times and opportunities to discuss health concerns, are more important in determining patient satisfaction than clinic type. Survivors' understanding of the purpose of follow-up is also integral in determining satisfaction.

Patient Education and Counseling, 2004

We report the evaluation of an information booklet aimed to explain the purpose of follow-up to s... more We report the evaluation of an information booklet aimed to explain the purpose of follow-up to survivors of childhood cancer. Evaluations drew on theoretical concepts in the elaboration likelihood and stages of change models. We predicted that survivors who adopt central rather than peripheral processing would show greater understanding and increased readiness to change health behaviour. Forty-eight survivors were shown an example page of the booklet in the clinic and then completed questionnaires about attitude to clinic, readiness to change behaviour, and the importance and scariness of the information. They were then given the whole booklet and asked to complete a second questionnaire at home. After reading the booklet, survivors reported a more positive attitude to clinic. Survivors using central processing rated information as more important and were more ready to change health behaviour than peripheral processors. We recommend that methods to encourage central processing should be routinely included when providing children with health information.

Psycho-oncology, 2005

Many parents find decisions about what to tell their child with cancer difficult. Open communicat... more Many parents find decisions about what to tell their child with cancer difficult. Open communication is generally considered the best policy and most health care professionals encourage parents to talk openly and honestly about the illness. However, parents differ in their views about what to tell the child. In this study 55 parents of children (36 boys and 19 girls, mean age = 7.33 years) newly diagnosed with acute lymphoblastic leukaemia (ALL) were interviewed about (i) the child's reactions and behaviour following diagnosis, (ii) their views about what to tell their child and (iii) factors influencing parents' communication with the child. Interviews were analysed using thematic analysis. Most children showed behavioural and mood difficulties after diagnosis. Older children were given more information. In addition, parents' perceptions of childhood cancer affect the way they communicate with their child. These findings may be used to inform training packages in order to facilitate improved communication amongst health professionals. Copyright © 2004 John Wiley & Sons, Ltd.

European Journal of Cancer, 2005

Follow-up of survivors of childhood cancer is recommended to improve detection of late-effects, a... more Follow-up of survivors of childhood cancer is recommended to improve detection of late-effects, and provide individuals with information and advice. This study aimed to follow-up survivors of childhood cancer and report on their attitudes to current follow-up methods. Twenty-six survivors (13-25 years) of childhood cancer and their parent(s) attended focus groups (n = 7) to discuss views about follow-up care. Transcripts were analysed using interpretative phenomenological analysis (IPA). Three themes were identified: strategies to achieve a normal life (through playing down possibility of late-effects or careful monitoring of health); expectations about follow-up (facts and information, advice about self-care, everyday living, and psychosocial consequences) and preferences for different models of care. Given that some families had reservations about the benefits of follow-up, it is important that services address survivorsÕ interests and meet their expectations. Changes to service delivery must take account of individual needs and expectations. Possible limitations of focus group methods (recruitment, bias reduction, methods of analysis and influence of other participantsÕ views) are discussed.

Journal of Cancer Survivorship, 2007

Introduction Recent advances in treatment of childhood cancer have resulted in overall survival r... more Introduction Recent advances in treatment of childhood cancer have resulted in overall survival rates approaching 75%, but approximately two-thirds experience late-effects related to the disease or treatment. Consequently, recommendations for comprehensive follow-up have been made. As the number of survivors of adult cancers increase, similar concerns about how to provide follow-up and achieve optimal quality of life are being raised. In this paper we propose that management of cancer survivors diagnosed in young adulthood (18–45 years) could benefit from experience gained treating survivors of childhood cancer. Materials and methods We reviewed research relating to differences in survival rates and late-effects; current arrangements for follow-up; effectiveness; and problems in organization of follow-up separately for survivors of child and adult cancers. Results A number of models of follow-up were identified. Rationale for follow-up included early identification and treatment of second cancer and late-effects, health promotion and screening. Increasing numbers of survivors and range of late-effects were identified as problems in organizing services. A possible solution is risk-stratified follow-up, currently being developed in pediatrics. Conclusions New models of follow-up are needed that take account of financial costs for health services and survivors’ concerns about their current and future health. Implications for continuing refinement of treatment protocols must be an integral part of the service. Implications for cancer survivors International standards are needed to ensure all survivors have access to expert follow-up care and can benefit from new information that might lead to earlier treatment of late-effects.

Medical Education, 2008

Context The white paper ‘Trust, Assurance and Safety: the Regulation of Health Professionals in ... more Context The white paper ‘Trust, Assurance and Safety: the Regulation of Health Professionals in the 21st Century’ proposes a single, generic multi-source feedback (MSF) instrument in the UK. Multi-source feedback was proposed as part of the assessment programme for Year 1 specialty training in histopathology.Methods An existing instrument was modified following blueprinting against the histopathology curriculum to establish content validity. Trainees were also assessed using an objective structured practical examination (OSPE). Factor analysis and correlation between trainees’ OSPE performance and the MSF were used to explore validity. All 92 trainees participated and the assessor response rate was 93%. Reliability was acceptable with eight assessors (95% confidence interval 0.38). Factor analysis revealed two factors: ‘generic’ and ‘histopathology’. Pearson correlation of MSF scores with OSPE performances was 0.48 (P = 0.001) and the histopathology factor correlated more highly (histopathology r = 0.54, generic r = 0.42; t = − 2.76, d.f. = 89, P < 0.01). Trainees scored least highly in relation to ability to use histopathology to solve clinical problems (mean = 4.39) and provision of good reports (mean = 4.39). Three of six doctors whose means were < 4.0 received free text comments about report writing. There were 83 forms with aggregate scores of < 4. Of these, 19.2% included comments about report writing.Results Specialty-specific MSF is feasible and achieves satisfactory reliability. The higher correlation of the ‘histopathology’ factor with the OSPE supports validity. This paper highlights the importance of validating an MSF instrument within the specialty-specific context as, in addition to assuring content validity, the PATH–SPRAT (Histopathology–Sheffield Peer Review Assessment Tool) also demonstrates the potential to inform training as part of a quality improvement model.

Pediatric Blood & Cancer, 2006

BackgroundDexamethasone is increasingly used as the steroid of choice in trials for standard risk... more BackgroundDexamethasone is increasingly used as the steroid of choice in trials for standard risk children with acute lymphoblastic leukemia (ALL). Improvements in event-free survival (EFS) have been attributed to lower CNS relapse rates, However, there are concerns that dexamethasone may be more toxic than previous conventional therapy with prednisone. Such toxicity raises questions about the implications for child neuropsychological function and HRQOL. Patients participating in the UK ALL 99/01 trial were randomized to receive dexamethasone or prednisone as their steroid in induction and maintenance chemotherapy. We compared the HRQOL and behavior in children randomized to receive both these agents.Dexamethasone is increasingly used as the steroid of choice in trials for standard risk children with acute lymphoblastic leukemia (ALL). Improvements in event-free survival (EFS) have been attributed to lower CNS relapse rates, However, there are concerns that dexamethasone may be more toxic than previous conventional therapy with prednisone. Such toxicity raises questions about the implications for child neuropsychological function and HRQOL. Patients participating in the UK ALL 99/01 trial were randomized to receive dexamethasone or prednisone as their steroid in induction and maintenance chemotherapy. We compared the HRQOL and behavior in children randomized to receive both these agents.ProcedureStandardized questionnaires to assess parent and child HRQOL at 3–6 months after diagnosis (T1) and 1 year later (T2) completed by mothers in family homes. Forty-five mothers of a child with ALL (32 male, 13 female; average age at T1, 7 years 3 months; at T2, 8 years 3 months) completed HRQOL questionnaires.Standardized questionnaires to assess parent and child HRQOL at 3–6 months after diagnosis (T1) and 1 year later (T2) completed by mothers in family homes. Forty-five mothers of a child with ALL (32 male, 13 female; average age at T1, 7 years 3 months; at T2, 8 years 3 months) completed HRQOL questionnaires.ResultsFor the total group, child HRQOL scores improved and behavior problems decreased significantly from T1 to T2. Comparison of HRQOL scores between the 17 children randomized to dexamethasone and 28 children randomized to prednisone showed no significant differences. The rate of improvement in HRQOL from T1 to T2 did not differ between children randomized to dexamethasone or prednisone.For the total group, child HRQOL scores improved and behavior problems decreased significantly from T1 to T2. Comparison of HRQOL scores between the 17 children randomized to dexamethasone and 28 children randomized to prednisone showed no significant differences. The rate of improvement in HRQOL from T1 to T2 did not differ between children randomized to dexamethasone or prednisone.ConclusionsDexamethasone is increasingly used in the treatment of ALL and has been linked with improved survival rates. Long-term use of dexamethasone raises questions about neuropsychologic toxicity. Although HRQOL increased significantly over the year for all children, the extent of this increase did not differ by chemotherapy. These results should contribute to lessened concerns about use of dexamethasone in the treatment of ALL. © 2005 Wiley-Liss, Inc.Dexamethasone is increasingly used in the treatment of ALL and has been linked with improved survival rates. Long-term use of dexamethasone raises questions about neuropsychologic toxicity. Although HRQOL increased significantly over the year for all children, the extent of this increase did not differ by chemotherapy. These results should contribute to lessened concerns about use of dexamethasone in the treatment of ALL. © 2005 Wiley-Liss, Inc.

Medical Education, 2005

Context The clinical consultation is an important aspect of the doctor's role. However, there is... more Context The clinical consultation is an important aspect of the doctor's role. However, there is a particular shortage of methods for assessing its quality, and its complexity makes it a considerable assessment challenge.Research question What are the key components of consultations involving children?Methods (1) A content analysis of relevant published and unpublished literature. (2) A nominal group consensus exercise with experienced paediatricians.Results The content analysis and consensus exercise suggested similar lists of doctor's characteristics, tasks and outcomes as being important components of the consultation. Doctor's characteristics include: clinical judgement, clinical knowledge, physical examination, information gathering, clinical questioning, information giving, patient-centredness, parent-centredness, interpersonal skills, and consultation management. Important tasks include: organisation and efficiency, rapport, information gathering, getting the family perspective, examination and procedures, evaluation, medically appropriate plans, family appropriate plans, enhancing understanding and recall, achieving consensus, sharing responsibility, family knows how to get further help and liaison with other relevant health-care professionals. Important outcomes include: family satisfaction, family perceptions, compliance, health, health-related problems and doctor's satisfaction. The studies reviewed in the literature also provided a catalogue of factors that have been shown to influence the doctor–patient interaction that could potentially confound the assessment of a doctor's performance. These include the doctor's: age, gender, training, speciality, income, social class and politics; the patient's: age, gender, health, prognosis, social class, education, health beliefs and preferences about control and risk. The length of the acquaintance between doctor and patient, and the workload and case-mix in the clinic also affect the interaction. In several studies it is clear that particular combinations of doctor-type and patient-type have especially good or bad interactions.Conclusions and further work These components are synthesised in a single model of the doctor–patient interaction to guide the development and evaluation of assessment instruments aimed at consultations involving children.

Medical Education, 2005

Context Only a patient and his or her family can judge many of the most important aspects of the... more Context Only a patient and his or her family can judge many of the most important aspects of the doctor–patient interaction. This study evaluates the feasibility and reliability of children and their families assessing the quality of paediatricians' interactions using a rating instrument developed specifically for this purpose.Methods A reliability analysis using generalisability theory on the ratings from 352 doctor–patient interactions across different speciality clinics.Results Ratings were normally distributed. They were highest for ‘overall’ performance, and lowest for giving time to discuss the families' agenda. An appropriate sample of adults' ratings provided a reliable score (G = 0.7 with 15 raters), but children's ratings were too idiosyncratic to be reproducible (G = 0.36 with 15 raters).Conclusions and further work Accompanying adults can provide reliable ratings of doctors' interactions with children. Because an adult is usually present at the consultation their ratings provide a highly feasible and authentic approach. Sampling doctors' interactions from different clinics and with patients of both genders provides a universal picture of performance. The method is ideal to measure performance for in-training assessment or revalidation. Further work is in progress to evaluate the educational impact of feeding ratings back to the doctors being assessed, and their use in a range of clinical contexts.

Medical Education, 2004

AIM To improve the quality of outpatient letters used as communication between hospital and prima... more AIM To improve the quality of outpatient letters used as communication between hospital and primary care doctors. METHODS On 2 separate occasions, 15 unselected outpatient letters written by each of 7 hospital practitioners were rated by another hospital doctor and a general practitioner (GP) using the Sheffield Assessment Instrument for Letters (SAIL). Individualised feedback was provided to participants following the rating of the first set of letters. The audit cycle was completed 3 months later without forewarning by repeat assessment by the same hospital and GP assessors using the SAIL tool to see if there was any improvement in correspondence. SETTING Single centre: general paediatric outpatient department in a large district general hospital. RESULTS All 7 doctors available for reassessment completed the audit loop, each providing 15 outpatient letters per assessment. The mean of the quality scores, derived for each letter from the summation of a 20-point checklist and a global score, improved from 23.3 (95% CI 22.1-24.4) to 26.6 (95% CI 25.8-27.4) (P ¼ 0.001).

Archives of Disease in Childhood, 2010

To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Too... more To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme. Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study. All UK Deaneries between August 2005 and May 2006. 577 year 2 and 4 Paediatric SpRs. Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores. 4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; or = 4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=-4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77). There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.

The Clinical Teacher, 2005

Multi source feedback (MSF) is a means of assessment based on collated questionnaires from a rang... more Multi source feedback (MSF) is a means of assessment based on collated questionnaires from a range of co-workers and may also include patient feedback. Alternative terminology includes peer assessment, peer review or ratings and 360-degree feedback. There is a ...

Medical Education, 2008

Context The Chief Medical Officer’s recommendations on medical regulation in the UK suggest that... more Context The Chief Medical Officer’s recommendations on medical regulation in the UK suggest that National Health Service (NHS) trusts should assess their doctors and confirm whether they remain fit to practise medicine.Objective We set out to evaluate the utility of hospital trust-based assessment in a ‘best-case scenario’ within existing resources.Methods We carried out a generalisability analysis, and feasibility and validity evaluation, based on an assessment process for 137 career-grade doctors at Chesterfield Royal Hospital, Chesterfield, UK, using validated multi-source feedback (MSF) and patient rating (PR) instruments.Results Uptake and response rates were good for MSF (91% and 85%, respectively). However, only 6% of non-clinical doctors and anaesthetists, and 48% of clinical doctors, obtained sufficient PR ratings. Aggregate scores were acceptably reliable. Nine combined MSF ratings and 15 PR ratings produce standard errors of measurement of 0.19 on a 6-point scale and 0.15 on a 5-point scale, respectively. Overall aggregate scores did not identify any doctor as unsatisfactory, but 6 doctors were scored as unsatisfactory by 2 or more colleagues or patients. These performance concerns appear to merit further investigation. Patients rated female doctors better than male doctors (4.61 versus 4.46; P < 0.05). Colleagues rated UK graduates better than non-UK graduates (5.31 versus 5.15; P < 0.05).Conclusions This study shows that the commissioning of professional services makes the implementation of an assessment process linked to appraisal feasible. However, trust-based assessment requires significant development: developmental appraisal needs protection; new instruments are needed for non-clinical specialties; PR requires specific administrative support, and guidance is required over concern thresholds and demographic effects. Disaggregated assessment data may help identify doctors with potential performance problems.

Medical Education, 2009

Objectives This study represents an initial evaluation of the first year (F1) of the Foundation ... more Objectives This study represents an initial evaluation of the first year (F1) of the Foundation Assessment Programme (FAP), in line with Postgraduate Medical Education and Training Board (PMETB) assessment principles.Methods Descriptive analyses were undertaken for total number of encounters, assessors and trainees, mean number of assessments per trainee, mean number of assessments per assessor, time taken for the assessments, mean score and standard deviation for each method. Reliability was estimated using generalisability coefficients. Pearson correlations were used to explore relationships between instruments. The study sample included 3640 F1 trainees from 10 English deaneries.Results A total of 2929 trainees submitted at least one of all four methods. A mean of 16.6 case-focused assessments were submitted per F1 trainee. Based on a return per trainee of six of each of the case-focused assessments, and eight assessors for multi-source feedback, 95% confidence intervals (CIs) ranged between 0.4 and 0.48. The estimated time required for this is 9 hours per trainee per year. Scores increased over time for all instruments and correlations between methods were in keeping with their intended focus of assessment, providing evidence of validity.Conclusions The FAP is feasible and achieves acceptable reliability. There is some evidence to support its validity. Collated assessment data should form part of the evidence considered for selection and career progression decisions although work is needed to further develop the FAP. It is in any case of critical importance for the profession’s accountability to the public.

Advances in Health Sciences Education, 2008

Purpose To design, implement and evaluate a multisource feedback instrument to assess Foundation ... more Purpose To design, implement and evaluate a multisource feedback instrument to assess Foundation trainees across the UK. Methods mini-PAT (Peer Assessment Tool) was modified from SPRAT (Sheffield Peer Review Assessment Tool), an established multisource feedback (360°) instrument to assess more senior doctors, as part of a blueprinting exercise of instruments suitable for assessment in Foundation programmes (first 2 years postgraduation). mini-PAT’s content validity was assured by a mapping exercise against the Foundation Curriculum. Trainees’ clinical performance was then assessed using 16 questions rated against a six-point scale on two occasions in the pilot period. Responses were analysed to determine internal structure, potential sources of bias and measurement characteristics. Results Six hundred and ninety-three mini-PAT assessments were undertaken for 553 trainees across 12 Deaneries in England, Wales and Northern Ireland. Two hundred and nineteen trainees were F1s or PRHOs and 334 were F2s. Trainees identified 5544 assessors of whom 67% responded. The mean score for F2 trainees was 4.61 (SD = 0.43) and for F1s was 4.44 (SD = 0.56). An independent t test showed that the mean scores of these 2 groups were significantly different (t = −4.59, df 390, p < 0.001). 43 F1s (19.6%) and 19 F2s (5.6%) were assessed as being below expectations for F2 completion. The factor analysis produced 2 main factors, one concerned clinical performance, the other humanistic qualities. Seventy-four percent of F2 trainees could have been assessed by as few as 8 assessors (95% CI ±0.6) as they either scored an overall mean of 4.4 or above or 3.6 and below. Fifty-three percent of F1 trainees could have been assessed by as few as 8 assessors (95% CI ±0.5) as they scored an overall mean of 4.5 or above or 3.5 and below. The hierarchical regression when controlling for the grade of trainee showed that bias related to the length of the working relationship, occupation of the assessor and the working environment explained 7% of the variation in mean scores when controlling for the year of the Foundation Programme (R squared change = 0.06, F change = 8.5, significant F change <0.001). Conclusions As part of an assessment programme, mini-PAT appears to provide a valid way of collating colleague opinions to help reliably assess Foundation trainees.

European Journal of Cancer, 2006

Introduction: Follow-up for cancer survivors is recommended to detect recurrence; monitor late-ef... more Introduction: Follow-up for cancer survivors is recommended to detect recurrence; monitor late-effects; record toxicity and provide care and education. We describe our experience with a three-level model developed to guide decisions about intensity and frequency of follow-up [Wallace WHB, Blacklay A, Eiser C, et al. Developing strategies for the long term follow-up of survivors of childhood cancer. BMJ 2001;323:271-274]. Procedure: One hundred and ninety eight survivors (52% male) recruited over 12-months: (mean age = 23.8 years, range = 16-39 years; mean time since diagnosis = 16.2 years, range 2.4-32.7 years) reported their number of symptoms and late-effects. Information was taken from the medical records to assign each survivor to the appropriate levels by six clinic staff independently.

Human Fertility, 2003

This review discusses the most commonly observed problems in young people who have survived child... more This review discusses the most commonly observed problems in young people who have survived childhood cancer, with particular emphasis on the issues of fertility. It also describes the role of the late effects clinic in transitional care and the multidisciplinarity of the clinicians required to meet the needs of these young people. There is a need for evidence-based guidelines for management of long-term survivors.

Pediatric Blood & Cancer, 2004

BackgroundChildhood cancer is rare, but there are now good survival prospects and in the UK appro... more BackgroundChildhood cancer is rare, but there are now good survival prospects and in the UK approximately 1 in 1,000 young adults is a survivor of childhood cancer. There are many adverse health outcomes associated with the treatment of childhood cancer often arising several years after completion of treatment. The aim of this study was to quantify the long-term clinical follow-up practices concerning survivors of childhood cancer.Childhood cancer is rare, but there are now good survival prospects and in the UK approximately 1 in 1,000 young adults is a survivor of childhood cancer. There are many adverse health outcomes associated with the treatment of childhood cancer often arising several years after completion of treatment. The aim of this study was to quantify the long-term clinical follow-up practices concerning survivors of childhood cancer.ProcedureA cross-sectional postal survey of 22 treatment centres of the United Kingdom Children's Cancer Study Group (UKCCSG) clinicians was carried out as well as a cross-sectional postal survey of general practitioners of most adult survivors of childhood cancer in Britain.A cross-sectional postal survey of 22 treatment centres of the United Kingdom Children's Cancer Study Group (UKCCSG) clinicians was carried out as well as a cross-sectional postal survey of general practitioners of most adult survivors of childhood cancer in Britain.ResultsSubsequent to 5 years after the end of treatment: 52% of UKCCSG clinicians follow-up all survivors for life, while 45% discharge some patients. Of those clinicians discharging: over 50% discharged benign, stage I or tumors treated with surgery alone, in contrast 16% reported discharging all or most patients; almost all (97%) clinicians discharged to a general practitioner. Only 14% of clinicians reported nurses undertook a specialist role. Sixty-five percent of the 10,979 general practitioners reported that their patient was not on regular hospital follow-up.Subsequent to 5 years after the end of treatment: 52% of UKCCSG clinicians follow-up all survivors for life, while 45% discharge some patients. Of those clinicians discharging: over 50% discharged benign, stage I or tumors treated with surgery alone, in contrast 16% reported discharging all or most patients; almost all (97%) clinicians discharged to a general practitioner. Only 14% of clinicians reported nurses undertook a specialist role. Sixty-five percent of the 10,979 general practitioners reported that their patient was not on regular hospital follow-up.ConclusionsThere are wide variations in the extent to which survivors of childhood cancer are discharged from hospital follow-up. There is a need for regularly updated national guidelines concerning the levels of follow-up required for specific groups of survivors defined principally by the treatment they received. Pediatr Blood Cancer 2004;42:161–168. © 2003 Wiley-Liss, Inc.There are wide variations in the extent to which survivors of childhood cancer are discharged from hospital follow-up. There is a need for regularly updated national guidelines concerning the levels of follow-up required for specific groups of survivors defined principally by the treatment they received. Pediatr Blood Cancer 2004;42:161–168. © 2003 Wiley-Liss, Inc.

BACKGROUND: There is a lack of evidence regarding current screening practices for incipient ovari... more BACKGROUND: There is a lack of evidence regarding current screening practices for incipient ovarian failure in young women following gonadotoxic therapy and the most appropriate form of estrogen replacement. This study examined the (i) prevalence and management of ovarian failure and (ii) quality-of-life implications of early menopause (EM). METHODS: A medical case note audit for 288 women with a history of gonadotoxic therapy (aged 18-50 years) was conducted. Self-reported quality-of-life data were obtained from 178 (62%). RESULTS: Ovarian screening was recorded in 44% of medical case notes, and ovarian failure was documented for 35%. From the self-reported data, 89/178 (50%) women reported experiencing an EM/ovarian failure. Worse menopausal symptoms were negatively associated with both sexual activity [pleasure (r 5 0.29, P < 0.01), discomfort (r 5 0.50, P < 0.001) and habit (r 5 0.22, P < 0.05)] and general quality of life (P 5 0.01). Hormone replacement therapy is the most commonly prescribed estrogen preparation; however, 34% of women with EM/ovarian failure reported not taking any replacement therapy. CONCLUSIONS: Given the extent and impact of menopausal symptoms, further work is needed to establish systems for screening ovarian function and to determine appropriate and effective management of ovarian failure.

European Journal of Cancer, 2006

Follow-up Clinic Satisfaction A B S T R A C T Childhood cancer survivors experience a wide range ... more Follow-up Clinic Satisfaction A B S T R A C T Childhood cancer survivors experience a wide range of late-effects. As survival rates improve, follow-up in paediatric clinics becomes less feasible, and alternative models of care have been proposed. In this study, satisfaction among those attending a traditional paediatric late-effects clinic was compared with a multi-disciplinary clinic in an adult setting. Survivors (adult clinic n = 93, paediatric clinic n = 105, age 16-39 years) completed measures of symptoms, understanding of vulnerability to late-effects, purpose of followup, satisfaction and number of topics discussed. Predictors of satisfaction were: number of topics discussed, greater understanding of the purpose of follow-up and sex. Females, and those reporting longer waiting time were less satisfied. Aspects of clinic organisation, including shorter waiting times and opportunities to discuss health concerns, are more important in determining patient satisfaction than clinic type. Survivors' understanding of the purpose of follow-up is also integral in determining satisfaction.

Patient Education and Counseling, 2004

We report the evaluation of an information booklet aimed to explain the purpose of follow-up to s... more We report the evaluation of an information booklet aimed to explain the purpose of follow-up to survivors of childhood cancer. Evaluations drew on theoretical concepts in the elaboration likelihood and stages of change models. We predicted that survivors who adopt central rather than peripheral processing would show greater understanding and increased readiness to change health behaviour. Forty-eight survivors were shown an example page of the booklet in the clinic and then completed questionnaires about attitude to clinic, readiness to change behaviour, and the importance and scariness of the information. They were then given the whole booklet and asked to complete a second questionnaire at home. After reading the booklet, survivors reported a more positive attitude to clinic. Survivors using central processing rated information as more important and were more ready to change health behaviour than peripheral processors. We recommend that methods to encourage central processing should be routinely included when providing children with health information.

Psycho-oncology, 2005

Many parents find decisions about what to tell their child with cancer difficult. Open communicat... more Many parents find decisions about what to tell their child with cancer difficult. Open communication is generally considered the best policy and most health care professionals encourage parents to talk openly and honestly about the illness. However, parents differ in their views about what to tell the child. In this study 55 parents of children (36 boys and 19 girls, mean age = 7.33 years) newly diagnosed with acute lymphoblastic leukaemia (ALL) were interviewed about (i) the child's reactions and behaviour following diagnosis, (ii) their views about what to tell their child and (iii) factors influencing parents' communication with the child. Interviews were analysed using thematic analysis. Most children showed behavioural and mood difficulties after diagnosis. Older children were given more information. In addition, parents' perceptions of childhood cancer affect the way they communicate with their child. These findings may be used to inform training packages in order to facilitate improved communication amongst health professionals. Copyright © 2004 John Wiley & Sons, Ltd.

European Journal of Cancer, 2005

Follow-up of survivors of childhood cancer is recommended to improve detection of late-effects, a... more Follow-up of survivors of childhood cancer is recommended to improve detection of late-effects, and provide individuals with information and advice. This study aimed to follow-up survivors of childhood cancer and report on their attitudes to current follow-up methods. Twenty-six survivors (13-25 years) of childhood cancer and their parent(s) attended focus groups (n = 7) to discuss views about follow-up care. Transcripts were analysed using interpretative phenomenological analysis (IPA). Three themes were identified: strategies to achieve a normal life (through playing down possibility of late-effects or careful monitoring of health); expectations about follow-up (facts and information, advice about self-care, everyday living, and psychosocial consequences) and preferences for different models of care. Given that some families had reservations about the benefits of follow-up, it is important that services address survivorsÕ interests and meet their expectations. Changes to service delivery must take account of individual needs and expectations. Possible limitations of focus group methods (recruitment, bias reduction, methods of analysis and influence of other participantsÕ views) are discussed.

Journal of Cancer Survivorship, 2007

Introduction Recent advances in treatment of childhood cancer have resulted in overall survival r... more Introduction Recent advances in treatment of childhood cancer have resulted in overall survival rates approaching 75%, but approximately two-thirds experience late-effects related to the disease or treatment. Consequently, recommendations for comprehensive follow-up have been made. As the number of survivors of adult cancers increase, similar concerns about how to provide follow-up and achieve optimal quality of life are being raised. In this paper we propose that management of cancer survivors diagnosed in young adulthood (18–45 years) could benefit from experience gained treating survivors of childhood cancer. Materials and methods We reviewed research relating to differences in survival rates and late-effects; current arrangements for follow-up; effectiveness; and problems in organization of follow-up separately for survivors of child and adult cancers. Results A number of models of follow-up were identified. Rationale for follow-up included early identification and treatment of second cancer and late-effects, health promotion and screening. Increasing numbers of survivors and range of late-effects were identified as problems in organizing services. A possible solution is risk-stratified follow-up, currently being developed in pediatrics. Conclusions New models of follow-up are needed that take account of financial costs for health services and survivors’ concerns about their current and future health. Implications for continuing refinement of treatment protocols must be an integral part of the service. Implications for cancer survivors International standards are needed to ensure all survivors have access to expert follow-up care and can benefit from new information that might lead to earlier treatment of late-effects.

Medical Education, 2008

Context The white paper ‘Trust, Assurance and Safety: the Regulation of Health Professionals in ... more Context The white paper ‘Trust, Assurance and Safety: the Regulation of Health Professionals in the 21st Century’ proposes a single, generic multi-source feedback (MSF) instrument in the UK. Multi-source feedback was proposed as part of the assessment programme for Year 1 specialty training in histopathology.Methods An existing instrument was modified following blueprinting against the histopathology curriculum to establish content validity. Trainees were also assessed using an objective structured practical examination (OSPE). Factor analysis and correlation between trainees’ OSPE performance and the MSF were used to explore validity. All 92 trainees participated and the assessor response rate was 93%. Reliability was acceptable with eight assessors (95% confidence interval 0.38). Factor analysis revealed two factors: ‘generic’ and ‘histopathology’. Pearson correlation of MSF scores with OSPE performances was 0.48 (P = 0.001) and the histopathology factor correlated more highly (histopathology r = 0.54, generic r = 0.42; t = − 2.76, d.f. = 89, P < 0.01). Trainees scored least highly in relation to ability to use histopathology to solve clinical problems (mean = 4.39) and provision of good reports (mean = 4.39). Three of six doctors whose means were < 4.0 received free text comments about report writing. There were 83 forms with aggregate scores of < 4. Of these, 19.2% included comments about report writing.Results Specialty-specific MSF is feasible and achieves satisfactory reliability. The higher correlation of the ‘histopathology’ factor with the OSPE supports validity. This paper highlights the importance of validating an MSF instrument within the specialty-specific context as, in addition to assuring content validity, the PATH–SPRAT (Histopathology–Sheffield Peer Review Assessment Tool) also demonstrates the potential to inform training as part of a quality improvement model.

Pediatric Blood & Cancer, 2006

BackgroundDexamethasone is increasingly used as the steroid of choice in trials for standard risk... more BackgroundDexamethasone is increasingly used as the steroid of choice in trials for standard risk children with acute lymphoblastic leukemia (ALL). Improvements in event-free survival (EFS) have been attributed to lower CNS relapse rates, However, there are concerns that dexamethasone may be more toxic than previous conventional therapy with prednisone. Such toxicity raises questions about the implications for child neuropsychological function and HRQOL. Patients participating in the UK ALL 99/01 trial were randomized to receive dexamethasone or prednisone as their steroid in induction and maintenance chemotherapy. We compared the HRQOL and behavior in children randomized to receive both these agents.Dexamethasone is increasingly used as the steroid of choice in trials for standard risk children with acute lymphoblastic leukemia (ALL). Improvements in event-free survival (EFS) have been attributed to lower CNS relapse rates, However, there are concerns that dexamethasone may be more toxic than previous conventional therapy with prednisone. Such toxicity raises questions about the implications for child neuropsychological function and HRQOL. Patients participating in the UK ALL 99/01 trial were randomized to receive dexamethasone or prednisone as their steroid in induction and maintenance chemotherapy. We compared the HRQOL and behavior in children randomized to receive both these agents.ProcedureStandardized questionnaires to assess parent and child HRQOL at 3–6 months after diagnosis (T1) and 1 year later (T2) completed by mothers in family homes. Forty-five mothers of a child with ALL (32 male, 13 female; average age at T1, 7 years 3 months; at T2, 8 years 3 months) completed HRQOL questionnaires.Standardized questionnaires to assess parent and child HRQOL at 3–6 months after diagnosis (T1) and 1 year later (T2) completed by mothers in family homes. Forty-five mothers of a child with ALL (32 male, 13 female; average age at T1, 7 years 3 months; at T2, 8 years 3 months) completed HRQOL questionnaires.ResultsFor the total group, child HRQOL scores improved and behavior problems decreased significantly from T1 to T2. Comparison of HRQOL scores between the 17 children randomized to dexamethasone and 28 children randomized to prednisone showed no significant differences. The rate of improvement in HRQOL from T1 to T2 did not differ between children randomized to dexamethasone or prednisone.For the total group, child HRQOL scores improved and behavior problems decreased significantly from T1 to T2. Comparison of HRQOL scores between the 17 children randomized to dexamethasone and 28 children randomized to prednisone showed no significant differences. The rate of improvement in HRQOL from T1 to T2 did not differ between children randomized to dexamethasone or prednisone.ConclusionsDexamethasone is increasingly used in the treatment of ALL and has been linked with improved survival rates. Long-term use of dexamethasone raises questions about neuropsychologic toxicity. Although HRQOL increased significantly over the year for all children, the extent of this increase did not differ by chemotherapy. These results should contribute to lessened concerns about use of dexamethasone in the treatment of ALL. © 2005 Wiley-Liss, Inc.Dexamethasone is increasingly used in the treatment of ALL and has been linked with improved survival rates. Long-term use of dexamethasone raises questions about neuropsychologic toxicity. Although HRQOL increased significantly over the year for all children, the extent of this increase did not differ by chemotherapy. These results should contribute to lessened concerns about use of dexamethasone in the treatment of ALL. © 2005 Wiley-Liss, Inc.

Medical Education, 2005

Context The clinical consultation is an important aspect of the doctor's role. However, there is... more Context The clinical consultation is an important aspect of the doctor's role. However, there is a particular shortage of methods for assessing its quality, and its complexity makes it a considerable assessment challenge.Research question What are the key components of consultations involving children?Methods (1) A content analysis of relevant published and unpublished literature. (2) A nominal group consensus exercise with experienced paediatricians.Results The content analysis and consensus exercise suggested similar lists of doctor's characteristics, tasks and outcomes as being important components of the consultation. Doctor's characteristics include: clinical judgement, clinical knowledge, physical examination, information gathering, clinical questioning, information giving, patient-centredness, parent-centredness, interpersonal skills, and consultation management. Important tasks include: organisation and efficiency, rapport, information gathering, getting the family perspective, examination and procedures, evaluation, medically appropriate plans, family appropriate plans, enhancing understanding and recall, achieving consensus, sharing responsibility, family knows how to get further help and liaison with other relevant health-care professionals. Important outcomes include: family satisfaction, family perceptions, compliance, health, health-related problems and doctor's satisfaction. The studies reviewed in the literature also provided a catalogue of factors that have been shown to influence the doctor–patient interaction that could potentially confound the assessment of a doctor's performance. These include the doctor's: age, gender, training, speciality, income, social class and politics; the patient's: age, gender, health, prognosis, social class, education, health beliefs and preferences about control and risk. The length of the acquaintance between doctor and patient, and the workload and case-mix in the clinic also affect the interaction. In several studies it is clear that particular combinations of doctor-type and patient-type have especially good or bad interactions.Conclusions and further work These components are synthesised in a single model of the doctor–patient interaction to guide the development and evaluation of assessment instruments aimed at consultations involving children.

Medical Education, 2005

Context Only a patient and his or her family can judge many of the most important aspects of the... more Context Only a patient and his or her family can judge many of the most important aspects of the doctor–patient interaction. This study evaluates the feasibility and reliability of children and their families assessing the quality of paediatricians' interactions using a rating instrument developed specifically for this purpose.Methods A reliability analysis using generalisability theory on the ratings from 352 doctor–patient interactions across different speciality clinics.Results Ratings were normally distributed. They were highest for ‘overall’ performance, and lowest for giving time to discuss the families' agenda. An appropriate sample of adults' ratings provided a reliable score (G = 0.7 with 15 raters), but children's ratings were too idiosyncratic to be reproducible (G = 0.36 with 15 raters).Conclusions and further work Accompanying adults can provide reliable ratings of doctors' interactions with children. Because an adult is usually present at the consultation their ratings provide a highly feasible and authentic approach. Sampling doctors' interactions from different clinics and with patients of both genders provides a universal picture of performance. The method is ideal to measure performance for in-training assessment or revalidation. Further work is in progress to evaluate the educational impact of feeding ratings back to the doctors being assessed, and their use in a range of clinical contexts.

Medical Education, 2004

AIM To improve the quality of outpatient letters used as communication between hospital and prima... more AIM To improve the quality of outpatient letters used as communication between hospital and primary care doctors. METHODS On 2 separate occasions, 15 unselected outpatient letters written by each of 7 hospital practitioners were rated by another hospital doctor and a general practitioner (GP) using the Sheffield Assessment Instrument for Letters (SAIL). Individualised feedback was provided to participants following the rating of the first set of letters. The audit cycle was completed 3 months later without forewarning by repeat assessment by the same hospital and GP assessors using the SAIL tool to see if there was any improvement in correspondence. SETTING Single centre: general paediatric outpatient department in a large district general hospital. RESULTS All 7 doctors available for reassessment completed the audit loop, each providing 15 outpatient letters per assessment. The mean of the quality scores, derived for each letter from the summation of a 20-point checklist and a global score, improved from 23.3 (95% CI 22.1-24.4) to 26.6 (95% CI 25.8-27.4) (P ¼ 0.001).

Archives of Disease in Childhood, 2010

To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Too... more To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme. Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study. All UK Deaneries between August 2005 and May 2006. 577 year 2 and 4 Paediatric SpRs. Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores. 4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; or = 4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=-4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77). There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.

The Clinical Teacher, 2005

Multi source feedback (MSF) is a means of assessment based on collated questionnaires from a rang... more Multi source feedback (MSF) is a means of assessment based on collated questionnaires from a range of co-workers and may also include patient feedback. Alternative terminology includes peer assessment, peer review or ratings and 360-degree feedback. There is a ...

Medical Education, 2008

Context The Chief Medical Officer’s recommendations on medical regulation in the UK suggest that... more Context The Chief Medical Officer’s recommendations on medical regulation in the UK suggest that National Health Service (NHS) trusts should assess their doctors and confirm whether they remain fit to practise medicine.Objective We set out to evaluate the utility of hospital trust-based assessment in a ‘best-case scenario’ within existing resources.Methods We carried out a generalisability analysis, and feasibility and validity evaluation, based on an assessment process for 137 career-grade doctors at Chesterfield Royal Hospital, Chesterfield, UK, using validated multi-source feedback (MSF) and patient rating (PR) instruments.Results Uptake and response rates were good for MSF (91% and 85%, respectively). However, only 6% of non-clinical doctors and anaesthetists, and 48% of clinical doctors, obtained sufficient PR ratings. Aggregate scores were acceptably reliable. Nine combined MSF ratings and 15 PR ratings produce standard errors of measurement of 0.19 on a 6-point scale and 0.15 on a 5-point scale, respectively. Overall aggregate scores did not identify any doctor as unsatisfactory, but 6 doctors were scored as unsatisfactory by 2 or more colleagues or patients. These performance concerns appear to merit further investigation. Patients rated female doctors better than male doctors (4.61 versus 4.46; P < 0.05). Colleagues rated UK graduates better than non-UK graduates (5.31 versus 5.15; P < 0.05).Conclusions This study shows that the commissioning of professional services makes the implementation of an assessment process linked to appraisal feasible. However, trust-based assessment requires significant development: developmental appraisal needs protection; new instruments are needed for non-clinical specialties; PR requires specific administrative support, and guidance is required over concern thresholds and demographic effects. Disaggregated assessment data may help identify doctors with potential performance problems.