Validation of expert assessment of occupational exposures (original) (raw)
Related papers
Self-assessed Versus Expert-assessed Occupational Exposures
American Journal of Epidemiology, 1996
While self-response to a checklist of substances may be a convenient and inexpensive method for obtaining information on occupational exposure, the validity of such information has not been evaluated. The objective of this report is to provide some evidence concerning validity of self-reported occupational exposures. In the context of a large case-control study, it was possible to compare self-reports with expert assessment, in which a team of industrial hygienists and chemists examined each job history individually and decided on likelihood of exposure. The subjects were 1,910 males who had participated in a population-based case-control study of cancer and occupational exposures conducted in Montreal, Canada, between 1979 and 1985. For each of 11 substances, the two methods of exposure assessment were compared by means of a kappa statistic and by computing the sensitivity and specificity of self-assessment against expert assessment. Kappa values ranged from 0.33 to 0.64. Compared with the expert assessment, specificities of the self-assessment were generally high (0.83-0.97, with a median of 0.90), but sensitivities were low (0.39-0.91, with a median of 0.61). The authors conclude that self-reports of occupational exposure are not sufficiently accurate to warrant their sole use in most community-based studies.
Assessment of occupational exposures in a general population: comparison of different methods
Occupational and Environmental Medicine, 1999
Objectives-To evaluate the relative merits of job specific questionnaires and various alternative assessment methods of occupational exposures often used in general population studies. Methods-Subjects were participants in a hospital based case-control study of risk factors for male infertility. Estimates of exposure to organic solvents and chromium, based on job specific questionnaires, generic questionnaires, self reports of exposure, an external job exposure matrix (JEM), and a population specific JEM were compared with passive diVuse dosimeter results and measurements in urine. Urine samples from the end of the shift were analysed for metabolites of toluene, xylene, several glycol ethers, trichloroethylene, and chromium. Passive dosimeter date, metabolites of specific solvents, and urinary chromium concentrations were available for 89, 267, and 156 subjects, respectively. The alternative methods and measurements in urine were compared by means of the Cohen's statistic and by computing the positive predictive value, sensitivity, and specificity of the alternative methods against measurements in urine. Results-Passive dosimeter results indicated that exposure classifications with job specific questionnaire information could discriminate between high and low exposures. The coeYcients were <0.4, so agreement between the various methods and measurements in urine was poor. Sensitivity of the methods ranged from 0.21 to 0.85, whereas specificity ranged from 0.34 to 0.94. Positive predictive values ranged from 0.19 to 0.58, with the highest values for job specific questionnaires.
Annals of Occupational Hygiene, 2012
Results are presented from a study that investigated the effect of characteristics of occupational hygienists relating to educational and professional experience and task-specific experience on the accuracy of occupational exposure judgments. A total of 49 occupational hygienists from six companies participated in the study and 22 tasks were evaluated. Participating companies provided monitoring data on specific tasks. Information on nine educational and professional experience determinants (e.g. educational background, years of occupational hygiene and exposure assessment experience, professional certifications, statistical training and experience, and the 'need for cognition (NFC)', which is a measure of an individual's motivation for thinking) and four task-specific determinants was also collected from each occupational hygienist. Hygienists had a wide range of educational and professional backgrounds for tasks across a range of industries with different workplace and task characteristics. The American Industrial Hygiene Association exposure assessment strategy was used to make exposure judgments on the probability of the 95th percentile of the underlying exposure distribution being located in one of four exposure categories relative to the occupational exposure limit. After reviewing all available job/task/chemical information, hygienists were asked to provide their judgment in probabilistic terms. Both qualitative (judgments without monitoring data) and quantitative judgments (judgments with monitoring data) were recorded. Ninety-three qualitative judgments and 2142 quantitative judgments were obtained. Data interpretation training, with simple rules of thumb for estimating the 95th percentiles of lognormal distributions, was provided to all hygienists. A data interpretation test (DIT) was also administered and judgments were elicited before and after training. General linear models and cumulative logit models were used to analyze the relationship between accuracy of judgments and the various characteristics describing the participants. Data interpretation training (P < 0.0001), the company that the hygienist worked for (P < 0.0001), the total number of years hygienists had experience doing exposure assessments (P < 0.0001), and professional certifications (P < 0.0001) held by hygienists were found to be significant determinants of accurately predicting the correct exposure category for DITs as well as for task-specific judgment accuracy. Years of experience with a particular task (P < 0.0001), task evaluated, and the number of datapoints used for making judgments were found to be significant predictors of task-specific judgment accuracy. The NFC score was a predictor of the improvement in task judgment accuracy after training. The NFC score was itself predicted by determinants, such as company, years on current job, years of exposure assessment experience, and professional certifications. The results of this study are relevant not only for the case of industrial hygienists making exposure judgments prospectively but also possibly for those hygienists engaged in retrospective exposure assessments for epidemiological studies.
Determinants of the accuracy of occupational hygiene expert judgment
INDUSTRIAL HEALTH, 2015
An experimental study was performed to determine the applicability and accuracy of occupational hygienist's expert judgment in occupational exposure assessment. The effect of tier 1 model application on improvement of expert judgments were also realized. Hygienists were asked to evaluate inhalation exposure intensity in seven operating units in a tile factory before and after an exposure training session. Participants' judgments were compared to air sampling data in the units; then after relative errors for judgments were calculated. Stepwise regressions were performed to investigate the defining variables. In all situations there were almost a perfect agreement (ICC >0.80) among raters. Correlations between estimated mean exposure and relative percentage error of participants before and after training were significant at 0.01 (correlation coefficients were −0.462 and −0.443, respectively). Results showed that actual concentration and experience resulted in 22.4% prediction variance for expert error as an independent variable. Exposure rating by hygienists was susceptible to error from several sources. Experienced subjects had a better ability to predict the exposures intensity. In lower concentrations, the rating error increased significantly. Leading causes of judgment error should be taken into account in epidemiological studies.
American Journal of Industrial Medicine, 2010
Background Often in exposure assessment for epidemiology, there are no highly accurate exposure data and different measurement methods are considered. The objective of this study was to use various statistical techniques to explore agreement between individual reports and expert ratings of workplace exposures in several industries and investigate the sociodemographic influences on this agreement. Methods A cohort of 1,282 employees at 4 industries/14 worksites answered questions on workplace physical, chemical, and psychosocial exposures over the past 12 months. Occupational hygienists constructed job exposure matrices (JEMs) based on worksite walkthrough exposure evaluations. Worker self-reports were compared with the JEMs using multivariable analyses to explore discord. Results There was poor agreement between the self-reported and expert exposure assessments, but there was evidence that agreement was modified by sociodemographic characteristics. Several characteristics including gender, age, race/ethnicity, hourly wage and nativity strongly affected the degree of discord between self-reports and expert raters across a wide array of different exposures. Conclusions Agreement between exposure assessment tools may be affected by sociodemographic characteristics. This study is cross-sectional and therefore, a snapshot of potential exposures in the workplace. Nevertheless, future studies should take into account the social contexts within which workplace exposures occur.
Occupational Exposure Decisions: Can Limited Data Interpretation Training Help Improve Accuracy?
Annals of Occupational Hygiene, 2009
Accurate exposure assessments are critical for ensuring that potentially hazardous exposures are properly identified and controlled. The availability and accuracy of exposure assessments can determine whether resources are appropriately allocated to engineering and administrative controls, medical surveillance, personal protective equipment and other programs designed to protect workers. A desktop study was performed using videos, task information and sampling data to evaluate the accuracy and potential bias of participants' exposure judgments. Desktop exposure judgments were obtained from occupational hygienists for material handling jobs with small air sampling data sets (0-8 samples) and without the aid of computers. In addition, data interpretation tests (DITs) were administered to participants where they were asked to estimate the 95th percentile of an underlying log-normal exposure distribution from small data sets. Participants were presented with an exposure data interpretation or rule of thumb training which included a simple set of rules for estimating 95th percentiles for small data sets from a log-normal population. DIT was given to each participant before and after the rule of thumb training. Results of each DIT and qualitative and quantitative exposure judgments were compared with a reference judgment obtained through a Bayesian probabilistic analysis of the sampling data to investigate overall judgment accuracy and bias. There were a total of 4386 participant-task-chemical judgments for all data collections: 552 qualitative judgments made without sampling data and 3834 quantitative judgments with sampling data. The DITs and quantitative judgments were significantly better than random chance and much improved by the rule of thumb training. In addition, the rule of thumb training reduced the amount of bias in the DITs and quantitative judgments. The mean DIT % correct scores increased from 47 to 64% after the rule of thumb training (P < 0.001). The accuracy for quantitative desktop judgments increased from 43 to 63% correct after the rule of thumb training (P < 0.001). The rule of thumb training did not significantly impact accuracy for qualitative desktop judgments. The finding that even some simple statistical rules of thumb improve judgment accuracy significantly suggests that hygienists need to routinely use statistical tools while making exposure judgments using monitoring data.
Occupational and Environmental Medicine, 2014
Background Retrospective exposure assessment in community-based studies is largely reliant on questionnaire information. Expert assessment is often used to assess lifetime occupational exposures, but these assessments generally lack transparency and are very time-consuming. We explored the agreement between a rule-based assessment approach and case-by-case expert assessment of occupational exposures in a community-based study. Methods We used data from a case-control study of childhood acute lymphoblastic leukaemia in which parental occupational exposures were originally assigned by expert assessment. Key questions were identified from the completed parent questionnaires and, based on these, rules were written to assign exposure levels to diesel exhaust, pesticides, and solvents. We estimated exposure prevalence separately for fathers and mothers, and used Kappa statistics to assess the agreement between the two exposure assessment methods. Results Exposures were assigned to 5829 jobs among 1079 men and 6189 jobs among 1234 women. For both sexes, agreement was good for the two assessment methods of exposure to diesel exhaust at a job level (κ=0.70 for men and κ=0.71 for women) and at a person level (κ=0.74 and κ=0.75). The agreement was good to excellent for pesticide exposure among men (κ=0.74 for jobs and κ=0.84 at a person level) and women (κ=0.68 and κ=0.71 at a job and person level, respectively). Moderate to good agreement was observed for assessment of solvent exposure, which was better for women than men. Conclusion The rule-based assessment approach appeared to be an efficient alternative for assigning occupational exposures in a community-based study for a selection of occupational exposures.
OccIDEAS: Retrospective occupational exposure assessment in community-based studies made easier
… of environmental and …, 2009
Assessing occupational exposure in retrospective community-based case-control studies is difficult as measured exposure data are very seldom available. The expert assessment method is considered the most accurate way to attribute exposure but it is a time consuming and expensive process and may be seen as subjective, nonreproducible, and nontransparent. In this paper, we describe these problems and outline our solutions as operationalized in a web-based software application (OccIDEAS). The novel aspects of OccIDEAS are combining all steps in the assessment into one software package; enmeshing the process of assessment into the development of questionnaires; selecting the exposure(s) of interest; specifying rules for exposure assignment; allowing manual or automatic assessments; ensuring that circumstances in which exposure is possible for an individual are highlighted for review; providing reports to ensure consistency of assessment. Development of this application has the potential to make high-quality occupational assessment more efficient and accessible for epidemiological studies.