Pamela Buckley | University of Colorado, Boulder (original) (raw)

Papers by Pamela Buckley

Hot spots policing is used by many U.S. police departments and is consistently supported by evide... more Hot spots policing is used by many U.S. police departments and is consistently supported by evidence from evaluations of its effectiveness. Hot spot police work requires identifying small areas where crime is especially concentrated and directing police resources to these areas. This paper meta-analyses 22 reports in which random assignment was used to increase police activities in some hot spots. The analysis finds that the police activities reduced crime by .046 to .051 standard deviation units, with the most plausible estimate being .046. All the average effect sizes were statistically different from zero. However, about a third of the study-specific estimates differed reliably from zero and the moderator variables failed to explain variation in impact. Hot spots policing reduces crime, but its effect is small and inconsistent. We discuss implications for policing theory and practice, and provide suggestions to improve the use of meta-analysis for identifying effective programs a...

Prevention Science

As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementatio... more As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementation (FOI) often wanes. This study explores the association between FOI and malleable variables within classrooms that could be targeted to optimize resources without compromising FOI as school-based EBIs are disseminated across real-world settings. We utilized process evaluation data from a national dissemination project of the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention shown to reduce substance use. The sample included 1,626 teachers in 371 schools across 14 states. Hierarchical linear models examined the relationship between observational measures of implementation factors and three domains of fidelity (e.g., adherence, student responsiveness, and quality of delivery). Findings suggest that curriculum modifications, student misbehavior, and shortage of time to implement the LST middle school program were factors most associated with lower ...

As the field of education and especially gifted education gradually moves towards open science, o... more As the field of education and especially gifted education gradually moves towards open science, our research community increasingly values transparency and openness brought by open science practices. Yet, individual researchers may be reluctant to adopt open science practices due to low incentives, barriers of extra workload, or lack of support to apply these in certain areas, such as qualitative research. We encourage and give guidelines to reviewers to champion open science practices by warmly influencing authors to consider applying open science practices to quantitative, qualitative, and mixed methods research and providing ample support to produce higher-quality publications. Instead of imposing open science practices on authors, we advocate reviewers suggest small, non-threatening, specific steps to support authors without making them feel overwhelmed, judged, or punished. We believe that these small steps taken by reviewers will make a difference to create a more supportive e...

Evaluation and Program Planning

Substance Use & Misuse

Proceedings of the 2020 AERA Annual Meeting, 2020

SSRN Electronic Journal

Objective. Because racial and ethnic minoritized groups disproportionately represented essential ... more Objective. Because racial and ethnic minoritized groups disproportionately represented essential workers and lacked equitable access to resources that mitigated exposure and mortality risk, the COVID-19 pandemic brought disparities to the forefront of public health, exacerbating existing discrepancies. These inequities highlight a pressing need for the prevention science eld to investigate whether interventions promote equitable well-being, which served as the impetus for this study. We examined 885 programs with evaluations published from 2010-2021 and recorded in the Blueprints for Healthy Youth Development registry of preventive interventions. Methods. Focusing on race and ethnicity but also tracking reporting of gender, location, and economic disadvantage, we conducted a descriptive analysis on the prevalence of culturally tailored interventions (i.e., those developed for speci c populations) and reporting of sample characteristics. In addition, inferential analyses examined reporting time trends, as well as the relationship between study quality (i.e., methodological soundness, bene cial effects) and culturally tailored programs, and racial ethnic enrollment. Results. Most studies were conducted in the U.S. (n=583). Two percent of programs were developed for Black or African American youth and four percent targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% were collapsed across racial categories or categorized with ethnicity, thus ignoring the intersectionality of race and ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees identi ed as Hispanic or Latino. Reporting did not improve over time, and there was no relationship between high quality evaluations and programs developed for racial ethnic minority groups, or samples with high proportions of racial ethnic enrollees. Conclusions. Research gaps on racial and ethnic groups identi ed in this study indicate the need for improved representation and clear reporting to reduce disparities and improve the utility of preventive interventions.

Objective. Because racial and ethnic minoritized groups disproportionately represented essential ... more Objective. Because racial and ethnic minoritized groups disproportionately represented essential workers and lacked equitable access to resources that mitigated exposure and mortality risk, the COVID-19 pandemic brought disparities to the forefront of public health, exacerbating existing discrepancies. These inequities highlight a pressing need for the prevention science field to investigate whether interventions promote equitable well-being, which served as the impetus for this study. We examined 885 programs with evaluations published from 2010-2021 and recorded in the Blueprints for Healthy Youth Development registry of preventive interventions. Methods. Focusing on race and ethnicity but also tracking reporting of gender, location, and economic disadvantage, we conducted a descriptive analysis on the prevalence of culturally tailored interventions (i.e., those developed for specific populations) and reporting of sample characteristics. In addition, inferential analyses examined ...

Global Implementation Research and Applications

This study serves to inform Colorado's understanding of concurrent enrollment as a driver of ... more This study serves to inform Colorado's understanding of concurrent enrollment as a driver of efficiency in getting to an on-time postsecondary credential. High school students that participate in the Colorado's concurrent enrollment program may enroll tuition-free in postsecondary courses and earn college credits that are transferable to any Colorado public university.

Prevention Science, 2021

Objective. Transparency of research methods is vital to science, though incentives are variable, ... more Objective. Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. Methods. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-19 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. Results. We found few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates, etc.). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Conclusions. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

Evaluation Review, 2020

This study compares prevention program registries in current use on their level of support for us... more This study compares prevention program registries in current use on their level of support for users seeking to implement evidence-based programs. Despite the importance of registries as intermediaries between researchers and the public, and although previous studies have examined how registries define their standards for methodological soundness and evidence of efficacy, little research has focused on the degree to which registries consider programs’ dissemination readiness. The result is that registry users are uncertain whether listed programs and their necessary support materials are even available for implementation. This study evaluates 11 publicly and privately funded prevention registries that review the evidence base of programs seeking to improve child health and prosocial outcomes on the degree to which they use dissemination readiness as an evidentiary criterion for rating programs, and the extent and type of information they provide about dissemination readiness to supp...

Prevention Science, 2021

Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluati... more Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention “works.” This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. Methods. We conducted a descriptive analysis of 851 RCTs published from 2010-2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically-proven and scalable interventions. We used Blueprints’ evaluation criteria that correspond to a subset of SPR’s standards of evidence. Results. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Conclusions. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

Criminology & Public Policy, 2020

The Journal of Behavioral Health Services & Research, 2005

This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and... more This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and Mental Health Services Administration's (SAMHSA 's) national Women Co-occurring Disorders and Violence Study (WCDVS). According to previous literature, it was hypothesized that women in the WCDVS would be at high risk of perpetrating child abuse. This research examined mothers' potential for physical child abuse and assessed the association between child abuse potential, current mental health symptoms, alcohol and drug use severity, and trauma. Results revealed that participants" had significant potential for child abuse. Hierarchical regression analyses revealed that current mental health symptoms were the strongest predictor of mothers' scores on the Child Abuse Potential (CAP) Inventory. This study highlights" the important relationships between commonly used instruments across the mental health, substance, and child welfare fields and the potential dual use of these instruments. Implications for policy and practice are discussed.

Prevention Science

Training prior to implementing evidence-based interventions (EBIs) is essential to reach high lev... more Training prior to implementing evidence-based interventions (EBIs) is essential to reach high levels of fidelity. However, the time and cost of in-person training are often barriers to implementation. Online learning offers a potential solution, though few studies examine the relationship between online training and fidelity of implementation. This study explored whether teachers trained online have similar levels of adherence, dosage, quality of delivery, and student responsiveness compared to teachers trained in-person on the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention proven to reduce substance use and violence, as part of a national dissemination project. This study involved a sample of 989 LST teachers across 114 school districts, representing 296 schools in 14 states. All teachers were first trained in LST implementation between 2016 and 2019. Hierarchical linear models were used to assess relationships between training modality ...

Prevention Science

A correction to this paper has been published: https://doi.org/10.1007/s11121-021-01250-7

Hot spots policing is used by many U.S. police departments and is consistently supported by evide... more Hot spots policing is used by many U.S. police departments and is consistently supported by evidence from evaluations of its effectiveness. Hot spot police work requires identifying small areas where crime is especially concentrated and directing police resources to these areas. This paper meta-analyses 22 reports in which random assignment was used to increase police activities in some hot spots. The analysis finds that the police activities reduced crime by .046 to .051 standard deviation units, with the most plausible estimate being .046. All the average effect sizes were statistically different from zero. However, about a third of the study-specific estimates differed reliably from zero and the moderator variables failed to explain variation in impact. Hot spots policing reduces crime, but its effect is small and inconsistent. We discuss implications for policing theory and practice, and provide suggestions to improve the use of meta-analysis for identifying effective programs a...

Prevention Science

As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementatio... more As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementation (FOI) often wanes. This study explores the association between FOI and malleable variables within classrooms that could be targeted to optimize resources without compromising FOI as school-based EBIs are disseminated across real-world settings. We utilized process evaluation data from a national dissemination project of the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention shown to reduce substance use. The sample included 1,626 teachers in 371 schools across 14 states. Hierarchical linear models examined the relationship between observational measures of implementation factors and three domains of fidelity (e.g., adherence, student responsiveness, and quality of delivery). Findings suggest that curriculum modifications, student misbehavior, and shortage of time to implement the LST middle school program were factors most associated with lower ...

As the field of education and especially gifted education gradually moves towards open science, o... more As the field of education and especially gifted education gradually moves towards open science, our research community increasingly values transparency and openness brought by open science practices. Yet, individual researchers may be reluctant to adopt open science practices due to low incentives, barriers of extra workload, or lack of support to apply these in certain areas, such as qualitative research. We encourage and give guidelines to reviewers to champion open science practices by warmly influencing authors to consider applying open science practices to quantitative, qualitative, and mixed methods research and providing ample support to produce higher-quality publications. Instead of imposing open science practices on authors, we advocate reviewers suggest small, non-threatening, specific steps to support authors without making them feel overwhelmed, judged, or punished. We believe that these small steps taken by reviewers will make a difference to create a more supportive e...

Evaluation and Program Planning

Substance Use & Misuse

Proceedings of the 2020 AERA Annual Meeting, 2020

SSRN Electronic Journal

Objective. Because racial and ethnic minoritized groups disproportionately represented essential ... more Objective. Because racial and ethnic minoritized groups disproportionately represented essential workers and lacked equitable access to resources that mitigated exposure and mortality risk, the COVID-19 pandemic brought disparities to the forefront of public health, exacerbating existing discrepancies. These inequities highlight a pressing need for the prevention science eld to investigate whether interventions promote equitable well-being, which served as the impetus for this study. We examined 885 programs with evaluations published from 2010-2021 and recorded in the Blueprints for Healthy Youth Development registry of preventive interventions. Methods. Focusing on race and ethnicity but also tracking reporting of gender, location, and economic disadvantage, we conducted a descriptive analysis on the prevalence of culturally tailored interventions (i.e., those developed for speci c populations) and reporting of sample characteristics. In addition, inferential analyses examined reporting time trends, as well as the relationship between study quality (i.e., methodological soundness, bene cial effects) and culturally tailored programs, and racial ethnic enrollment. Results. Most studies were conducted in the U.S. (n=583). Two percent of programs were developed for Black or African American youth and four percent targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% were collapsed across racial categories or categorized with ethnicity, thus ignoring the intersectionality of race and ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees identi ed as Hispanic or Latino. Reporting did not improve over time, and there was no relationship between high quality evaluations and programs developed for racial ethnic minority groups, or samples with high proportions of racial ethnic enrollees. Conclusions. Research gaps on racial and ethnic groups identi ed in this study indicate the need for improved representation and clear reporting to reduce disparities and improve the utility of preventive interventions.

Objective. Because racial and ethnic minoritized groups disproportionately represented essential ... more Objective. Because racial and ethnic minoritized groups disproportionately represented essential workers and lacked equitable access to resources that mitigated exposure and mortality risk, the COVID-19 pandemic brought disparities to the forefront of public health, exacerbating existing discrepancies. These inequities highlight a pressing need for the prevention science field to investigate whether interventions promote equitable well-being, which served as the impetus for this study. We examined 885 programs with evaluations published from 2010-2021 and recorded in the Blueprints for Healthy Youth Development registry of preventive interventions. Methods. Focusing on race and ethnicity but also tracking reporting of gender, location, and economic disadvantage, we conducted a descriptive analysis on the prevalence of culturally tailored interventions (i.e., those developed for specific populations) and reporting of sample characteristics. In addition, inferential analyses examined ...

Global Implementation Research and Applications

This study serves to inform Colorado's understanding of concurrent enrollment as a driver of ... more This study serves to inform Colorado's understanding of concurrent enrollment as a driver of efficiency in getting to an on-time postsecondary credential. High school students that participate in the Colorado's concurrent enrollment program may enroll tuition-free in postsecondary courses and earn college credits that are transferable to any Colorado public university.

Prevention Science, 2021

Objective. Transparency of research methods is vital to science, though incentives are variable, ... more Objective. Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. Methods. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-19 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. Results. We found few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates, etc.). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Conclusions. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

Evaluation Review, 2020

This study compares prevention program registries in current use on their level of support for us... more This study compares prevention program registries in current use on their level of support for users seeking to implement evidence-based programs. Despite the importance of registries as intermediaries between researchers and the public, and although previous studies have examined how registries define their standards for methodological soundness and evidence of efficacy, little research has focused on the degree to which registries consider programs’ dissemination readiness. The result is that registry users are uncertain whether listed programs and their necessary support materials are even available for implementation. This study evaluates 11 publicly and privately funded prevention registries that review the evidence base of programs seeking to improve child health and prosocial outcomes on the degree to which they use dissemination readiness as an evidentiary criterion for rating programs, and the extent and type of information they provide about dissemination readiness to supp...

Prevention Science, 2021

Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluati... more Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention “works.” This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. Methods. We conducted a descriptive analysis of 851 RCTs published from 2010-2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically-proven and scalable interventions. We used Blueprints’ evaluation criteria that correspond to a subset of SPR’s standards of evidence. Results. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Conclusions. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

Criminology & Public Policy, 2020

The Journal of Behavioral Health Services & Research, 2005

This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and... more This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and Mental Health Services Administration's (SAMHSA 's) national Women Co-occurring Disorders and Violence Study (WCDVS). According to previous literature, it was hypothesized that women in the WCDVS would be at high risk of perpetrating child abuse. This research examined mothers' potential for physical child abuse and assessed the association between child abuse potential, current mental health symptoms, alcohol and drug use severity, and trauma. Results revealed that participants" had significant potential for child abuse. Hierarchical regression analyses revealed that current mental health symptoms were the strongest predictor of mothers' scores on the Child Abuse Potential (CAP) Inventory. This study highlights" the important relationships between commonly used instruments across the mental health, substance, and child welfare fields and the potential dual use of these instruments. Implications for policy and practice are discussed.

Prevention Science

Training prior to implementing evidence-based interventions (EBIs) is essential to reach high lev... more Training prior to implementing evidence-based interventions (EBIs) is essential to reach high levels of fidelity. However, the time and cost of in-person training are often barriers to implementation. Online learning offers a potential solution, though few studies examine the relationship between online training and fidelity of implementation. This study explored whether teachers trained online have similar levels of adherence, dosage, quality of delivery, and student responsiveness compared to teachers trained in-person on the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention proven to reduce substance use and violence, as part of a national dissemination project. This study involved a sample of 989 LST teachers across 114 school districts, representing 296 schools in 14 states. All teachers were first trained in LST implementation between 2016 and 2019. Hierarchical linear models were used to assess relationships between training modality ...

Prevention Science

A correction to this paper has been published: https://doi.org/10.1007/s11121-021-01250-7