Review of Existing Evaluations of Child Representation Download PDF
Overview and Purpose
Our review of Evaluations of Child Representation provides insights into how evaluations have been carried out and what they have found. There were two interrelated goals for conducting an evaluation review as part of the Needs Assessment. The first objective of the evaluation review was to develop a better understanding of the range of topics that had previously been studied in the field of child representation and the general findings associated with these studies. The second objective focused on understanding the range of methodological approaches involved with each of these studies. Findings from this review were used when considering promising approaches for improving child representation in the research and demonstration (R&D) sites and in developing an evaluation plan. The focus of this review was to address the following questions:
· What topics within the field of child representation were previously researched?
· What types of research designs were utilized?
· What were the general findings from the previous research?
The primary findings from this review are the following:
· The majority of evaluations have focused on comparing “who delivers” child representation and attendant
measurement of impact. Specifically the impact of using CASAs has been examined, as well as the role of
private attorneys, staff attorneys, law students, and lay volunteers (not CASA). Although the R&D sites will
not use CASA as the primary representative for children,[1] we reviewed these evaluations for relevant
information on outcomes, measures, and methods.
· Less commonly evaluated subjects include those elements more closely related to the 1996 ABA standards;
namely, the impact of caseload standards and training. Data collection and analysis of stakeholder perceptions
and attitudes has also received relatively less attention.
· Reflective of the difficulty of carrying out experimental design of systemic court interventions (the most rigorous
evaluation design involving random assignment), the most common type of evaluation design utilized was
quasi-experimental. Both historical and same-time comparisons were made between treatment and non-treatment
groups.
· Only 14 evaluations were found in this area, pointing to the need for additional research.
The following presents our approach and findings to the evaluation review.
Methodology
Research articles and evaluations specific to topics of child representation were gathered through a literature search. Members of the project’s Advisory Committee were also asked to suggest or provide any articles or evaluations they thought would be helpful. Each of these articles and evaluations were reviewed and information was extracted into templates for further analysis. Specifically, evaluations and articles were reviewed for the following pieces of information:
· Topic or focus of article
· Target population
· Description of program/intervention
· Overall research design
· Research question
· Variables being measured
· Data sources
· Findings
· Limitations to the study
Information was reviewed and synthesized to address each of the three main questions. The following table provides a listing of all of the evaluations that were included in this review.
|
Evaluation/Research Article
|
Topic/Focus
|
Research Design
|
|
|
Court-appointed special advocates (CASA)
|
Experimental Design
|
|
|
Court-appointed special advocates (CASA)
|
Descriptive
|
|
|
Court-appointed special advocates (CASA)
|
Quasi-experimental
|
|
|
Court-appointed special advocates (CASA)
|
Quasi-experimental
|
|
|
Guardian ad litem
|
Descriptive
|
|
|
Type of representation and training
|
Quasi-experimental
|
|
|
Type of representation
|
Quasi-experimental
|
|
|
Guardian ad litem project in North Dakota
|
Descriptive
|
|
|
Caseloads
|
Descriptive
|
|
|
Court-appointed special advocates (CASA)
|
Quasi-experimental
|
|
|
Guardian ad litem statute in Michigan
|
Descriptive
|
|
|
Guardian ad litem system in Nebraska
|
Descriptive
|
|
|
Guardian ad litem
|
Descriptive
|
|
|
Legal Aid’s Foster Children’s Project
|
Quasi-experimental
|
A limited number of evaluations and research articles were identified for this review, illustrating the lack of research conducted in the area of child representation. A total of 14 evaluations/articles were included in this review, focusing on the following areas:
Court Appointed Special Advocates (CASA)
A total of 5 studies examined the role of a CASA as a child representative. Four studies utilized an experimental or a quasi-experimental design to examine the effectiveness of CASA in meeting intermediate and long term goals in the child welfare case. It should be noted that only one study (Abramson, 1991) of the 14 reviewed studies focused on working with a specific target populations. In the Abramson study minority volunteers were trained and matched with minority children.
Guardian Ad Litem (GAL)
Five studies focused on a GAL[2] program or system. Four of these studies included evaluations that examined various components of utilizing a GAL, and the fifth study focused primarily on the child’s perspective of their GAL.
Types of Representation
Two studies examined the differences between different types of representation. The Goodman study compared county attorneys to independent attorneys while the Duquette and Ramsey study examined lay volunteers, law students and attorneys.
Training
In addition to examining the type of representation, the Duquette/Ramsey study also addressed the topic of training, by providing each type of representative with training and comparing outcomes to child representatives who did not receive training.
Caseloads
One study from California focused on gathering information from attorneys to best understand appropriate caseload. It should also be noted that the Zinn & Slowriver study examined the Foster Children’s Project in Palm Beach County, a child representation model that includes low caseloads. However, this specific variable was not tested to identify the impact directly related to lower caseloads.
Specialized Projects
One study examined the impact of the Foster Children’s Project (FPC) in Palm Beach County. At the time of the study the FCP model included ten attorneys, two permanency planners, and a number of other support personnel. As noted previously attorneys involved in this project carried a lower than average caseload of 35 cases.
Experimental Design
In the Abramson study random assignment was used to assign cases (that met a certain criteria) to either the treatment group who received a volunteer or to the control group who did not receive a volunteer. This was the only study that used this rigorous design.
Quasi-Experimental Design
A total of six studies used a quasi-experimental design, meaning that random assignment into a comparison group was not included in the overall design. These studies did attempt to identify equivalent comparison groups and all of them used a “between group analysis” to identify differences between the treatment and comparison groups. In the Zinn & Slowriver study, age was the only significant difference with the comparison group being slightly older in age. Two studies found that the treatment groups who were receiving CASA services had cases with a higher level of severity. This type of finding could be expected if CASA services are targeted for cases that are more complex.
Descriptive Design
Seven studies used a descriptive design to examine a variety of research questions. It should also be noted that some of the studies that are categorized as an experimental or quasi-experimental design also included descriptive data to supplement outcome findings that resulted from between group analysis. The descriptive designs utilized a constellation of methods to obtain data including interviews, focus groups, surveys and gathering descriptive data from files and records.
Several outcomes were examined in the one experimental and six quasi-experimental studies. This section includes a summary of the most commonly addressed outcomes across the seven studies.
Recidivism
Two studies that examined the role of CASA, the Abramson study and the Caliber Associates study, included this long-term outcome as part of their analysis, however neither study found a significant difference between the treatment and comparison groups in terms of the number of new petitions being reported after the dismissal of a case. Although the sample size of dismissed cases was small, the findings from the Abramson study did indicate that more families in the comparison did have a new petition after case dismissal and that this finding did approach statistical significance.
Services for Children
Both the Caliber Associates and Litzelfelner studies found that children with a CASA volunteer received significantly more services than the comparison group. The Caliber Associates study specified that children with a CASA volunteer were more likely to receive mental health services and medical services.
Placement Stability
Three CASA studies (Caliber Associates, Calkins & Millar, and Litzelfelner) examined the number of placements. Two of these studies found that children with CASA had significantly fewer placements than children without CASA. One study found that there was not a significant difference between the treatment and comparison group. In the Goodman et al. study the mean number of placements for children was higher when the children were represented by county affiliated attorneys than by independent attorneys.
Two studies took a slightly different approach to understanding placement. In the Duquette and Ramsey study, researchers examined the types of placement to see if those representatives who were trained were more likely to have more home placements and fewer court orders for foster care placement. Since a series of additional data was collected on the representatives’ attitudes following training, the findings indicate that a representative who scored high on an Investigation-Interaction scale were more likely to have orders relating to both home and placement and other types of placement. This analysis suggests that it is not simply the training that leads to changes in orders for placement but the representatives’ actions in investigating the case and interacting with other professionals that are correlated to higher court orders for placement.
As noted previously, in the Litzelfelner study number of placements were examined as part of the analysis but researchers in this study recognized that it is important to understand why a child would be moved and if that move was a positive or negative move for the child. In comparing CASA cases and non CASA cases it was found that there were no significant differences in the number of children who experienced positive or negative moves.
Rate of Reunification
Two studies examining the role of CASA collected data specific to the number of children who were reunified with their parents. The findings for this outcome differed between the two studies. In the Caliber Associates study, those who had a CASA volunteer were significantly less likely to have been reunified. In contrast, the Calkins and Millar studies that for those cases that achieved permanency, children with a CASA were reunified more often than those without a CASA. This finding approached (but did not achieve) significance.
Court Process Timeframes
Two studies collected data specific to certain timeframes within the court process. In the Zinn & Slowriver study that looked at a Specialized GAL project, no significant differences in the length of time between removal and adjudication and adjudication and case plan approval were found between the treatment and comparison group. In the Duquette and Ramsey study representatives who had participated in training and scored high on an Advocacy Scale reached first disposition significantly faster than the comparison group.
Overall Timeliness
Timeliness was measured in different ways across the studies including: time to permanency, length of stay in care and length of time children were in court jurisdiction.
· Time to permanency: In the Zinn and Slowriver study, children in the specialized project were found to exit
to permanency faster than comparison children.
· Length of stay in foster care: Calkins & Millar found that cases with CASA experienced a significantly shorter
time in foster care than the comparison group. In the Goodman et al. study, children in counties where independent
representation was used tended to remain in foster care longer than children in counties employing county affiliated
types of representation, this finding approached (but did not achieve) significance.
· Length of time under court jurisdiction: There were no significant differences in terms of length of time between
the comparison group and the group that received CASA.
Additional Measures
Several other measures were less commonly measured across studies however they are reported in this summary because the findings provide valuable insight into further understanding the outcomes affected by child representation.
· Attitude and reported activities: As previously mentioned, the Duquette and Ramsey study collected data specific
to the representatives’ attitude and their actions regarding investigation, interaction and advocacy through interviews
with the representatives. The representatives who received training scored higher on a number of measures that
indicated that they were more likely to investigate their cases thoroughly, be more involved, and be better advocate
for their client.
· Number of continuances: In the Litzelfelner study looking at the role of CASA, there was no significant difference
in the number of continuances between the treatment and comparison group.
· Case closure rates: In the CASA study conducted by Litzelfelner there were no significant differences in the number
of cases closed between the treatment and comparison group.
· Orders for visitation: Findings from the Duquette and Ramsey study indicated that representatives that participated
in the training and scored high on an Investigation/Interaction scale were more likely to have orders related to visitation.
· Developmental measures: The Caliber Associates study utilized 16 developmental measures but found a significant
difference in only one measure examining support for adolescents. Findings indicated that adolescents who had a
CASA volunteer reported slightly less adult support than adolescents who did not have a CASA volunteer.
Stakeholder Perceptions
Most of the descriptive studies collected data from multiple stakeholders to capture several different points of view. This approach allowed researchers to triangulate their data and gather multiple perspectives in specific areas of interest. The data was collected through different methods including: focus groups, interviews and surveys. The following table includes a summary of stakeholders and a general summary of major findings. It should be noted that one quasi-experimental design study (Zinn & Slowriver) also included data from stakeholder perceptions and is also included in this table.
|
Child Representation Evaluations
|
|
Study
|
Stakeholders from which
Information was Collected
|
Findings
|
|
|
Attorneys/
GALS
|
CASA/
volunteers
|
Project/Program
Admin/Staff
|
Judges
|
Child Welfare
Staff
|
Other Professionals
|
Youth
|
Parents/Foster Parents
|
|
|
Berliner, L., Fitzgerald, M. and Alving, M. (1998)
|
|
X
|
X
|
X
|
X
|
X
|
|
|
Professionals who work with volunteers felt that the volunteers contribute an independent and valuable perspective during the investigation phase.
|
|
CSR, Inc (1995).
|
X
|
X
|
|
|
X
|
|
|
|
CASA visited clients more than GALs and had the most contacts with clients, and in court were more likely to submit a written report. GALs were successful in negotiating an agreement 66% of the time.
Caseworkers were contacted most frequently by CASA as opposed to staff or private attorneys.
|
|
Hess, C., Swanke, S. and Batson, A. (2007).
|
X
|
|
X
|
X
|
X
|
|
|
|
Across stakeholder groups there was agreement that the NDGAL Project was useful in meeting the needs it was designed to address and that children and families are better served because of the GAL.
One specific finding across stakeholders was the common perception that the personal knowledge that GALs have about children and the quality of information they provide are helpful to the court-related decision making.
|
|
Lukowski, G.A and Davies, H.J. (2002).
|
|
|
|
|
X
|
|
|
X
|
Approximately half of the judges indicated that since the new legislation there has been no change in the quality of information attorneys provide to the court. In addition, a third to one-half of the foster parents did not feel that the attorneys had enough information. In general it appeared that GALs were not routinely requesting, school, and health and delinquency records.
Although attorneys reported that they do conduct an investigation more than half of caseworkers felt that they rely on the caseworker for information, and 40% said that they attorney never visit the child in his home.
Inconsistent perceptions were reported from foster parents on whether attorneys advocate appropriate visitation.
Trainings appeared to be inconsistent and unstructured.
|
|
Pitchal, E.S., Freundlich, M.D., Kendrick, C. (2009).
|
X
|
|
|
X
|
|
X
|
X
|
|
Attorneys and judge felts that caseloads were adequate.
Although most stakeholders agreed that GALs are promptly appointed, youth reported that did not meet their GALs for a very long time.
GALs felt that training they received did not provide them with everything they needed to serve as a GAL.
Most GALs reported that they only “sometimes” have direct communication with children’s teachers, and that the GALs knew very little about the grades, school transfers or attendance of most youth. Furthermore, youth did not feel that they met with their GAL frequently or for very long when they did meet.
|
|
Stotzel, M. and Fegert, J.M. (2006).
|
X
|
|
|
|
|
|
X
|
|
A majority of the children held an appropriate understanding of their guardian’s role and felt satisfied with their guardian. A smaller portion of children felt either mixed feelings or negative feelings toward their guardian.
|
|
Zinn, A. E. & Slowriver, J. (2008)
|
X
|
|
|
X
|
X
|
X
|
X
|
X
|
Most youth felt that they did have control over the events and outcomes that occurred in court but felt that having an advocate in court with them helped to alleviate the anxiety of the court process.
|
Additional Descriptive Data
Several studies, including those identified as quasi-experimental, collected quantitative data on general process components. This included a range of measures pertaining to the child representative’s such as actions or activities in and out of the court room, the time related to certain activities, the number and types of trainings that were attended, the number of caseloads, cost effectiveness, rate of compensation and the time to appointment. This data was often collected to track program implementation or the work of the child representative. In the Judicial Council of California caseload study, attorneys used self-report to track their hours and activities to prove data for developing a proposed caseload number.
In general, studies varied in their methodological approach and selection of outcomes. CASA studies showed positive results in the services provided to children, but findings were more mixed for rate of reunification, number of placements and timeliness. Findings were also mixed for measures of court process timeframes depending on the intervention, where positive findings were noted in the Duquette & Ramsey (for child representatives that had high Advocacy scores), but not in the Zinn & Slowriver study (which examined outcomes from their specialized GAL Project). One emerging theme from the qualitative data is the need for more structure and consistency in terms of child representative activities especially related to their gaining knowledge of the child. Both evaluations that collected youth perceptions in the United States (Zinn and Slowriver and Pitchal et al.), showed that the children either felt that their representative knew them very well or felt that as children, they did not have control over the outcomes, pointing to the potential importance of an effective representative. Due to the limited number of studies and the varying topic of focus it is challenging to generalize findings without additional research.
[1] Applicant R&D sites could propose lawyer/CASA teams with both members providing case activities associated with representation.
[2] The term “Guardian Ad Litem” was defined differently by sites and studies in terms of using attorneys or lay people, and whether they strictly adhered to a best interests of the child model of representation.