The Impact of Teacher Training on Student Achievement: Quasi-Experimental Evidence from School Reform Efforts in Chicago

I. Introduction

It was made clear in most literatures on relationship between teacher attributes and students quality, that certain teachers are more efficient than others in enhancing students’ performance. However, there is still ambiguity on whether evident teacher attributes like education, or experience would yield greater student performance.

Although most studies are concentrated on broad skills, school districts and states often depend on in-service staff development as a way to enhance student education. Professional development is widely practiced in U.S. Public Schools as well as taking training on means of implementing new teaching methods; however, the intensity of training is rather low .  Furthermore, most of the existing studies on in-service training are at a disadvantage due to the fact that the training is endogenously identified by teachers and schools.

This paper exploits the existence of strict cut-offs in Chicago Public School system in 1996 (that produced a highly non-linear link between a school’s reading success and the probability that the school was on probation in succeeding years) in order to determine the effect of teacher training on student achievement.[1] This strategy, however, does not determine the school probation policy’s cumulative effect; instead, it efficiently identifies the influence of the resources given to particular low-achieving schools placed under the probation policy.

The paper concentrates on the teacher training’s impact given the consideration that it takes into account the contribution of all of the resources given to schools under the probation policy.

II. Background

A. Prior Literature

Finding shows that in a higher analysis only 12 of 93 studies in the teacher development impact on student illustrate positive effects of employee development (Kennedy (1998)). This result is consistent with the research of Corcoran (1995) and Little (1993) in which employee development is a low intensity matter that needs continuity and accountability.

On the contrary, Bressoux (1996), and Dildy (1982), found that teacher training improves student performance. Wiley and Yoon (1995) and Cohen and Hill (2000) also found teacher enhancement programs to have at least small effects on student achievement.

Meanwhile, it is the work of Angrist and Lavy (2001) that observed strong impact of teacher training, in addition to their finding that there appears to have been more studies on it in developing countries than in developed countries.

Although this study provides strong evidence on the potential efficiency of teacher development programs, this analysis has several limitations. Besides financing teacher training, the intervention includes other factors that may have enhanced student performance, such as having a learning center to aid failing pupils after school and a project to assist immigrant students and their families.

B. Background on School Reform in Chicago

The Chicago Public School system (CPS) is the nation’s third-biggest school district, catering to above 430,000 mostly low-income pupils. In 1996, it initiated a highly publicized reform effort that laid emphasis on holding students, teachers and administrators responsible for academic performance.

Stated in the Chicago policy, schools having below 15 percent of students meeting the national norms on standardized reading exams will be placed under probation.

To enhance student performance in these schools, the CPS supplied probation schools further resources to buy staff development services from an external organization of their choice. On top to these direct resources, the CPS also gave such schools technical assistance and monitored the school’s progress.

The results provided evidence that suggests that probation raised the frequency of professional development activities by roughly 25 percent in the first year. Furthermore, there is some evidence that the quality of teacher development activities in probation schools improved from 1997 to 1999.

III. Data

This paper makes use of the administrative data from the CPS. Student records present detailed demographic and educational background data on individual pupil per academic year, as well as earlier achievement scores, previous school and housing mobility, birth date, race, gender, family composition, free lunch status, and special education and bilingual services attained. School records, on the other hand, give mean demographic data at the school level, including fraction of low-income, mean daily attendance, and school average test scores. The main performance measures used are math and reading scores from the Iowa Test of Basic Skills (ITBS), which CPS pupils take year in grades two to eight.

The base data of the study with 131,314 samples consists of the group of third through sixth grade pupils enrolled in a Chicago elementary school in the Fall of 1996. This limitation is due to the fact that this paper measures performance gains beyond three years and ITBS scores are not available for students beyond eighth grade.

The following are removed from the sample: 198 pupils who are enrolled in a special needs school; 3,981 pupils who are lacking student or school demographic data; and 26,907 pupils who did not take the ITBS exam in the Spring of 1996. This leaves the paper with 100,288 sample pupils from 461 different schools.

The finding presented that:

• The most lacking students in the CPS were served by probation schools.
• Over 95 percent of pupils enrolled to a probation school got free lunch in contrast to just 76 percent of pupils not enrolled in a probation school.
• Pupils in probation schools were almost twice as likely to be in a foster home in Fall 1996.
• There is considerably less odds that Hispanic pupils are to attend a probation school than Black students.
• Probation school pupils experienced school level mobility, non-attendance, and low-income rates substantially greater than those in non-probation schools.

 IV. Empirical Strategy

The education production function shows that a student’s teacher receiving in-service training is influenced by student demographic and past performance, teacher and school attributes, as well other unobserved school quality and constant unobserved student ability. The challenge however in estimating the underlying effect of training is that teachers and schools may opt, or be chosen, into training based on attributes that are unobservable to the researcher.

With regards to teacher training, according to Lavy (1995) and others, often, there is a negative correlation between school inputs and student performance since measures of socioeconomic disadvantage are used to decide which schools get the most inputs. On the contrary, with teacher training often a deliberate activity chosen by the teachers and administrators in a certain school, it is likely that the most determined teachers and schools look for training thus leading to an upward bias.

The Chicago school reform efforts present a unique chance to determine the causal effect of teacher training on student performance. The stringent test score cut-off for probation produced a highly non-linear link between school reading performance in 1996 and the mean number of years a pupil spent in a school on probation between 1997 and 1999.

The regression discontinuity design strategy used shows that if there is a perfect correlation between 1996 school reading performance and the number of years a pupil stayed in probation school, then a properly defined OLS model consisting of a dummy variable pointing out whether the pupil attended a school below the cut-off in 1996 would give unbiased estimates of the training impact.

However, the relationship of years in a probation school with 1996 school reading achievement is not perfect for various reasons:

1. Various schools with scores below the probation cut-off were excused from the policy;

2. 25 probation schools in 1996-97 increased achievement enough to be excluded from probation for succeeding two years.

3. There was considerable student mobility between probation and non-probation schools during this period.

Taking these, conceptualizing the regression discontinuity approach in an instrumental variables framework is perceived to be helpful. In order to attain consistent estimates, teacher development must be identified by some variable that is not directly linked with pupil achievement.

One may consider that the grant of waivers in the marginal range just below the cut-off may be endogenous. Thus it does not influence the reliability of the estimates since the strategy used depends on the change in years in a probation school which is based on the observed distance below the cut-off.

Furthermore the study’s approach depends on various assumptions.

1. Provided sharp regression discontinuity designs, it is enough to assume that unobserved attributes do not alter discontinuously at the cut-off.

2. Having an unclear design, further assumption would be that unobserved attributes are not connected to school performance similarly as treatment distribution.

Another concern is that educators or school administrators may try to sway student scores on the margin. Jacob and Levitt (2002) found cases wherein Chicago teachers might have inappropriately aided students on exams; however, such behavior seems restricted to a fairly small number of classrooms and is unlikely to influence the results. To guarantee there is no bias in the results brought about by teacher cheating, all estimates are re-ran, removing the small set of schools the authors identified to have a high degree of cheating in 1996. The results were close to identical.

With regards to student mobility rates which are normally greater in lower performing schools, it must be expected that there is higher mobility among probation schools as compared to non-probation schools. Even though high mobility is not much of a problem, if probation leads to high-achieving or determined students to exit the CPS, the estimates may be biased. However the finding suggests that probation did not induce student mobility, hence supporting the power of the achievement estimates.

V. Results

A. Main Findings

The findings suggest that if teacher training has a considerable effect on academic performance, a fast change in the main achievement level around the probation cut-off is expected. Also, if the teacher training connected to probation were advantageous, a drop in performance as school reading performance approached and exceeded the cut-off is expected. The teacher training in Chicago, however, was found to have no sufficient effect on pupil achievement.

The study indicates that probation schools provided for a notably more disadvantaged student population than other schools. Also, the estimates for the reading and math test scores are found to be negatively significant.

Meanwhile it was observed that probation bears no economically or statistically significant impact on reading and math achievement, nor do teacher training and/or technical assistance provided to probation schools seem to have any important effect.

Furthermore, the results insensitivity to the inclusion of control variables implies that after controlling for school reading performance, pupil enrolled in schools roughly above and below the cut-off have comparable observable attributes, hence providing more validity to the assumption that the unobservable student attributes in schools roughly above and below the cut-off are comparable too.

It is helpful to recall that the Chicago probation policy was aimed both to encourage low-performing schools and to enhance student performance by offering them the resources to do so. The motivation for teachers and administrators to progress was impelled by the risk of reconstitution, which would lead in the relocation and probable severance of all school employees.

According to the study of Jacob (2002) the incentives brought about by the probation policy generally did lead to a significant rise in performance among low-achieving schools.

In the interpretation of the results, several points are kept in mind.

1. The estimates incorporate the teacher training effect within schools that have significant motivations to enhance student achievement. So far as training under these condition is more or less efficient than training given no incentives, the findings may differ from other assessment of training programs.

2. The estimates capture the impact of any differential incentives experienced by schools on either side of the cut-offs. Since, on one hand, schools that only missed being put on probation in 1996 were susceptible of being placed under probation in succeeding years, they also had a motivation to improve student performance. On the other hand, probation schools in 1996 may have had even greater incentives to improve achievement. Because these factors are likely to function in similar direction, suggesting that if teacher training has positive effect on student performance, then probation schools students in 1996 should do better than students in schools that barely evaded probation that year.

These imply that as the study seems to constantly find zero impact it only emphasizes the conclusion that the training simply did not influence student performance.

B. Other Effects of Probation

Probation may have an effect on student mobility and test-taking patterns. Specifically, motivated families might prefer to remove their children from probation schools and probation schools might opt to stay away from testing the lowest performing pupils.

The study suggests that being enrolled in a probation school in 1997 has no significant impact on the likelihood of attending in the CPS in 1999. Also probation seems to augment the chances that a student transfers schools by 1999. Moreover, being placed on probation does not lead to discouragement of administrators to low performing pupils from being tested or from having the test scores included in school evaluation. In general, probation may influence a student choices concerning school attendance within the CPS. However, there is no evidence that those in probation school in 1997 would be urged to leave the district, to avoid testing, nor to have their scores excluded for evaluation purposes.

C. Heterogeneous Effects and Robustness Checks

Since probation is identified by the percent of pupils scoring above the 50th percentile, the policy generates an incentive for schools to concentrate on students close to this point, since they have the bigger chance of meeting such standard given sufficient support. However, probation does not seem to bear any greater impact on students in the second and third quartiles than on pupils found at the extremes of the ability distribution.

Moreover probation was found to have no significant impact on reading and math achievement.

Meanwhile, various schools scoring just above the probation cut-off, or were exempted from probation, were put on remediation, wherein such schools did not get the same close monitoring or financial aid as compared to that of probation schools, but they were exposed to a rather heightened supervision. However, such keen oversight was observed to have no effect on achievement.

Schools removed from probation were still required to keep a relationship with their external partner for an extra year. Meanwhile, some low-achieving schools not placed on probation opted to employ an external partner regardless it was not officially necessary. The impact of having external partner, on achievement levels in low-achieving elementary schools (both probation and non-probation schools) was statistically insignificant. It also seems that the students obtained no important gain from being enrolled in a probation school. Lastly, probation does not seem related to the performance of those students who changed schools.

VI. Conclusion

In an attempt to enhance student accomplishment in Chicago, the CPS placed nearly 20 percent of the lowest performing elementary schools in the city on probation. The financial and technical aids supplied to schools on probation were devoted to developing classroom instruction, mainly through teacher training and staff development. Teachers in probation schools showed fair increases in the rate with which they participated in professional trainings and more considerable increases in the quality of the professional development they obtained.

Prior analysis, conversely, states that the training given to teachers in probation schools had no observable influence on student performance. Such results are robust to various alternative provisions and do not vary across student ability, gender, race, or family income.

Meanwhile, teacher development can have a significant, positive effect on student performance under generally favorable situations; however those benefits rely on the programs’ framework and quality. The findings indicate that restrained increases in the intensity of the professional training attempts similar to that of Chicago program will likely fall short in developing the ability of students enrolled in failing schools.

Finally, as a cure intended to enhance the student achievement in failing schools, the teacher training offered to probation schools in Chicago seems to be completely inefficient. Teacher and school administrators must cautiously assess the nature of teacher professional development in this country.

Source:

Brian A. Jacob and Lars Lefgren, he Impact of Teacher Training on Student Achievement: Quasi-Experimental Evidence from School Reform Efforts in Chicago”, Journal of Human Resources, Vol. 39, No. 1, Winter 2004, pp. 50-79

---

[1] Schools under probation are provided special financing for employee development and technical assistance and improved monitoring. Qualifications for probation were determined based on standardized reading scores: schools have less than 15 percent of pupils scored at or above national standards in reading were placed under probation.