School Vouchers: Results from Randomized Experiments

Voucher programs have spread in cities like Milwaukee, Cleveland, Indianapolis, and San Antonio in the mid-1990s. Back then, their evaluations were restricted by data quality or research methodology because evaluation planning often ensues after the experiment was under way. Subsequent evaluations were later designed so that information of higher quality will be gathered.

Program evaluations were designed as randomized field trials because vouchers were awarded a lot. Data regarding students’ test scores and family background characteristics were collected before conducting the lotteries and then after a few years to find any statistically significant differences between students offered a voucher and those who are not offered a voucher. Differences may be attributed to the intervention because – through randomized selection – average student initial abilities and family background are similar between two groups.

A.   The Three Voucher Programs

The design of the three voucher programs was similar since: all were privately funded; all were targeted to poor families; and all provided partial vouchers. On the other hand, the programs differed in size, timing, and administrative details.

1. School Choice Scholarships Foundation Program in New York City (SCSF)

SCSF announced that it would provide 1,300 scholarships worth up to $1,400 annually for at least three years in 1997. After its announcement, SCSF received more than 20,000 applications. To be eligible, students have to be: under grades level one to five; living in New York; enrolled in a public school; and a member of a low-income family.

SCSF assists its scholars’ families in identifying the private schools they can attend. About 82% of the students who participated in the evaluation were using a scholarship; 79% used the voucher for two years; while 70% used them for three years.

2. The Parents Advancing Choice in Education Program in Dayton, Ohio (PACE)

PACE is a privately funded non-profit corporation which started to offer scholarship to almost 800 students in the Dayton metropolitan area in 1998. During the first year, PACE received over 3,000 student applications. Half the tuition of selected students was covered (up to $1,200). Support was granted up to at least four years up to the completion of high school. In 1999, funds were further increased. Among the students that were offered a scholarship, 78% attended private schools in the first year while 60% were in private schools after two years.

3. The Washington Scholarship Fund Program in Washington D.C (WSF)

WSF was established in 1993. This privately funded school voucher program served around 460 children at 72 private schools in 1997. It then received huge funds from two philanthropists which expanded the program in October 1997. This expansion was publicized both in general news announcements and paid advertising.

Student applicants have to reside in Washington D.C. and must enter grades K-8 in the fall of 1998. Families with incomes at or below the poverty line were awarded vouchers that amount to 60% of tuition or $1,700, whichever is less. Those with incomes above the poverty line are awarded less; no family whose income was 2.7 times the poverty line were selected. High school students, on the other hand were given scholarships up to a maximum amount of $2,200. WSF promised to support its scholars up to three years; and if funds permit, until they finish high school. Out of the 7,500 applicants, 1,000 new scholarships were given that year, 811 of which are students who have not attended a private school. They were allowed to attend any private school in the Washington area. Out of the evaluated students, 68% attended a private school in the first year, 47% in the second year, and 29% in the third year.

B.   Evaluation Procedures

The evaluation team collected baseline data prior to the lottery, administered the lottery, and then collected follow-up data after one or two years. Such methodology follows the randomized field trial procedure.

1. Baseline Data Collection

Applicants in the first grade and higher took the Iowa Test of Basic Skills (ITBS) in reading and mathematics. In addition, students in grades four to eight completed a short questionnaire about their school experiences. Meanwhile, the adults accompanying the students filled out questionnaires regarding their children’s school satisfaction, their involvement in their children’s education, and their demographic characteristics.

In New York, 5,000 students participated in baseline testing; 960 of them were selected and participated in the control group. In Dayton, 1,440 were tested at baseline, 803 of which are not attending a private school and 690 completed questionnaires. In Washington, 2,023 were tested at baseline, 1,582 of which did not attend a private school and 1,446 completed questionnaires.

2. The Lottery

The lotteries were conducted in May 1997 in New York and April 1998 in Dayton and Washington D.C. All children in selected families were offered a scholarship. In Dayton and D.C., separate lotteries were conducted in public and private schools, ensuring random assignment to test and control groups of families joining evaluation. Since vouchers were given by lottery, students who were offered scholarship are expected not to differ from members of the control group. Baseline data confirm this assumption.

3. Collection of Follow-Up Information

In New York, testing and questionnaire administration procedures were similar with baseline processes. Students – both the voucher students and those from the control group – were tested in locations other than their school and parents were asked to fill out questionnaires regarding the experiences of their oldest child. The above exercises were included as requirements for the renewal of scholarship. Meanwhile, non-voucher students were compensated for their participation and were promised that they will be automatically included in the next lottery.

C.   Data Analysis and Reporting Procedures

Because lottery was used to allot the scholarships, any statistically significant differences between the voucher and control group may be attributed to their school experience, not their baseline characteristics.

The data that were gathered address two questions. First, is the impact on educational outcomes was of an offer of a voucher to low-income families (intention-to-treat or ITT effect). A statistical model that looks into the student’s treatment or control group status – as well as baseline math and reading baseline scores – was estimated. The next question is the impact on educational experiences, parental satisfaction, and test score performances one, two, and three years after switching from a public to a private school (treatment-on-the-treated or TOT effect). A two-stage least squares model was estimated to compute the impact of on those who used the scholarship to transfer to a private school.

D.   Test Score Findings

1. Prior Research

Low-income and African American students who are attending private schools were found to outperform their public school peers. Students from all racial groups are likely to enrol to college if they attended Catholic school; the effects are more noticeable in urban minorities. Moreover, the effects are even more pronounced for students with achievement test scores in the lower half of the distribution.

Previous studies cannot ensure that they have controlled an important intangible factor – the willingness of parents to pay for their tuition, and consequently, the importance they put on education. It becomes indefinite whether findings reflect differences between private or public schools or just the differences in the characteristics of the families. This self-selection problem can be solved through random assignment of students to test and control groups. This study gives an opportunity to effectively examine the effects of school vouchers on students from low-income families in central cities as it used random assignment as part of its research methodology.

2. Impacts of Private-School Attendance on Test Scores

Importance is put on the estimated effects of attending a private school on combined test scores for all three cities because average estimates for more than one city are better indicators of programmatic effects than do results from one city. The results are also more stable across time and from place to place.

It was observed that there is no significant differences between the test score performance (including reading and math tests) of non-African American students switching from a public to a private school and the performance of their peers in the control group, after one to three years. Meanwhile, when the results of the three cities are combined, it was seen that African American students who switched from public to private schools scored higher on their tests after a couple of years. Taking each city individually, New York reported the largest difference. Test score increases also appear to grow over time in this city as well as in Dayton; unclear results were observed in Washington.

3. Interpreting the Magnitude of the Test Score Effects

The effects on student test scores of enrolling in a private school are huge. African-American students yielded higher test scores every year. Meanwhile, in Texas and North Carolina, a remarkable one-year gain was observed in the math scores of students. The shift from public to private schools also facilitated a class size reduction of between 7 to 8% (in Tennessee alone).

4. Cost-Benefit Analysis

According to Richard Murname and his colleagues (2000), a 0.30 standard deviation increase in average math achievement of an African American student will yield a 5% gain in earnings seven to ten years after a student finishes high school. For example, if an African American student in the control group will earn $30,000 a year in his late twenties, a student who has switched from public to private school would be expected to earn an additional $1,500 per year. This implies that investments in vouchers might give a reasonable return to African American families.

In spite of the positive rate of return for investments in vouchers, families are restricted by credit constraints. Private lenders may hesitate to make long-term loans to low-income families. And even if they lend at high rates, the rate of return will be less attractive for the families. The voucher lowers the initial cost of low-income families, which makes switching to private schooling more attractive.

5. Additional Methodological Considerations

Two methodological considerations are addressed in this segment. First, the status of background variables is assessed. The impact of test scores of African Americans who switched from a public to private school remained almost exactly same even if variables such as mother’s education, mother’s employment status, family size, and the presence of family welfare benefits are included in the analysis. When effects within individual cities are analyzed, minor differences are still observed.

The second methodological consideration deals with the possibility that African Americans displayed significant effects because they received a more similar treatment. This question arises from the possibility that students who used vouchers were disproportionately concentrated in a small number of good private schools, or conversely, their peers concentrated in a small number of bad schools, the error term would be smaller for African Americans and this would make the observed impacts more pronounced for African American scores. However, this does not have much basis because the size of the standard errors is not all that differentiates the effects for African Americans and other ethnic groups. Also, there is little evidence that treatment effects were more uniform for some groups than others.

Parent Satisfaction

Studies found that families who use vouchers – and switch from public to private schools – are more satisfied than those who do not. Features such as school safety, teaching, parental involvement, class size, school facility, and student respect for teachers are considered in assessing the said claim. To verify whether satisfaction levels result from a Hawthorne effect, the propensity of individuals to welcome change for its own sake, an index of satisfaction was constructed. It was found that Hawthorne effects are minimal although overall satisfaction levels decrease across time.

E.   Other Voucher Impacts

1. School Facilities

Schools expenditures are higher for public school students than private school students because more services are available to the former. But even if the kinds of services rendered are adjusted, public schools in New York City, Dayton, and D.C. still spend around twice as much as public schools.

This can be explained by using parental reports. Private schools normally do not have facilities like a library, nurse’s office, cafeteria, child counsellors, and programs for non-English speaking students. Public schools are also usually bigger than private schools. However, private schools usually have individual tutors and after-school programs.

Despite the limitations, parents of children who transferred to private schools reported that their children attended classes with an average of twenty students – three students less than an average public school class. Such reduction was not large enough to yield significant gains.

2. School Climate

Chubb and Moe (1990) observed that the educational environment of private schools was more conducive than that of public schools. Public schools are governed by requirements such as federal and state regulations. Because of them, the morale of educators whose original objective is to help children learn is undermined. In contrast, private schools have greater autonomy, which allows them to focus more on providing quality education. Teachers enjoy higher morale and students experience a positive learning environment. The collected data confirm this fact as public schools are more likely to report higher incidences of fighting, tardiness, student fights, and racial conflict.

3. Homework and Parental Communication

Hoffer, Greeley, and Coleman (1985) accredited high levels of private school student performance to the amount of homework that are required from students and the frequency of communication between the school and parents. Around 72% of parents who have children in private schools reported that their child had at least an hour of homework a day, whereas only 56% of the control group reported a similar amount of work. Parents of students in private schools also stated that they received information regarding their children’s grade halfway through the semester as well as during parent-teacher conferences. They were also notified when they are sent to the principal’s office or whenever they have displayed untoward behaviour.

F.   Conclusions

Because the assignment of scholarships is done at random – through lottery – any statistically significant differences between the voucher and control group may be attributed to their school experience, not their baseline characteristics. The analysis made use of the consolidated information from the three sources of data, New York, Dayton, and Washington, because it provides a stronger basis for conclusions and generalizations.

In all three cases, it was observed that through the use of vouchers, African American students who shifted from public to private schools have registered higher test scores on the following years. Their parents reported smaller school and class sizes, a better disciplinary environment, an increased amount of schoolwork, and closer parent-teacher communication.

In this study, only a small portion of children were offered vouchers. A much larger program might yield different program outcomes. Nevertheless, the positive effects of school choice as a consequence of voucher programs may prove encouraging for those who intend to expand the program. The negative findings, on the other hand, represent the issues that must still be addressed.

Source:

Paul E. Peterson, William G. Howell, Patrick J. Wolf, and David E. Campbell, “School Vouchers: Results from Randomized Experiments”, In Caroline M. Hoxby, The Economics of School Choice, Chicago: University of Chicago Press, 2003, pp. 107-144.