Bend-La Pine Schools performed above the state average in an experimental evaluation metric, but administrators question the technique, especially its role in a federal program that awards local teachers performance bonuses.
Value-added measurement (VAM) compares a student’s growth on standardized test scores over time to the typical growth for a student with similar characteristics and past achievement. Bend-La Pine’s use of VAM is part of Oregon’s $24.4 million federal Teacher Incentive Fund (TIF) grant, a five-year program studying the efficacy of paying teachers bonuses based on their value-added measurement data and principal evaluations in 29 district schools.
In Bend-La Pine, the VAM data is only reported on a schoolwide level and appears as a percentile. A school at the 90th percentile had better student growth than 90 percent of Oregon schools. Conversely, a school at the 10th percentile only outperformed 10 percent of state schools.
“This is our third year seeing data, and during our first year with VAM, we had no basis on which to make much sense of it,” said Bend-La Pine Schools Superintendent Ron Wilkinson.
“We still have a small sample, but we’re starting to understand how to use the data. And overall, we are doing really, really well. Most of our included schools are in the top half, and that’s not typical. Most districts would have more schools toward and below the 50th percentile.”
Of the 29 schools rated in the district, only three fell below the 50th percentile in reading and two in math. Additionally, one of the low-performing schools in each category is a high school, which administrators said produces the least useful data.
The issue in evaluating high schools is due in large part to how the Oregon Assessment of Knowledge and Skills (OAKS), the state’s standardized test, is administered. State evaluations of schools are largely based on how many students in each school and district meet target OAKS scores. While OAKS is geared toward measuring proficiency, value-added measurement looks at OAKS score growth — a student who went from a dismal to a mediocre score may fail OAKS both years, but nonetheless appear as a positive example of growth under value-added measurement.
“Data on the high schools is very problematic, because there’s a big gap between what VAM tries to do and what high school OAKS tests do,” said Dave VanLoo, Bend-La Pine’s director of school improvement. “In high school, student growth is compared to eighth-grade OAKS scores, as high schoolers don’t take the test each year but really only once. And in high school, the emphasis is on getting students to pass, not on getting the highest score possible. As a result, we don’t get a good picture of year-to-year growth in high school.”
VanLoo said that value-added measurement data for middle schools is the most useful, as students take OAKS tests each year, and growth is most clearly discernible.
Bear Creek Elementary was the district’s standout school, reaching the 90th percentile in math and the 95th in reading. The data under consideration corresponds to the tenure of former Principal Matt Montoya, who was fired by the school board in September for job performance reasons, including the failure to conduct required teacher evaluations.
“The performance of Bear Creek speaks to good teaching practices at that school,” Wilkinson said. “We know they’ve been doing good for a long time, especially with disadvantaged subgroups that statewide perform poorly. For about a decade it has tended to meet and exceed goals for those students, which I suspect would have been reflected in VAM data.”
Despite Bend-La Pine’s strong showing with value-added measurement data, administrators acknowledged questions concerning the usefulness of the metric.
“One reason we didn’t want teacher-level data is that research has shown that teacher data can swing quite a bit from year to year,” VanLoo said. “It’s hard to say a teacher is effective or not when that is the case.”
Wilkinson said another problem with VAM is that the data does not pinpoint what can be improved if a school were to perform poorly.
“Most everyone in education is trying to find ways to improve what we do,” Wilkinson said. “But with VAM, a bad score doesn’t point to any particular area to work on. So what I can say is that it’s another helpful lens to have when we evaluate how our schools are doing.”
As part of the federal Teacher Incentive Fund program, Bend-La Pine Schools teachers can receive a bonus depending on how their school performs. Teachers who work at schools in the 90th or higher percentile in reading and math receive an extra $1,665 for each subject. Teachers in the 80-89 and 70-79 percentile ranges receive smaller bonuses.
Teachers can earn a third bonus if they are highly evaluated by their principal. If a teacher were to receive the maximum bonus in all three areas, they could earn just over $5,000. Teachers working in schools that fall below the 70th percentile and receive a poor evaluation earn no bonus.
“The grant is trying to measure whether incentives make a difference, and looking at the data we have, I don’t see a pattern that would suggest that,” Wilkinson said.
Bev Pratt, Teacher Incentive Fund manager for the Portland-based Chalkboard Project, which oversees the Oregon TIF program, said she does not believe bonuses will create incentives for better job performance.
“I can say that I don’t think teachers are necessarily working harder because they can earn VAM-based incentive pay. But they do appreciate getting incentive pay based on the hard work that they’re doing,” Pratt said.
Bend-La Pine will exit the TIF grant at the end of the 2014-15 school year. Wilkinson is unsure if the district will seek to replace VAM or find an alternative.
“It’s something we’ll want to monitor over time, but right now it’s just another tool among many,” he said.
— Reporter: 541-633-2160, firstname.lastname@example.org