Bend-La Pine policy

Return-to-play procedures after a concussion: The athlete must meet all of the following criteria in order to progress to activity.

• Asymptomatic at rest and with exertion (including mental exertion in school).

• Within normal range of baseline on post-concussion neurospsychological testing.

• Have written clearance from a medical provider.

Snapshot of a district

A study conducted within Fairfax County Public Schools, a Virginia school district with 25 high schools, found that football, lacrosse and soccer had the highest rates of concussions between 1997 and 2008.

Rate is per 1,000 practices or games:

Sport -- Injury --Rate

Boys

Football -- 1,407 --0.6

Lacrosse -- 244 -- 0.3

Soccer -- 103 -- 0.17

Wrestling -- 123 -- 0.17

Basketball -- 77 -- 0.1

Baseball -- 32 -- 0.06

Girls

Soccer -- 195 -- 0.35

Lacrosse -- 114 -- 0.2

Basketball -- 120 -- 0.16

Softball -- 47 -- 0.11

Field hockey -- 58 -- 0.1

Cheerleading -- 131 -- 0.06

All boys -- 1,986 --0.34

All girls --665 --0.13

All athletes -- 2,651 -- 0.24

Source: American Journal of Sports Medicine

Andy Zeigert / The Bulletin

Many concussion experts have serious doubts about the accuracy and reliability of the neurocognitive computer test used to diagnose concussions and make return-to-play decisions for local high school athletes.

The testing program, known as ImPACT (Immediate Post-Concussion Assessment and Cognitive Testing), has become the centerpiece of the community's efforts to reduce concussion-related injuries. But a number of independent studies have found the test gives inconsistent results, has high error rates and can be easily misled by neurological factors unrelated to concussions.

Officials with The Center Foundation, which provides the tests for free to Central Oregon athletes in high-risk sports, stress that ImPACT is just one of many tools used to diagnose and manage concussions, and not the sole determinant of whether an athlete can return from a concussion. Yet, it remains the only one of those tests that student athletes in the region are required to take in order to play.

“There's not a lot of research that has shown conclusively that ImPACT assessment, in and of itself, adds significant benefit to the other things that one could assess in a face-to-face setting with a physician,” said Dr. Richard Wennberg, a neurologist at the Krembil Neuroscience Centre of Toronto Western Hospital in Canada.

Early adopter

The Center Foundation, the charitable arm of The Center: Orthopedic & Neurosurgical Care & Research in Bend, brought ImPACT to the area in 2001. One of the first sites in the nation to provide baseline testing to high school athletes, The Center quickly expanded its program to encompass multiple schools and school districts.

Each year hundreds of Central Oregon high school students participating in high-risk sports are required to take the computerized test that aims to record their normal brain function. The test consists of a series of questions or exercises, measuring reaction time, verbal and visual memory and processing speed.

The results are used to set a baseline in case the athlete incurs a concussion. Doctors and athletic trainers can then retest the player to see whether brain function has returned to normal before approving a return to the field of play.

In 2012, the program conducted 707 free baseline tests among high school students playing football, soccer, basketball and wrestling, and 233 follow-up tests. The program identified 117 concussions, about a third of them in football players.

ImPACT was still rather new when it was first brought to Bend, developed by concussion experts at the University of Pittsburgh Medical Center in the 1990s. Their timing was fortuitous. As concerns spread about the dangers of concussion in professional and amateur sports, doctors and trainers were eager for a tool that made concussion diagnosis and return-to-play decisions easier and faster.

Enter ImPACT, which was easy to implement and interpret, bringing hard statistical data and an objective, quantifiable approach to a field that had been characterized mainly by subjective decision-making informed by years of training and experience.

Use of the program swept through professional football and hockey teams, college sports and high schools, particularly in more affluent areas. By 2009, computerized neuropsychological testing was used in the evaluation of 41 percent of concussions nationwide, a 15 percentage point jump from 2008 alone.

Yet some clinicians using ImPACT found the test results often ran counter to what they were observing in their patients. Steven Broglio, director of the NeuroSport Research Laboratory at the University of Michigan, said most athletes with concussions experience a big dip in cognitive function immediately after the concussion, but then make steady progress toward recovery. Each individual may progress at a different rate, but generally patients continue to improve unless they return to play too early or engage in other types of discouraged activities.

While at the University of Illinois, Broglio and his colleagues were regularly seeing athletes who improved on ImPACT scores a week after their concussion, but when retested four or five days later, were down on some measures and up on others. Their scores seemed to rise and fall with no rhyme or reason.

“It just didn't make sense to us clinically and that was actually what prompted our investigation,” Broglio said.

In 2007, they conducted baseline tests with more than 100 students, using a variety of computer-based neurocognitive tests, including ImPACT. They tested the same students 45 and 50 days later. None of the students had incurred a concussion during the test period, yet the retest results showed little consistency with the baseline tests. On a scale from 0 to 1, where 0 reflects no correlation and 1 perfect correlation, the five scores generated by ImPACT at baseline and retest had correlation ranging from 0.15 to 0.39. An acceptable cut-off for a useful test, according to the researchers, is about 0.75.

Too often the ImPACT incorrectly gave scores consistent with having a concussion.

“Keep in mind these are all healthy college students,” Broglio said, “and we had somewhere in the neighborhood of a 40 percent false-positive rate.”

The more independent researchers looked at the reliability and validity of the ImPACT test results, the more concerns were raised. Some questioned whether there was a “practice effect,” that the more times someone took the test, the better they did. That could skew results, suggesting patients were improving when they were really only getting better at taking the test. Others questioned whether it was smart for concussed athletes to be taking a 20-minute computer test when they should be resting their brains.

Even the concept of baseline testing has been cast in doubt.

“Any test that we do — and that includes computer neurocognitive testing, balance testing or eye movements, you name it — these things are all going to be affected by a number of factors that don't have anything to do with injury,” said Dr. Jeffrey Kutcher, an associate professor of neurology at the University of Michigan Medical School and a co-author of the American Academy of Neurology's concussion guidelines. “I'm not going to rely on these things to diagnose concussion. I use the data as part of my total clinical evaluation.”

Kutcher said the various tests can be easily swayed by how much sleep the athletes had the night before, whether they're in pain or taking any medication, how easily distracted they are.

“Even within that subset of baseline tests, (ImPACT) is one of the worst, in my opinion, because of the length and the difficulty and the actual tests that are used,” Kutcher said. “The more simple a test is, the more reliable that data is. That's where I think ImPACT really suffers.”

ImPACT responds

Developers of ImPACT argue — and critics acknowledge — that the test is being held to a standard that many of the other tests used in concussion management and medicine in general haven't met.

“No test is perfectly reliable,” said Dr. Mark Lovell, who co-developed ImPACT and until recently served as CEO of ImPACT Applications, the company that markets the test. “Many of the tests — blood pressure, heart rate, EEG — that neurologists use are remarkably unreliable, and we accept that because they're the best tools we have.”

Lovell believes part of the criticism of ImPACT stems from the misuse of the test as a stand-alone product to diagnose concussions and make return-to-play decisions.

“I need to emphasize this, and I have always said this: I don't think that ImPACT or any other test should be used in a vacuum,” Lovell said. “It shouldn't be looked at to say, 'He's good to go or he's not good to go.' We've never said that.”

Lovell said that of the nearly 200 research studies published about ImPACT, albeit many of them by ImPACT's developers themselves, only a small percentage have raised concerns about reliability or validity of the results.

“It's as reliable as any other test,” he said. “And we don't have anything better.”

Lovell said the least reliable test is asking the athletes how they feel.

“Multiple studies have shown that's dangerous,” he said. “Kids lie to you all the time. So we need something other than, 'He looks OK; he says he's OK.' And right now, I think ImPACT is the best thing we have.”

There are steadfast supporters of the ImPACT test as well. Philip Schatz, a neuropsychologist with St. Joseph's University in Philadelphia, found a higher correlation on ImPACT scores in his test-retest scenario than did Broglio's group.

“In many ways, it depends on what type of statistical approach you use,” he said. “There's data that would suggest that ImPACT is highly reliable and there's data that says that ImPACT is not highly reliable.”

Schatz said ImPACT is particularly good at identifying concussions when they occur and ruling out concussions when they haven't. But critics argue there are other ways doctors can confirm a concussion without the need for widespread baseline testing, a rigorous follow-up test, or the purchase of an ImPACT subscription.

“Given the costs and time involved in baseline testing, there should be some clearly defined added benefit of baseline testing over the use of symptom checklists,” Dr. Christopher Randolph, clinical professor of neurology at Loyola University Medical Center, wrote in the journal Current Sports Medicine Reports in 2011. “Obviously, baseline testing adds nothing to return-to-play decision-making in an athlete who is still reporting subjective symptoms.”

ImPACT testing would be more useful, he argues, if it could pick up ongoing cognitive impairment after symptoms have disappeared. And the proof that ImPACT can do that is much more murky.

ImPACT developers have published studies showing athletes haven't returned to baseline on the ImPACT test well after symptoms disappear. They offer that as proof that the athletes have not fully recovered. Yet, in the absence of any sort of gold standard test to judge recovery from concussion, there is no way to determine whether what ImPACT is finding in those cases is really there or truly meaningful.

“Most of their studies have a bit of a circular argument to them,” Wennberg, the Canadian concussion expert, said.

Lovell counters that he'd rather risk a few false positives than return an athlete to play too soon.

“I think it's better to be cautious,” Lovell said. “We don't know everything about concussions yet. So if I'm going to hold somebody out a couple more days, I'd rather do that, and have the potential for being overly conservative, than send somebody back too soon and have them have a catastrophic brain injury.”

There is, to date, no outcomes research of ImPACT that would look at whether the test is better at protecting athletes from second concussions or other potential harms than concussion management without computer testing.

Lovell said that ImPACT has quickly become the standard of care for return-to-play decisions, and that no research setting would allow part of a study group to be denied the standard of care.

“It's a good question, it comes up all the time,” he said. “But I don't know how to do that study.”

Concussion researchers have been looking for a so-called biomarker for concussion, a blood test or other type of measure that could clearly show whether a brain has recovered from a concussion and whether it is safe for an athlete to return to play.

“The moment we have such a test, I think ImPACT vanishes as a commercial necessity,” Wennberg said. “But to leave the impression that it somehow is a significant black-and-white test for the diagnosis of concussion or the diagnosis of recovery from concussion is misleading. It only has that status right now because there is nothing else.”

Financial concerns

ImPACT may have incurred additional scrutiny because the test, developed in an academic setting, has now become a profitable commercial venture.

“It's all colored by the fact that it's become a big commercial enterprise,” Wennberg said. “They had the commercial stroke of genius to not just sell it on a CD, but to offer subscriptions.”

That means each year, school districts and leagues across the country pay for a package of baseline and follow-up tests, while clinics, such as The Center, pay to have doctors and trainers go through ImPACT workshops. ImPACT sells packages of baseline and follow-up tests; 300 baseline test and 90 retests go for $500. The company provides video-on-demand and live online workshops for free, while live training ranges from $50 to $250 for some of the all-day seminars.

The Center Foundation spends $10,000 to $12,000 each year on its concussion testing program, including purchasing the tests and accounting for staff time. Central Oregon athletes who are required to take the baseline tests pay nothing themselves, while those in other sports who elect to complete a baseline test pay $10. Additionally, some local teams have raised the money to have their athletes baseline tested.

Before ImPACT, clinicians relied primarily on paper-and-pencil tests that were developed by consensus groups and made available for free. But those tests were never vetted to the extent that critics demand of ImPACT.

“I'm sure there's a lot of jealousy, and I'm sure that a lot of criticism does come from people who have other things they're selling, either disclosed or undisclosed,” Lovell said. “But it's not as profitable as you might think.”

At the same time, the testing program has become a viable way for clinics to attract new patients. By providing the baseline tests, clinics establishes themselves as experts in concussion management and the source for following ImPACT testing.

“That's a very good way to capture patients in your clinic,” Kutcher said. “It becomes a tool for health systems and clinics to market themselves and get patients in.”

Of the 117 concussions incurred in Central Oregon schools using ImPACT during the last school year, 65 were referred to The Center's neurologists.

Wennberg said he does find ImPACT useful to put numbers behind his subjective findings, making them easier for players, parents and coaches to understand. He also finds athletes tend to be a little more forthcoming about their symptoms as part of a computer test than they are with a physician.

But Wennberg couldn't recall a case where the ImPACT test results led him in a different direction than his clinical judgment.

“You can usually tell how they are doing and make a recommendation for whether or not they should stay out for a while longer irrespective of whether you do an ImPACT test,” he said.

He worries that many communities and schools are using ImPACT as a substitute for a one-on-one assessment with a medical concussion expert.

In Central Oregon, use of the ImPACT test is evolving. Initially high school athletes who incurred concussions had to achieve scores within a normal range of their baseline before they could return to play. In 2009, however, Oregon passed a new law requiring that a medical professional sign off on the return-to-play decision.

That, in part, changed the standard and limited the wholesale reliance on the ImPACT test.

“The concussion program and the way we provide it has been a process,” said Carol Stiles, executive director of The Center Foundation. “Initially all we had was this tool called ImPACT. As more people have done research, we now have more tools.”

Dr. Vivian Ugalde, a physical medicine specialist at The Center and medical director for the concussion program, stressed that the ImPACT testing is only one of several tools clinicians and trainers use to make the return-to-play decision. The test results are combined with balance tests, a symptom checklist and reports from the athlete and parents to reach a decision.

“There's no perfect way to tell if they're resolved,” she said, “so we're trying to use all of these tools together.”

Ugalde said ImPACT testing has convinced her to keep patients out longer, even after other symptoms have resolved. But sometimes, the athlete will return to baseline on all but one score. A week later, that score might return to baseline but a different score becomes too high.

“Sometimes when kids are sort of hovering like that, I wonder if that's a problem with the test itself,” she said. “Right now, I think it's a reasonable tool to use. I think we're still in the process of evaluation to see if we want to change our protocols or use it differently.”

Sondra Marshall, a neuropsychologist with St. Charles Behavioral Health, often gets involved in concussion management cases, particularly if symptoms don't resolve quickly. Marshall said she often has to educate athletes and their parents about the ImPACT test, that it's not the thing they have to pass but just another source of information to help guide the decision.

She, too, has seen an evolution in how much stock local clinicians are putting into the ImPACT test scores.

“I'm less inclined to rely on that and more inclined to use some really good clinical skills,” she said.

Yet, the Bend-La Pine Schools' written protocol for concussion management stipulates that student athletes must be “within normal range of baseline on post-concussion neuropsychological testing,” in addition to getting medical clearance, in order to return to play. If athletes can't “pass” the ImPACT test, they can't go back.

Crowding out

The ImPACT program also tends to overshadow another important effort from The Center Foundation, providing athletic trainers to work with the high school teams. The trainers attend practices and games and can make quick judgments on when players need to be pulled from play immediately and sent for follow-up evaluation for a concussion. The trainers have been trained on the use of ImPACT, but also know the athletes better than doctors or psychologists who are called in after a concussion has occurred. They can provide significant input on whether an athlete has returned to his or her normal personality after a concussion, helping clinicians make the return-to-play decision.

Tyson Langeliers, a certified athletic trainer with Desert Orthopedics who works primarily with athletes from the Timber FC soccer team and the Mt. Bachelor Sports Education Foundation, said ImPACT plays a role in that decision but can sometimes be misleading.

“There are cases when ImPACT scores don't seem accurate,” he said. “So that's when we rely heavily on our neuropsychologists and our team. What do we think about these test scores? Is there something going on? Were they excessively tired that day? Why are these scores not right?”

But while athletes in Bend may have access to multiple neurologists, neuropsychologists and sports medicine specialists, schools in more rural districts may rely more on primary care physicians and pediatricians to make the return-to-play decisions. And in those cases, the ImPACT test results may play a bigger role.

Dr. Lester Mayers, a team physician for the Pace University varsity sports teams in New York, said often trainers are reluctant for legal reasons to send players back to play until ImPACT scores have returned to baseline, even when they are convinced the player has recovered from the concussion.

That could have the effect of artificially prolonging the recovery time for student athletes, something some concussion experts believe isn't necessarily a bad idea.

Legal concerns could also prevent schools from dumping ImPACT testing altogether. If a player is cleared by ImPACT and incurs a traumatic event, schools districts, coaches, doctors and trainers can always point to the ImPACT data.

“I do know that nobody has died after using ImPACT,” Lovell said. “I can't point to one person and say it saved them, but I know that a lot of other people have died when they weren't doing anything for their concussion.”

Wennberg said he's seen cases where ImPACT scores return to baseline, and yet the athlete quickly incurs a second concussion from a fairly mild impact anyway. And he's seen cases where test scores have not returned to normal but the athlete might be able to return to play safely.

“It goes against the better-safe-than-sorry thing we all do, but it's an open scientific question,” Wennberg said.

He does, however, worry that programs like ImPACT provide cover for school and league to suggest they're doing something to combat concussions even though baseline testing does nothing to prevent concussions in the first place.

According to data from the National Center for Catastrophic Sport Injury Research, cited in the Journal of the International Neuropsychological Society, there were 50 cases of permanent disability and 38 deaths due to cerebral injuries among American football players from 1997 through 2006. The vast majority of those cases were due to subdural hematomas, bleeding between the surface of the brain and its outer covering. The authors noted that this type of injury is almost invariably the result of a single impact, and as such, unlikely to be prevented through baseline testing.

The threat of second-impact syndrome, where a second impact before the brain has recovered from a first concussion causes traumatic injury, has been used to justify the need for baseline testing such as ImPACT. Yet the authors found that even at a minimal cost of $20 per test, the syndrome is so rare, it would cost some $36 million to prevent a single case of second-impact syndrome.

Researchers are concerned about the long-term effect of multiple concussions, even when athletes are able to fully recover from each one before returning to play. But it's unclear how baseline testing could reduce such risks.

Wennberg, who consults for the National Hockey League, has been a vocal proponent of increasing the size of the ice rink to Olympic hockey standards. The additional space on the ice would lower the number of collisions, which would likely reduce the number of concussions. Similarly, high school football coaches are cutting back on live contact drills during the week, to reduce the risk of head injuries.

Yet, even before the school year started, there were multiple concussions among football players and cheerleaders in Central Oregon. Stiles argues that the region's concussion program, providing baseline testing and trainer oversight, does prevent concussions, namely the second ones.

“But I really don't want to give the impression that we're going to prevent all of the concussions.” Stiles said. “It's not going to work.”

2811076