It’s been nearly a year since local police started experiencing connectivity problems with emergency dispatch.
On Tuesday, the likely source of the problem was revealed: an aging software Deschutes County 911 uses to prevent the devices in patrol cars from having to reboot after going in and out of coverage areas.
Since about May of last year, mobile data terminals — the devices police use in patrol cars that receive information from dispatch and other sources — were prone to hiccups, but it wasn’t clear why.
Police reported being involuntarily logged out of their devices, which they contended meant they had inconsistent access to crucial information about calls and suspects.
Theories were floated. Perhaps it was a problem within the system Bend Police and other local departments use to manage records electronically. Or maybe it was the dispatch system maintained by 911. Or, public safety officials mused, it could be user error.
But Deschutes County 911 Executive Director Steve Reinke and a consultant hired to look at the issue now say the likely “culprit” is a database in the software. That database is intended to protect the information sent from dispatch to patrol cars from unauthorized eyes.
Reinke said in an interview Tuesday the issue may have arisen around the same time that Verizon — which provides a private cloud for the county’s emergency communications — was down for a day and a half last May. Once the system was up and running again, Reinke said, a database in the software may have gotten a bit jumbled.
Since that time, Bend Police had been vocal about connectivity problems their officers experienced. Those problems grew more common, but not universal, over time, Reinke said.
“This was so veiled by just occasional issues — not everybody saw it — that it masked the true nature of the problem for quite some time,” Reinke said.
The consultant, Tyler Hardison of Redhawk Network Security, briefly presented his findings Tuesday at the monthly meeting of the Deschutes County 911 User Board.
The board is a group of public safety and emergency response officials whose agencies use 911’s system.
Now that officials have tentatively identified the source of the problem, they are testing a software upgrade on a few patrol cars in the field.
They also purchased a new server — simply put, the hardware that hosts the NetMotion software — which is running parallel to the old server and serves those cars being tested.
Reinke said the district did not want to transfer everybody over to the new software and server until the county’s information technology staff concluded that the changes they’ve made resolve the connectivity issues.
“Once … staff is confident we have not seen problems with test units … we’ll migrate everything over on a pretty rapid basis,” Reinke said. “Essentially, most of our users are still operating on the old server with the older version of the software.”
But Reinke said that so far, the patrol cars testing the upgraded software and new server have not had the same problems. He expects the wait-and-see period will conclude in one to two weeks.
Redhawk was retained about two months ago to perform a security assessment. The Bend firm was later asked to evaluate the connectivity issue. The total cost for both tasks was about $13,000 to $15,000, Reinke said.
The NetMotion server was last upgraded in 2012, and Reinke said the district had plans to upgrade it — and the software, which most recently underwent an update a couple of years ago — later this year.
“Plans were in place to do that,” Reinke said. “Once we discovered this looked like the likely culprit, we accelerated the replacement.”
Software and hardware like servers and PCs have a typical life expectancy of three and a half to four years when used in emergency communications, he said.
Bend Police Chief Jim Porter said during the meeting Tuesday that when he worked a graveyard shift several weeks ago he was disconnected three times. In a warm-and-fuzzy moment rare in law enforcement circles, he indicated his gratitude for the consultant’s findings.
“You know, I’d give you a hug if I was any closer,” he told Redhawk’s Hardison.
The connectivity question arose in the context of an ongoing discussion between 911 and its user agencies about whether to use the same company to manage records and dispatch. The two systems are operated by separate companies.
Reinke maintained Tuesday he supported looking into integrating the two systems.