The Mifflin County, PA Office of Public Safety oversees emergency management, public safety and 9-1-1 communications. The office is responsible for connecting 17 volunteer fire departments, five EMS systems, two law enforcement agencies, and a host of traditional county agencies including the Sheriff’s Department, Probation and Children and Youth Services. Collectively, the office processes roughly 25,000 calls to 9-1-1 and 78,000 non-emergency calls annually.
With so much trust placed upon the office of public safety, it is crucial that their staff has the proper infrastructure in place to allow them to perform their jobs to the best of their ability—and a majority of that infrastructure consists of technology to process calls. If the technology fails, every second of downtime could mean lives lost. So, when the time came for the Mifflin County, PA Office of Public Safety to update their systems, decisions were not taken lightly. It was critical to find an emergency solution that would perform 24/7, 365 days of the year. Staying Ahead of the IT Curve
Mifflin County’s original 9-1-1 call center, created in 1991, was the first computer-based center in all of Pennsylvania. While the system had been continuously updated through the years, the technology was beginning to age and become inefficient. In addition, the actual physical setup of the center was illogical. Literally on the floor at each dispatch position were electric receptacles that supplied power for the Computer Aided Dispatch (CAD) system, a radio system, an E9-1-1 telephone system and other accessories. As the technology clutter grew, so did inadvertent problems. There was a point at which staff would stretch their legs and unplug the machines, bringing down the entire network. It was time for a change.
Mifflin County faced several major objectives in assembling the new 9-1-1 center. The primary goal was to gain efficiencies by consolidating data from several sources in the county network and ensuring that information was updated regularly in the CAD system. Second, redundancy was needed to ensure there was no single point of failure. Third, information technology (IT) staff needed to be able to update their systems and information much more rapidly than in the past to minimize planned downtime. Fourth, the system would have to be Windows-based to ensure the most familiar working environments are used. Finally, they wanted to reduce overall costs associated with the CAD system. On top of meeting all of the above goals, the technology selected also needed to be inherently robust, cost-effective and perform with reduced ongoing operational costs.
Evaluating System Configuration: What Exists Now and What’s Next?
The Mifflin County Office of Public Safety is set up so as each emergency call comes in, the CAD system alerts the appropriate agency, combining call information with data from several county offices including the Assessment Office and the Tax Claim Bureau, with Graphical Information System (GIS) information making it available to dispatch positions in the 9-1-1 center.
To design and implement this new center, Mifflin County turned to Logistic Systems Inc. (LogiSYS). The Office of Public Safety and LogiSYS have had a long and successful association, having worked together on their previous 9-1-1 system. Through ongoing discussions, it was clear that LogiSYS shared the county’s vision—to create a system that would be productive, capable of handling high call volumes during a wide spreadsheet emergency and would continue to operate normally, even if one part of the system failed.
A More Productive System to Ease Transition
To begin the 9-1-1 center re-haul project, Mifflin County first looked at their main CAD application and determined the features and functionality in their current dispatch software met their dispatch requirements. To minimize training costs and to maximize familiarity with the system, Mifflin County made the decision to stay with their current LogiSYS CAD, but update it to the Windows-based NorthStar platform. In moving to next generation technology, many decisions were based on what staff was already comfortable using, recognizing if the staff wasn’t comfortable or familiar with the equipment they were using, it would make the entire process go much slower. In the 9-1-1 crisis world, speed and accuracy determine how well the job is done—it is paramount to operation.
As part of the expansion, Mifflin County wanted to double capacity and create operator positions for at least half of their staff at any given time. As they worked to set the number of positions, they determined the best processing and storage capacity to deal with the expected increase in call volume brought about by raising the number of positions. Mifflin’s next concern was to make sure the systems placed into service would have the redundancy and capacity to deal with the variable event loads. It is important to note the county is also home to a state college, which brings swings in call volume during the school year.
The community of approximately 46,000 residents averages around 200 calls a day, but based on load activity, special events or extreme weather, the 9-1-1 communications center can see call volumes increase 25 fold. During Hurricane Ivan for example, dispatchers went from 200 calls a day to 200 calls an hour, necessitating a scalable and flexible system that could handle variable call response and application data requirements.
To thoroughly vet the solutions, the Mifflin County technology team, along with LogiSYS assistance, brought in alternative solutions and information before making a final purchasing decision. They wanted their selection to provide high reliability and continue to run normally without interruption, even if one of its hardware parts failed. Mifflin County also researched reliability issues to determine the level of redundancy required in each system in the event of problems.
The Solution: NorthStar CAD System on an NEC Infrastructure
At the end of the process, Mifflin County chose to upgrade its 9-1-1 dispatch center to run a LogiSYS NorthStar CAD system on an NEC infrastructure driven by the NEC Express5800/300 series fault tolerant (FT) server with Xeon processors. The overall system required only one server. However, because Mifflin County was running multiple critical applications on a single server (CAD processing, caller information and an extensive GIS system), a robust server is mandatory.
NEC’s Express5800 FT server provides high-performance for the county’s demanding, mission-critical applications. NEC’s fault-tolerant servers deliver up-time through dual modular hardware redundancy. Main hardware components, including CPU and memory, are replicated to run in lockstep, where the two redundant modules process the same instructions at the same time. If any hardware component fails in one module, the faulty component is automatically isolated and processing continues uninterrupted in the other module without downtime or loss of data. The failed module can be replaced while the other module continues to operate.
By using fully redundant and hot-swappable hardware components operating in lockstep across two complete server modules, the system achieves an average of less than five minutes of downtime per year. The only way a fault tolerant server can go down is when both of its redundant modules fail during the repair period after the first failure. This is extremely rare and the odds of this happening are many times less likely as compared to alternative solutions, or no solution at all. In the event that a hardware component fails, the active redundant server module continues to run the application software and allows for the repair of the affected module without interrupting server operations. Because failover is virtually instantaneous and there is no single point of failure, downtime is nearly eliminated.
Prior to shipping, NEC worked with LogiSYS to preconfigure the entire system at their site in Montana, and ran basic test procedures to make sure the infrastructure was solid. When the system was installed onsite, NEC technicians also set up remote monitoring of the server.
Since going live in October 2008, the new 9-1-1 dispatch center hardware has had zero failures. Calls come into dispatchers and CAD shares that information, combining it with ESRI mapping data that provides up-to-the-second resource availability and unit status with all dispatch positions. That information is then sent out via voice, e-mail, text pagers and printers to various fire stations and first responders.
One of the impetuses for expanding the system—and going with the system currently in use—is being prepared for increased call volume and capacity, which was previously not an option. By doubling the number of seats available, the systems are more than capable of dealing with crisis loading that had previously been experienced.
The system is set up to log each and every transaction to a DB2 database, tracking every change in each transaction to provide accountability. Each interaction involved with the transaction, including “When was an address changed?” “Who changed it?” “When did the unit respond?” “When was its status changed?” “Who changed those indicators?” generates a tremendous number of smaller transactions that need to be processed in the system. These ongoing transactions, combined with GIS mapping requirements and the overall intensive nature of CAD software, require a server that delivers processing power to keep the application running at peak performance.
One of the major decision points for Mifflin County was the ability to update GIS mapping information on a daily basis and handle it live as needed. With the NEC server there is no downtime while running updates. It used to take up to an hour to upload new data or map files, and now those issues have been eliminated by the new platform and processing speed.
On the back side, administratively and system maintenance-wise, there has been a significant increase in speed, building the data files, maintaining the data files, and doing any sort of testing. Applications are running 75% faster than they used to and there has been significant savings on the administrative side for maintaining the system, operating and testing it.
Mifflin County has plans for the system to evolve to meet challenges involved with next-generation 9-1-1 management, including location of cellular and VoIP calls, dealing with text messaging, video messaging, other available data sources, and working with other services to provide much more information to the dispatch center. Having the capacity to expand or deal with those enhancements is one of the bigger challenges all public safety offices face.
Phil Lucas is the director for the Mifflin County Office of Public Safety.