Hendon Publishing - Article Archive Details
Is your agency prepared?
In the event of disasters, does your communications center have a backup site and a thorough disaster plan? What would you do if some type of emergency forced you to leave your communications center? Are you able to transfer operations from your primary site to the backup site? Most important, how do you mitigate risk and assure business continuity?
To ensure emergency medical care continuity to its residents during major disaster situations, Emergency Medical Care Inc., (EMC) the contracted provider of ambulance services for the Emergency Health Services System of Nova Scotia, has developed a thorough disaster plan that includes maintaining a fully redundant back-up site. In addition to continual maintenance to the primary and backup site facilities, the staff at EMC conducts semi-annual evacuations to the backup site in order to verify its operational readiness in the event of a disaster situation. These planned evacuations (Category 1 tests) allow EMC to make adjustments in operations and the disaster plan to ensure emergency medical care continuity with minimal disruption to services. This year, EMC conducted a Category 3 evacuation emergency scenario disaster test for the first time named “Operation Firestone.”
As the only secondary public safety answering point (PSAP) in the province, it is of crucial importance that EMC constantly provides the highest level of service and conduct operations without disruptions, especially during disasters. It is important to note that the EHS system is not a 9-1-1 service provider, but rather, a customer of the provincial 9-1-1 service, albeit the only customer in the province of this service.
“EMC is the sole provider of emergency medical services and ambulance transports for the Emergency Health Services System of Nova Scotia, so we provide all opportunities that allow this service to operate efficiently, including the provision of a fully redundant backup facility,” said Deborah R. MacKay, director of Communications and Dispatch Services Emergency Health Services, Nova Scotia. “With this model, we are assured that EMC [is] providing an optimum level of service during routine operations and [is] fully prepared for disaster scenarios.”
Building and Maintaining a Backup Site
Because Nova Scotia is almost completely surrounded by water, in which hurricanes are a formidable threat, the fear of losing primary operations is always there. In view of this threat, the province provided a backup facility at a location determined by the province. This facility has several key benefits that make it ideal, including that it is located further inland than the primary operations center; it is in close proximity to the primary site to minimize travel time during an evacuation; it has a separate electrical grid in the event of power failures; and it has a redundant, duel-entrance telecommunications voice and date network access point. The communication center backup site is also equipped with backup generator power and building / network security, plus the appropriate software and hardware similar to what is used in the primary communications center.
In the primary communications center and in the backup site, EMC utilizes TriTech’s VisiCAD Command computer-aided dispatch software (CAD). VisiCAD Command is a robust CAD solution that empowers EMC with the ability to rapidly respond to emergency situations and effectively manage any requests for emergency medical transport. It features an easy-to-use graphical-driven interface, and an integrated digital mapping system for fast, accurate dispatch and resource management. EMC’s computer-aided dispatch solution is critical application for EMC to consistently meet its contractual response times provided by EHS of Nova Scotia.
Additionally, EMC utilizes VisiCAD Command Disaster Recovery Solution powered by Replistor in its backup site. The VisiCAD Command solution remains online and fully available during data backup processes without disruption to the end users and ensures continuous availability with updated information at the backup site. Complete backups of the Master, MSDB, Messaging, System, and Streets databases are performed on a daily basis, with system database transaction log backups occurring at shorter intervals, such as every 30 minutes. The disaster recovery solution maximizes business continuity by providing file, database, and volume-level protection, intelligent and rapid data recovery and transactional-consistent recovery of active data. With the VisiCAD Command Disaster Recovery Solution, EMC’s primary CAD is replicated to the backup site in real time.
Because of the mission-critical software required for use by EMC, it has implemented a Stratus fault-tolerant server to provide 99.999% uptime and continuous availability. In this way, EMC eliminated the possibility of fail-over and avoided the ongoing administration needed to ensure that a multi-server cluster is properly configured for high availability. With reliable software and hardware in place, at the primary and backup site, EMC staff can focus on operations and maintaining a high level of service to the citizens of Nova Scotia. “TriTech and Stratus worked with our staff to ensure that our CAD and Disaster Recovery software and hardware is fully redundant and operational during a disaster scenario,” said Paul Gareau, manager of IT Services for EMC. “Once our technology infrastructure was in place, our operations staff comes in and tests the software to ‘break it’ to ensure that it is ready for live use.”
Testing the Disaster Plan
In addition to regular software and hardware readiness checks, EMC began conducting a thorough testing of its evacuation plan including “unplanned” evacuation scenarios to ensure disaster preparedness. The tests involve evacuating the communications center in real time as if it was a true disaster scenario. Operation Firestone was the first test of this kind, conducted in the EHS system by EMC. During the initial testing, EMC staff was able to evacuate the primary center and become fully operational at the backup site in less than 15 minutes, without prior notification or planning. (The 15 minutes is attributed primarily to the vehicle drive time to the backup center.) EMC will continue to conduct this type of testing as part of the semi-annual tests to the backup facility; refining the process and evacuation plan with each test.
“We take our disaster testing very seriously and conduct operations as if we were in a real disaster situation,” said Jeff Fraser, communications manager of EMC. “Each time we undergo an evacuation test, we conduct a thorough debrief to discuss what went wrong and what was right. We continually learn ways that we can improve our processes and ensure a seamless transition to backup operation and back again.”
The evacuation plan is a living document with minor tweaks and changes made after each test is performed, such as transportation and security processes. However, the cutover to VisiCAD Command and VisiCAD Command Disaster Solution software and Stratus hardware at the backup site is seamless, and limited changes are made to the hardware and software configuration. Technology and operations personnel at EMC work closely together to ensure seamless transfer of operations during evacuation plan testing and in preparation for disasters. Operation Firestone was a success in that EMC was able to undergo its disaster preparedness procedures without any disruption to public safety and seamless transitions to the outside world.
Benefits of a Backup Center
While evacuation tests are undergone twice a year as part of a contractual commitment, EMC is also planning to incorporate unplanned evacuation / disaster tests once every year as part of the semi-annual test. In addition, the backup site is also used as a test bed for new technologies, such as software upgrades and hardware installations before installing in the primary site. Because the backup site and the primary site have the same software and hardware configuration, it is possible to conduct rigorous testing, evaluate performance, and incorporate any necessary changes in the backup site before installation in the primary site. Similarly, the backup site is also used as a training facility for new dispatchers as well as a location for off-site meetings.
Most important, the backup site provides an assurance to the residents of Nova Scotia. With a fully functional redundant backup site, equipped with VisiCAD Command Disaster Solution Software and Stratus hardware, the public would not be aware of any disruption in operations in the event of an emergency.
Published in Public Safety IT, Mar/Apr 2008
Rating : 5.0
Click to enlarge images.