IT audit finds holes in disaster recovery plan
By Simon Isham—
One flood on U of L’s campus could erase your academic career. A recent university audit of IT’s disaster recovery plan raised serious questions about how recoverable the university’s data is in the event of an emergency.
U of L performs regular offsite data backups. But opinions differ on how long it would take to restore the information.
A current U of L IT employee familiar with the data center in Miller IT Center said there are flaws in the university’s disaster recovery program. Because the employee fears retaliation from IT leadership and is not legally protected from such retaliation, the Cardinal has chosen not to publish the employee’s name. Earlier this year, current and former IT employees accused the department of discrimination.
“If there is flooding (in the data center), certain software will not be able to be restored,” the employee said. “If there were a fire tomorrow, it would take a month at best to get your services running again.”
Blackboard and ULink, the employee said, could be back up and running in one week’s time. PeopleSoft, a human resources management system, would take a month minimum. There are also several other programs on these computers that would require restoration.
“Imagine if something happened in the middle of final exams. Wouldn’t you, as a student, be angry?” the employee said.
Vice President for IT Dr. Priscilla Hancock refuted this information.
“Since all university systems are replicated offsite, they are recoverable and will be recovered in the event of a disaster. The length of time depends upon the amount of data to be restored. We anticipate Blackboard and PeopleSoft to be restored within 72 hours. Student email would not be impacted because it is stored offsite so there would be no loss of service,” she said.
“I’d say (72 hours) is fairly typical for a full disaster recovery situation when a hot site strategy is not affordable. As I understand it, U of L contracts with a vendor to provide off-site (disaster recovery) services. This approach leverages the expertise of DR specialists and utilizes their backup data centers at a lower cost than maintaining our own hot site somewhere. This plan would only be executed under major disaster circumstances, however,” said Dr. Andrew Wright, assistant professor of computer information systems at U of L.
The university has a contract with IBM in Sterling Park, N. Y. to provide off-site disaster recovery services. “All servers and systems located in the Miller IT Data Center can take advantage of these disaster recovery capabilities,” the U of L disaster recovery site reports.
The last audit of IT’s disaster recovery plan was completed November 2013. A report of the findings was submitted to IT and university administration in March. It was filed by Senior Auditor Barry Scott and Director of Audit Services Cheri Jones. The Cardinal obtained a copy of the report via FOIA request.
The audit says a member of IT staff present at the IBM backup facility could not access any of the data backups over the facility’s network. Scott noted that “This issue was addressed by alternative procedures, but will require resolution prior to the next recovery exercise.”
Of this issue, Hancock said, “The problem was fixed during the disaster recovery test. This is why we do the test.”
Scott identified two issues which he marked as “high priority” in the audit report.
The report said that in December of 2009, a prior audit noted the department was not reviewing and updating its disaster recovery plan on a regular, scheduled basis. In response, at the time, IT agreed to perform a review of the disaster recovery plan four times per year.
The 2013 audit found that the plan still included:
- Mentions of a former backup facility, but no mention of the current IBM backup facility
- References to technologies that the university no longer uses
- No references to application upgrades or changes since 2009
- Administrative contact lists containing the names and contact details of people who no longer work for the university, including the tax-evading former Dean of Education Robert Felner
“The lack of updated documentation suggests that a periodic review of the disaster recovery strategy and plan is not being performed as recommended in the December 2009 audit report,” Scott wrote.
When asked why the plan was not updated as scheduled, Hancock said, “Portions of the plan have been updated on a regular basis. However, there were areas that were not updated and should have been. We are currently updating those areas and expect to have this completed by September.”
The IT department responded in the report that it would update the information and review the disaster recovery plan by Sept. 30, 2014.
Scott also found that the tests performed in November were not extensive enough to ensure reliable data recovery.
“While it appears the exercise did achieve the objectives of the test plan,” Scott wrote, “the limited scope does not provide attestation to successful recovery of the university’s critical enterprise systems and applications during a disaster event.”
He recommended that disaster recovery testing strategy be expanded to include the most critical systems and applications, and should include a double-check by data-users to make sure the backup worked. The IT department should also prioritize the systems and applications that are most vital to the operations of the university.
The IT department responded that it would also address this issue by Sept. 30.
“This was an adequate test for it being the first disaster recovery test at a new site with a new vendor. We are expanding the number of systems that will be tested in our next annual disaster recovery exercise. Priorities were already in place and are now being reviewed on an annual basis,” Hancock said when asked about this issue.