Security News > 2004 > September > Nasdaq tests everyone's disaster recovery plans

http://www.computerworld.com/securitytopics/security/story/0,10801,95734,00.html By Elana Varon SEPTEMBER 07, 2004 CIO.com Three years ago on Sept. 11, Kamran Rafieyan and his co-workers walked down 83 floors of World Trade Center Tower One to safety. "Miraculously enough, we didn't lose any people," says Rafieyan, CIO of Lava Trading Inc. His then-fledgling company, a service bureau that routes equity orders for brokers, lost its data center when the tower collapsed. "We were in the midst of building our backup site, and we had to scramble for four weeks to get it up and running." Today, Lava Trading counts among its clients 16 of the top 20 investment banks and helps to process 15% of the daily trading volume on Nasdaq -- which is why on two Saturdays earlier this year, Rafieyan joined 49 other brokers and service providers in Nasdaq-sponsored disaster recovery tests. It was the first time that Lava Trading was able to test its disaster plans in an everyday business setting, rather than in a simulated environment. In the first test, Nasdaq had customers test connectivity from their backup sites to Nasdaq's primary site in Connecticut; in the second, customers tested how either their primary or backup trading systems connected to Nasdaq's backup site in Maryland. Steve Randich, Nasdaq's executive vice president and CIO, reports nary a technical hiccup in the entire proceedings. The tests were the first Nasdaq offered to its entire customer base. (In the past, Nasdaq has accommodated individual requests for testing whenever Nasdaq conducted its own.) With 9/11 and the August 2003 Northeast blackout behind them, it's becoming clear to many financial services companies that their survival depends on that of their trading partners. Regulators, meanwhile, are pushing securities traders to have proven disaster recovery plans in place. In April, the Securities and Exchange Commission approved a rule issued by Nasdaq's parent company, the National Association of Securities Dealers, requiring market participants to develop and disclose to its customers such plans by September 2004. Randich says his goal was "to be a host" and to allow customers to confirm their ability to failover to their backup systems.The key benefit, he adds, "is to promote the resilience of the market in terms of investor protection. If there were to be a major event, we can go to bed knowing it's not troublesome to restore operations in the morning." During the tests, participants worked individually with Nasdaq. Each company was asked to test whether it could submit orders, update quotes, submit and receive trade execution reports, and scan the system for executed and unexecuted orders. A few companies also used the opportunity to test their ability to send orders directly to each other (rather than to the market as a whole), just as they would during the course of regular business. Disaster recovery capabilities are often an important selling point for companies like Lava Trading that provide services to brokers. "If you're signing a contract with a large customer for trading systems, it involves sensitive data and mission-critical systems," notes Rafieyan. "They want to view your fault tolerance plans [and] your disaster recovery plans. They might ask to do regular quarterly testing." Through Nasdaq's tests, Rafieyan was able to confirm that his disaster recovery plans work, but he also discovered procedures he could improve. Combining some steps and automating others, he concluded, would enable the company to recover more quickly from a disaster. Restoring service might have taken only a few minutes during a test. In a crisis, "It could take two to three times as long," while the company runs through its escalation procedure to determine whether it needs to invoke its backup plans. "That's hard to simulate," Rafieyan says. Collaboration in business continuity planning may be more widespread in financial services than in other industries because financial institutions are used to collaborating at the transaction level, says Adrian Bowles, principal research fellow with the Robert Frances Group Inc. consultancy. But companies in other industries could benefit from cooperating as well. For instance, says Bowles, most businesses rely on a few large logistics companies to ship packages. "If [they] all go down the same day, business stops. It would make sense from an infrastructure standpoint to look at common failure points." Randich says Nasdaq gives frequent tours of its data centers to visitors from Fortune 100 companies looking for business continuity advice, including a large logistics company. "Their transactions are completely different, but their operation and technology requirements are similar," he says. Ultimately, maintaining business operations depends on cooperating with more than just a company's immediate trading partners. The next challenge, says Randich, is to be able to sustain operations if, say, a blackout extends for 36 hours and leads into a second trading day. "Then telecom companies and brokers start running out of diesel fuel to power their generators. They need transportation to help get deliveries" and a plan for emergency personnel to help make that happen. "The next level of improvement is a broader degree of cooperation from the metro police and service providers." _________________________________________ Donate online for the Ron Santo Walk to Cure Diabetes - http://www.c4i.org/ethan.html

News URL

http://www.computerworld.com/securitytopics/security/story/0,10801,95734,00.html