advertising information

CNN.com
 MAIN PAGE
 WORLD
 ASIANOW
 U.S.
 LOCAL
 POLITICS
 WEATHER
 BUSINESS
 SPORTS
 TECHNOLOGY
   computing
   personal technology
   space
 NATURE
 ENTERTAINMENT
 BOOKS
 TRAVEL
 FOOD
 HEALTH
 STYLE
 IN-DEPTH

 custom news
 Headline News brief
 daily almanac
 CNN networks
 CNN programs
 on-air transcripts
 news quiz

  CNN WEB SITES:
CNN Websites
 TIME INC. SITES:
 MORE SERVICES:
 video on demand
 video archive
 audio on demand
 news email services
 free email accounts
 desktop headlines
 pointcast
 pagenet

 DISCUSSION:
 message boards
 chat
 feedback

 SITE GUIDES:
 help
 contents
 search

 FASTER ACCESS:
 europe
 japan

 WEB SERVICES:
COMPUTING

From...
Network World Fusion

Online brokerages weathering storms

April 13, 1999
Web posted at: 11:56 a.m. EDT (1556 GMT)


In this story:

Growing pains
Dramatic rise in complaints
How it works
The Net is the problem
Preemptive measures
RELATED STORIES, SITES icon



by Lauren Gibbons Paul

Matteson
Fred Matteson   

(IDG) -- Fred Matteson, executive vice president of technical services at Charles Schwab & Co., sounds weary when discussing the recent highly publicized failures of his company's online trading site.

"I've had better months in my 21-year technology career," Matteson says, alluding to two lengthy February outages experienced by Schwab, the top online retail brokerage. Hot on the heels of these incidents came another crash in early March.

"The good news is we had no network-related outages - and have had none in recent memory," Matteson says, knocking on wood as he speaks.

Matteson and his counterparts at other online brokerages have learned to be humble. In the volatile online trading world, customer demand for speed and availability is taxing online trading networks. In the corridors of Schwab's San Francisco headquarters, this phenomenon is called Market Storm. Reminiscent of the Gulf War's Operation Desert Storm, the name reflects Schwab's view that the struggle to maintain adequate capacity is nothing short of war.
INTERACTIVE

Do you conduct any stock trading online?

Yes No
View Results

Online brokerages are racing to shore up their systems to enable them to flow with the tide of trading volume. All brokerages are going over their networks with a fine-toothed comb to ensure there is ample capacity and no single point of failure. Some brokerages, such as Fidelity, are using sophisticated Web management tools to closely track application performance and warn of looming availability crises well in advance.

Growing pains

Without question, 1998 was the year of the online brokerage. The total number of online retail trading accounts nearly doubled from 3.7 million in 1997 to 7.3 million in 1998, according to U.S. Bancorp Piper Jaffray, a Minneapolis-based securities market research company. The number of accounts is forecast to grow to more than 12 million by the end of 2000, and brokerages stand to make $3 billion this year from online trades. etrade

Along with the explosive increase in online trading has come a host of capacity-related problems. Schwab, E*Trade Group and Ameritrade each suffered significant outages in February. Nearly all of the major online brokerages have sustained service interruptions in the past two years, although not all were due to network problems. Schwab's recent outages, for example, were caused by glitches ranging from a limitation in a storage-management software program to human error in a DB/2 configuration (see chart).

Dramatic rise in complaints

Naturally, consumer complaints to the Securities and Exchange Commission (SEC) have burgeoned along with the outages (see chart). Delay or failure to execute a trade is the top consumer complaint, according to John Nestor, a spokesman for the SEC in Washington, D.C. Other common gripes include errors in processing and an inability to access the online system and the telephone support system.

Tales of online trades gone awry abound. Business Week recently recounted the misadventures of a Massachusetts man who reportedly lost $25,000 as the result of an error in his online Schwab account. E*Trade's stock price tumbled last month after a group of angry customers announced it was suing the No. 3 online brokerage over delays and errors in their trades. With nearly 100,000 online trades per day, Schwab is clearly at the center of the Market Storm. Its closest competitor, Waterhouse Securities, executes an average of only 42,000 trades per day, according to Piper Jaffray.

How it works

Online customers connect to schwab.com via one of hundreds of Netscape Enterprise Servers, which validate users' identities in the demilitarized zone just outside Schwab's firewall. Once a user is authenticated, he is given a session key and is allowed into the secure area. If a user wishes to make a trade, he is assigned to a trading server, one of hundreds of IBM SP/2 boxes running AIX. The trading server manages access to data and initiates transactions with the mainframes that handle mission-critical tasks, such as updating accounts and sending messages showing users' positions.

For example, if a user wanted to buy 100 shares of Microsoft stock at market price, the trading server would route the request to a third-party market-making firm, such as Mayer & Schwitzer, which executes the trade. (Schwab uses several of these firms.) The trade is reported into the NASDAQ system, where Microsoft is traded. NASDAQ routes a notice of execution back to the Schwab mainframe, which in turn sends the notice to the user.

A trade officially takes three days to process - a rare glitch could prevent the third party from executing the trade for a variety of reasons - but shipping the notice of execution back to the user usually takes only a few seconds. In most cases, the user can see his updated position in a matter of minutes.

Problems tend to occur around market opening, at 9:30 a.m. Eastern Standard Time, which is when traders begin their daily trading flurry. This new breed of online investor pushes Schwab and other online brokerages' systems to the wall. "Day traders believe their fortune is tied to being able to be in and out of securities in the same day," Matteson says.

MORE COMPUTING INTELLIGENCE
  IDG.net home page
  Network World Fusion home page
  Free Network World Fusion newsletters
  Tips for online investors
 Reviews & in-depth info at IDG.net
    Causes of some highly publicized online brokerages outages
  Tips for online investors
 
  Questions about computers? Let IDG.net's editors help you
  Search IDG.net in 12 languages
 News Radio
  Fusion audio primers
  Computerworld Minute
   

Day traders demand quick execution and notice of execution so they can check their accounts on an almost real-time basis. Schwab's Big Iron processes upwards of 150,000 trades per day - 93,000 of which are online trades - and more than 3.5 million CICS transactions per hour. A full 100,000 users can sign onto the system simultaneously. If all goes as planned, that number will soon increase to 200,000 when Schwab adds an eighth mainframe at the primary data center.

As for traffic volume, Schwab's recent high-water mark was 50 million encrypted hits on Jan. 8, although new high-water marks come fast and furious these days. "Our goal is to provide unencumbered access to the system at market open," Matteson says, adding that he hopes to reach that state by mid-year.

Schwab's experiences provide a valuable lesson for others in the industry. "The biggest leaps forward happen after outages, when we find a weakness we didn't know we had," Matteson says. "We rarely have the same problem twice." Matteson sees the recent storage-related problems as one of the last stages in the evolution of systems difficulties.

The Net is the problem

But there is still the Internet to deal with. "The Internet was designed to be an academic medium. It was not designed to handle this level of transactions," Matteson says.

Three years ago, when Schwab debuted its online service, most outages were caused by ISP problems. In 1996, for example, a rat stumbled onto a transformer, got electrocuted, and caused a huge short that brought down an Internet site owned by BBN Planet. "So the first problem we went after was resiliency," Matteson says. Toward that end, Schwab established multiple access lines to five ISPs. With three T-3 lines, Schwab has a total of more than 1G bit/sec Internet access capacity.

"Even with the outages that we suffered, our availability is still 99.1%, and the call centers were open to take trades during the outages," Matteson says. "We now have a highly redundant network inside. There is no single point of failure once the consumer has reached our site. We understand those problems very well. The sheer size of Schwab's online operations also brings a number of challenges. "More than one vendor has gotten that deer-in-the-headlights look when they see how much we're processing. We learn together," Matteson says.
Bai
Joe Bai   

Life has been pretty good for Fidelity, the No. 2 online brokerage in terms of number of accounts and No. 5 in trade volume, according to Piper Jaffray. The firm has experienced no outages this year, says Joe Bai, vice president of Internet engineering for Fidelity Personal Investing and Brokerage Group, a subsidiary of FMR Corp. in Boston. "I'm sitting here with a big smile," he says, albeit with his fingers crossed.

Fidelity says it is now processing 45,000 trades per day, up from Piper Jaffray's figure of 33,100 for the fourth quarter of 1998. Even the higher figure is fewer than half of Schwab's 93,000 trades per day, so Fidelity's trading systems don't quite face the same stress level as Schwab's. Still, Bai sees part of his job as keeping his company's name out of any headlines trumpeting system crashes.

"We make availability one of our differentiators; performance and features come next," he says. To safeguard against service interruptions, Fidelity duplicated its three-tier, Unix-based client/server system at three locations: Marlborough, Mass., Merrimac, N.H., and Dallas. WebSpective Software's WebSpective product provides load balancing across the three locations and allows Fidelity personnel at each site to manage content distribution as well as monitor system and application performance.

WebSpective Java agents collect data from Fidelity's various servers and react if any component isn't up to snuff. For example, Fidelity sends out small content updates to its 100 Web servers every 10 minutes. If, for some reason, one of the servers can't accept the update, WebSpective makes sure the fresh content is no longer sent to that location and the server is taken offline. This move guards against the possibility of some users receiving stale information, says Megan McDonald, director of marketing/communications for WebSpective in Needham, Mass.

Fidelity also employs WebSpective's Interceptor load-balancing capability at its primary ISP to protect against carrier outages.

"We worry most about backhoe incidents when the carrier disappears from our point of view. We have no control over this," Bai says. Should one ISP link go down, WebSpective can shift the load elsewhere.

"This is incredibly proactive load management," Bai says. "We need that because we have an enormous amount of load devoted to the Web channel. That worries me. I don't want to get blamed for [problems resulting from] a busy day."

Discover Brokerage, No. 8 on the list of top 10 online brokers with 11,530 trades per day, views each component of the network as a possible point of failure. The company has two mirrored data centers - one in San Francisco and one in Sandee, Utah - running Sun Unix boxes and Cisco network equipment. Each site operates at less than 50% capacity.

Preemptive measures

Darrell Davis, vice president and director of technical operations for Discover in San Francisco, says although his company hasn't experienced any serious outages lately, reading about Schwab's and E*Trade's recent woes provided a wake-up call.

"[The outages] prompted me and my staff to go look at our failover systems. I wanted to reevaluate our exposures very carefully," Davis says. Like Bai at Fidelity, Davis worries particularly about ISP-related outages.

"Things can happen downstream in the trading process to affect us that are not our fault. Our ISPs may not have enough redundancy," Davis says. Like many online brokerage firms, Discover routes its trades through various service bureaus, such as Automatic Data Processing (ADP). If ADP experiences an outage, a Discover broker calls and alerts ADP to hand off trades to another service bureau. Davis is evaluating ways to automate this process.

While Davis feels good about his company's recent track record, he knows he's not immune to problems. "I've been around long enough to know problems do occur. And Schwab and E*Trade have some very smart people working to solve those problems," he says. One thing is clear: The shift from conventional trading to trading via the Internet has meant a sea change for network managers and other IT professionals in the brokerage business.

"Our notions of scale have been tested again and again," Schwab's Matteson says. "We must have transaction by transaction redundancy and reliability - and all this must happen automatically. These are big network challenges."

Matteson admits he'll sleep better once the company's new mainframe has been brought successfully online.

Paul is a freelance writer in Belmont, Mass. She can be reached at laurenpaul@sprintmail.com


RELATED STORIES:
Good-bye to cut-rate Internet stock trades?
March 19, 1999
Get ready for round-the-clock trading on the Web
March 13, 1999
Online brokers under scrutiny
February 9, 1999
Complaints by online traders skyrocket
February 1, 1999

RELATED IDG.net STORIES:
Causes of some highly publicized online brokerages outages
Network World 4-12-99
Tips for online investors
Network World 4-12-99

Note: Pages will open in a new browser window
External sites are not endorsed by CNN Interactive.


RELATED SITES:
Schwab
E*Trade
Fidelity
Discover

Note: Pages will open in a new browser window
External sites are not endorsed by CNN Interactive.

 LATEST HEADLINES:
SEARCH CNN.com
Enter keyword(s)   go    help

Back to the top   © 2001 Cable News Network. All Rights Reserved.
Terms under which this service is provided to you.
Read our privacy guidelines.