NYSE Mayhem Traced to a Staffer Who Left a Backup System Running

(Bloomberg) — More than 700 miles from Wall Street, the New York Stock Exchange’s backup data center on Cermak Road in Chicago is supposed to safeguard US markets, standing by at all hours in case disaster ever strikes the world’s largest venue for trading shares.

When markets are closed, it participates in a well-worn routine, with NYSE staffers turning on and off systems to ensure everything works. But heading into Tuesday, an NYSE employee failed to properly shut down Cermak’s disaster-recovery system — leading to a disaster.

That human error, described by people with direct knowledge of NYSE’s internal operations, is what triggered wild market swings when trading opened Tuesday morning in Manhattan. The chaos affected more than 250 companies including Wells Fargo & Co., McDonald’s Corp., Walmart Inc. and Morgan Stanley, in some cases sending stock prices swinging by 25 percentage points in a matter of minutes.

The episode has prompted the exchange to cancel thousands of trades at a cost that’s still being determined. Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration.”

“They’re going to need to come up with something better” to assuage investors and regulators, said Joseph Saluzzi, a partner and co-founder of Themis Trading LLC, whose firm avoided losses. “Though systems fail, and we understand that, there’s zero tolerance when it comes to the opening and the close.”

Read more: Schwab criticizes NYSE over system failure that roiled trading

In plainer English, according to one of the people, the turmoil resulted because the backup system in Cermak was left running. That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices.

Without that routine step, orders flowed through at prices all over the place. That soon tripped circuit breakers designed to prevent violent market swings, setting off alarms on screens across the investing world.

NYSE executives spent hours pinpointing the problem until they were confident there wouldn’t be further fallout, said the people, who spoke on the condition they not be identified discussing the internal matter. Officials also began reviewing whether the dislocated trades could be labeled “clearly erroneous” under the market’s rules and then canceled. Shortly before 3 p.m. Tuesday, the exchange announced that it would unwind the most egregious transactions.

Fielding Claims

NYSE has yet to determine how much the debacle may cost. It’s already fielding claims from firms under the exchange’s rules, the people said. Those affected have until Friday, three days after the event, to describe any damages.

The venue gathers about $500,000 per month for a fund to compensate exchange members — mostly broker-dealers representing their investor clients — when such mishaps occur. Decisions on claims for Tuesday’s episode will be made by the end of the month.

Behind closed doors, NYSE executives are grappling not just with the potential cost, but the impact on the exchange’s reputation. They’re in contact with regulators and lawmakers to explain what happened. And they plan to examine the platform’s procedures and management, potentially reworking rules to be more flexible and provide further protections.

The exchange hadn’t determined as of late Wednesday whether anyone would face disciplinary action.

Charles Schwab Corp., the brokerage catering to retail investors, openly criticized the NYSE on Wednesday, saying it’s disappointed by how the incident was handled and that individuals may not get a fair shake when it comes to compensation. Schwab said the episode should also raise concerns about regulatory proposals that could push more trading onto stock exchanges.

“Unfortunately, the NYSE has not owned up to their full responsibility and retail investors will have to go through a lengthy process to correct orders, with no guarantee of a reasonable outcome,” Schwab spokesperson Mayura Hooper said. “It further heightens our concerns that routing even greater levels of retail orders to the exchanges will dramatically reduce the quality of the investing experience for America’s retail investors.”

 

More stories like this are available on

bloomberg.com

 

©2023 Bloomberg L.P.

 

 

 

Follow us on social media for the latest updates in B2B!

Image

Latest

IP730
ICOM at IWCE 2024: IP730 is the Solution to Scalable and Effective Communication Systems for Critical Services
March 28, 2024

At the International Wireless Communication Expo (IWCE) 2024 trade show in Orlando, the spotlight was on the latest advancements in communication technology, crucial for sectors like the military, emergency services, and public transportation. This event is a vital meeting point for professionals seeking innovative solutions to enhance their operational communication capabilities across vast distances. […]

Read More
Brand experience with APL
Stepping into the Future at Shoptalk 2024: Creating a Flagship Brand Experience with APL
March 28, 2024

The customers of today are no longer satisfied with mere products; they crave brands that provide immersive, distinctive, and tailored brand experiences. Athletic Propulsion Labs (APL) epitomizes this shift, setting a new standard in the luxury performance market with its innovative athletic footwear. APL’s newly opened flagship boutique in Soho, with its artisan plaster walls […]

Read More
Balancing professional growth and personal well-being
Routines, Rituals and a Mission-Driven Approach: MarketScale’s CRO Tim Maitland on Balancing Professional Growth and Personal Well-Being
March 28, 2024

In a world that often demands the relentless pursuit of success, finding the sweet spot between professional growth and personal well-being can seem like an elusive goal. As we navigate the complexities of modern work-life dynamics, the conversation around maintaining this balance has become more pertinent than ever. In this episode of Growthwell, we […]

Read More
marketing careers
The Impact of AI on Marketing Careers
March 28, 2024

In this episode of The Marketing AI SparkCast, host Aby Varma, a B2B Marketing Leader and the Founder of Spark Novus, sits down with Matt Gill, the Founder and Managing Partner of MICA Consulting Group. Spark Novus is a consultancy specializing in strategic AI adoption in marketing. MICA is a leading boutique national marketing […]

Read More