This did not affect you as a retail investor, as you get the NBBO price. This only affected you if you were an HFT firm with bad infrastructure who depended on the aggregate feed and not their direct line.

Like tptacek said, the price for getting a direct line, while expensive, is not unreasonable for a business. The largest cost will be salaries for the people writing your code and maintaining your infrastructure.

Maybe I misunderstood, but it seems like they're saying that's not the case:

FTA: "First, we had to rule out the possibility of the delay being caused by the consolidation process (which people often erroneously point out)."

They get their data from NYSE from apparently an aggregate feed, which was unintentionally delayed by NYSE. In my opinion, they should have been getting the direct feed and doing all consolidation themselves.

However, HFT firms will still have a vested interest in making more efficient aggregate feeds than the consolidators/exchanges.

Be that as it may in reality, per below quote I assume that behavior is still illegal.

"A crucial sub-ruling in the regulations prohibits exchanges from giving stock quotes to special groups faster than to the public. This sub-rule is of such importance, that without it, the rest of the rules (Reg. NMS) essentially become meaningless."

Please tell, how the world is supposed to provide that for free to anyone who wants it?

Colocated servers are the public.

The consolidated feeds are necessarily slower than the direct feeds, due to processing delays, unless the exchanges intentionally slow down their direct feeds. There are several companies that will sell you expensive FPGA or IBM Cell-based cards that will locally create a consolidated feed from all of the direct feeds. These cards aren't cheap, but are necessary for trading latency arbitrage strategies, and are also useful for low-latency market making.

Of course the consolidated feed requires a slight processing delay, so unless they intentionally slow down the direct feed, quotes from NYSE will appear first in the direct feed and very slightly later in the SIP feed. The argument here is about if the delay between SIP and the direct feed is small enough to be permissible.

The "public" being talked about is the paying public. Nothing in the regulation requires a free SIP feed, and I assure you NYSE doesn't provide a free real-time SIP feed. The regulation is to prevent selling an intentionally slow connection to everyone, and giving a faster feed to only their top tier customers.

Disclaimer: I'm not a lawyer.

Pretty simple.

The exchanges take in $500M a year in SIP fees, which are supposed to go to the most efficient hardware and software available. Pretty sure that's not happening.

Oversimplification, forgetting the laws of geometry and physics.

The triangle inequality and relativity conspire against the SIP. If you're getting the SIP from B, any new prices at C go to B and then forwarded to you, so it's faster (best case equal in the degenerate triangle case) to get direct feeds from B and C and consolidate locally.

I was just pointing out that

> If a private company can consolidate faster than the SIP, then the SIP isn't using the most efficient hardware and software available now is it?

is faulty reasoning.

But the law's the law. "It doesn't make sense" isn't a valid defense.

Usually due to capex: energy infrastructure, network effect: financial, or limited availability of resources: wireless spectrum.

If it's in the public interest to have trucks and no firewood seller would otherwise provide them. That's absolutely something government can (and IMHO should) mandate in exchange for a firewood license.

The retail investor gets the slowed down price (the NBBO is really what the internalizer/wholesaler saw from the SIP when they executed your order). That same internalizer buys/sells from the faster feed. Whenever the two feeds are out of sync, it's like printing money.

"The price for getting a direct line is not unreasonable?" I just laughed out loud reading that. Is $60,000/month for just one exchange (there are a dozen), reasonable? Then you need network engineers, infrastructure, etc. It used to be nowhere near this expensive for "real-time" data. Orders of magnitude less.

How much latency would be acceptable for a fully consistent, high throughput distributed log of events, that when it becomes unavailable the entirety of US trading shuts down and if it ever gets out of order can cause gigantic lawsuits?

How would you change the obvious organizational disincentives as well? Given that maintenance of the SIP is full cost, no profit operation for the exchanges?

Finally, don't take this as a criticism of your whistle blowing, I think that is critical to a functioning market, I am curious to know how you would solve the obvious technical challenges the consolidated feed faces?

I'd still love to hear you discuss it...

Similarly, people complain about the SIP feeds being latent and I'm sure it would be a very real problem if any single exchange being unavailable made the entirety of the SIP unavailable.

Its a fascinating problem because the engineering trade-offs are relatively static and well known, but each and every decision you could make provides an opportunity for arbitrage market activity that someone would consider predatory.

That we've layered a regulatory requirement that is pretty vague on some of the edge cases on top of this, just adds to it.

If you are asking about evidence that some people have claimed that the SIP being non-consistent is a problem for audit and regulatory compliance, is that not what you yourself are arguing in this post? http://www.nanex.net/Research/IsNBBOIgnored.html

I think maybe you are reading into my comment something that isn't there?

Edit: wow, rate limited after three posts this morning. A new HN low.

My response to tptacek below:

My dumb order?

How can HFT's intercept and redirect my trades to their, appently, captive pool of dumb trades?

Sure sounds like multiple markets are operating...and you are even telling me that my orders will be scraped before they can even reach some markets.

>is that HFT market makers will give you a price break to route your dumb order to them

You claim a price break on a private exchange. How can they (HFT market makers) deliver a better price than the public exchange? Presumably they cannot have bought it on the public market at the better price, so some party must absorb the loss on the price break.

My question can now be translated to:

Who absorbs the loss from the price break?

As a small trader, you get access to special lower prices that a hedge fund can't get. Your broker will be routing your order to a wholesaler who will fill it at lower prices (ie, narrower spreads) than you would get on the open market, because they have a legal obligation to not screw you over.

You can request that they route it to anywhere you want, and they are legally obligated to do it if you ask, but you really don't want to do that. Unless you like giving money away of course.

(Also, you don't seem to understand basic market mechanics. You use words like "scraped" or "picked up" which are nonsensical in context.)

IEX recently suggested that retail investors should request their broker to route their orders to IEX, for which they got criticised very harshly. And rightly so.

I used the word scrape in response to tptacek's response below my above post. If a trader is not aware that what tptacek describes is happening to their orders, then I believe my characterization is accurate.

As a small investor, your order is probably routed to a wholesaler like Citadel, where you will get a better price than you would otherwise get.

And you are upset because you incorrectly believed that your order would be routed somewhere else (where, exactly?), where you would get a worse price, which you think would be better because...why?

> I believe my characterization is accurate.

It's not even slightly accurate; it's literally the opposite of the truth.

Would you trust Wall Street to give you the best price on anything if there was no way to verify it?

There is no way to reasonably know if you are getting the best price (because they can change many times in the same second).

You are relying on the internalizer to give you the best of many possible prices. History says: "that's not going to happen"

Full disclosure, I have in fact worked with your data and at an HFT in the past but have moved out of the industry, if you were going to ask.

1) A dark pool. This is just another name for an exchange, but one that is not public. I don't know if NYSE runs one, but it wouldn't be surprising. The theory behind these is that they are provided as a service to large institutional investors to trade with each others outside the vagaries of the public markets for lots of reasons (mostly advertised as a way to not impact prices). The reality of these has been mixed at best. You have to opt into using a dark pool.

2) Payment for order flow. In this case your broker (not the exchange) has an agreement with a third party to sell your trades to them so that they can either trade with you directly or route the trade to an exchange. The reason they want to do this is that the aggregate of all the trades they are buying is not price opinionated (as opposed to hedge funds say) so the market maker can make the aggregate difference between all of them. This mechanism is largely how all discount brokers offer cheap or free trading, so it is likely a huge boon to you. A couple of things to note about this, your broker has to disclose it to you, I believe you are still required to get the NBBO price, & I believe you are required to be able to opt out by directing your order to be directly routed.

[edit] To clarify your question `jsprogrammer `tptacek is referencing #2 above. The reason the technical oddities & violations at NYSE don't have a huge impact on a retail investor is that unless otherwise stated, they are interacting with a wholesale market maker before their order hits any exchange, in a deal worked out with your broker and governed by legal obligations on both their parts.

A U.S. equities exchange has to be registered with and approved by the SEC. There are currently 12 of them. As a registered exchange you have to play by certain rules, and in exchange your displayed top of of book is subject to the order protection rule, meaning that no other venues (exchanges, dark pools, ECNs, or broker dealers) are allowed to trade through your top of book quote without first routing an order to you for the full amount of the displayed size. Exchanges also get a share of SIP revenue (which is important to exchange economics).

In it's most technical definition, a dark pool is an ATS (specifically not an exchange) that has no displayed quote i.e. no market data whatsoever. Most ATSs in the US fit this definition, though there are also ATSs that display a quote. For example, IEX is an ATS that has feed for their top of book quote... so technically not a "dark" pool.

NYSE doesn't run a dark pool and it would be weird if they did. Most US equity exchanges support fully hidden (completely non-displayed) orders, so if you want to trade in the "dark" on NYSE you can just send a hidden order. I think this is one area where futures exchanges and equity exchanges differ. In futures you have icebergs but not fully hidden.

2)

Wholesalers aren't allowed to give you a worse fill than the NBBO. That's the law. Often they will actually give you a tiny bit better than the NBBO on marketable orders. I think they do this so retail brokers can advertise price improvement. For most of the low cost retail platforms there is no ability to direct your order. You give it to the broker and I think basically if it's a marketable order they send it to the wholesaler and if not they post it to an exchange that offers a liquidity adding rebate (the brokerage keeps the rebate, natch). If you are willing to spend a little more on your brokerage account, you can use a platform like Interactive Brokers which does let you route orders directly.

Thanks for the clarification on the NBBO ramifications on wholesalers. I have never worked on a order flow trade and would love to see an insider account of them (its sort of what I was hoping Flash Boys would be about).

This is the case for retail orders, but not for trading firms. Prop trading firms don't tend to route their orders through wholesalers; they have deals with execution platforms (or implement their own) and route their orders to exchanges.

But there's a whole 'nother set of reasons why we're not especially concerned about competitions between prop trading firms on speed/latency/whatever: if they're impacted by HFT at all, they're competing in a zero-sum game to arb prices, and there's no moral reason why a slower trading firm is entitled to some equitable share of all the available profit from some arbitrage strategy.

If you were a retail investor or a mutual fund, what NYSE was doing probably didn't impact you at all.

   Q (CNBC): Should long-term investors care about milliseconds?

   A (Hunsader): Yes, they should.

Then where does all the money HFT firms make come from?

Second, HFT firms compete with other finance firms, so what money they do make comes from bidding down the costs those firms were imposing on the rest of the market.

If you're a retail trader, automated electronic traders make money off you by outbidding the markets to quote good prices to you for your dumb market orders, and then pocketing the spread --- in other words, they make money the same way market makers have always made money; they just make less of it at an instant, but at a far larger scale.

The same is true of mutual funds, which is why Vanguard --- the firm most people would say is the most trustworthy in all of finance --- has repeatedly claimed that HFT has lowered their cost of trades.

That seems like a lot of cost everyone could save on by simply having the exchange enforce some minimum timings.

There are two straightforward problems with microsecond-speed electronic trading:

* At very small timescales, possibly as a sort of inevitable consequence of the CAP theorem, correlations between instruments that should trade in lock step start to break down. Since you can make money correcting these mispricings, time and energy gets sunk into doing that, and that imposes costs on the rest of the market. It's uncertain how high those costs are, but they are small, and clearly a pittance compared to non-automated market making.

* Electronic trading provides very nice paychecks for pretty interesting technical work, which means some talent gets attracted to the market that would otherwise get paid less money doing something with more social value. How big a deal this is to you depends a lot on your worldview.

To me, neither of these issues seems worth the galactically high cost of restructuring all the markets in such a way that we'd reliably avoid these problems.

I don't mean to get rid of automatic market making, I'm saying the exchange could just move to a system where it checkpoints every 10ms or whatever, still much faster than humans can blink, but no detriment to the algorithms. Then everyone has 10ms to come up with orders that make sense in the current market situation, instead of the ridiculous timing game.

I'm not saying the markets must function this way, only that they do, and it's not evidence of a giant conspiracy or grift that they do.

The rest of my comment suggests that while we could potentially quantize the markets, it doesn't seem like we have much to win from doing that, and, further, that it's likely that the problems we're trying to solve by doing that would persist.

The window does nothing to remove the need for speed.

The world is a continuous system. No matter how hard you try you can't force it to be discrete.

(There are other problems in your proposed scenario to having to do with mismatched buy vs sell volume, but we'll ignore those for now.)

I think the above poster meant to imply that the 10ms delay would cover all market transactions, so if you put in an order now, then no one knows it till the next 10ms stop post.

Essentially its buffering all the data then releasing it at once, so it's not a truly continuous system anymore.

Trades occurring at other locations. Even if the other location are operating in discretized time, there's going to be variability in how fast different market participants can transmit information from one location to another. From Wolfram Alpha, Tokyo to NYC is 36ms at speed of light in vacuum and 51ms at speed of light in fiber, so someone with a faster network could be multiple discretized ticks ahead.

Stuff that happens in the real world. The stuff that all security prices are eventually based on. It's not a closed system you know?

For example, infrequent important events will result in a race for people to submit at the last microsecond, but because they're infrequent 99% of the time the 10ms buffering will effectively change the behaviour of market participants.

You can design the rules to optimize for one thing or another, continuous world notwithstanding. I'm suggesting a way to optimize for something more useful.

Are you perhaps confusing exchange in this case with a broker?

This is good for you, not bad.

What changes? What's the end result? How do normal people or society benefit? What would have to happen for the change to be declared a success. Conversely, what would have to happen for the change to be declared a failure?

What exactly is an "Exotic quote type"?

Do you mean order type because there is a huge difference. If you don't understand the difference it makes it hard to take your assertion seriously.

1) It is a common fallacy that there is "the exchange" or "the market". The market is an aggregation of exchanges (and non-exchange traded instruments) across a wide variety of jurisdictions and localities. But even though there is no central coordination between all that jumble, the instruments are correlated. That is, the change in price, demand, or availability of one is a signal about the pice, demand and availability of the others.

2) The market is made up of participants who exist on a band of price/time sensitivity. That is, someone like a retail trader doesn't have extreme price sensitivity but may have time requirements. In trading that style of participant is known as "dumb" (oh traders, you so PC).

3) Market makers make all their money trading spread for time but thats not where the latency games come into play. The latency is because...

4) Smart money is price sensitive (or price informed) and they tend to be where market makers lose money. Smart money tends to be either prop traders with particular algos, large block hedge funds or other market makers. Those are the people who cause market makers to play latency games, because its a vector a market maker can control that keeps them smart. They will never be smarter than the hedge fund that knows its going to buy a bunch of shares and therefore drive up the price, but they can at least jump out of the way quicker than the other smart money.

So in order to be successful, especially when your margins are razor thin (and they pretty much always are), you have to be very, very good at canceling your orders as quickly as possible any time it looks like the market is likely to move against you.

I.E. how much money is a good HFT firm making at scale?

Actual results for 2015. Your numbers are a fair bit off of those. What you can see from the results is that their revenue/profit numbers are small in real terms, but their margins are big compared to other businesses, even technology ones.

Personally, I'm not offended by that and think that the entire conversation about HFT is funny given how little money is actually involved, but if you view profitability in any context as evil or rent seeking, then you have ammunition for that argument with Virtu.

[edit]

Another good one to look at is Knight (Getco) http://investors.kcg.com/phoenix.zhtml?c=105070&p=irol-SECTe....

Had a much better year last year vs the previous 2 and it still was a fundamentally little amount of money.

[edit again]

That Knight statement is very interesting because it shows that the vast amount of their profit came from execution services and that their market making business was very low margin. Now that could be because their market making business is taking the full shot of the infrastructure accounting that enables the execution services business, but its interesting none the less.

> Full Year 2015:

> Net Income of $197.5 million

> Adjusted Net Trading Income of $500.7 million

His numbers seem pretty accurate.

That said, whenever any trading entity talks about their trading income, be suspicious, because that is usually just the difference between bought/sold (and fees) and doesn't account for the infrastructure costs of trading.

Thats why I like these public SEC documents so much, because they have to use GAAP so you know exactly what they mean.

1: http://www.bloomberg.com/news/articles/2015-04-21/goldman-sa...

Second: I didn't say HFT wasn't lucrative at all. I said it wasn't lucrative compared to other finance specialties. I would feel worse about this misunderstanding if the thread you were commenting on didn't include actual numbers backing this up.

Third: the ultimate reason exchanges "cater" to "HFT" is that they get paid based on order volume. That's what it means to be an exchange. Exchanges compete with each other. If nobody trades on your exchange, you don't make much money with it. Exchanges would rather you do business with them than not. It's not complicated.

Fourth: if you're in the normal demographic of HN (age 20-35 or so), you've never bought or sold a share of stock that went through a human market maker. Market-making is automated now and has been for something like two decades.

Fifth: if you're an individual buying and selling stocks for your own account by clicking buttons in an online brokerage, you are placing market orders. There aren't "minimum" and "maximum" prices. There's a current best bid, and a current best offer, and a gap between those two prices. We have decades worth of research on what automated trading is doing to that gap: it is slashing it from dollars to pennies.

Sixth: if you are placing limit orders, there is nothing an HFT or anyone else can do to "front-run" you to cheat on your order limit. Incoming orders are priced according to the price on the order resting on the book. When you place a limit order that doesn't immediately execute (in other words: when you place a limit order that is actually meaningful), that's the price your order is going to trade for, if it ever trades.

If you disagree with any of this, it would be helpful if you could lay out a hypothetical sequence of events in which this "front-running" is actually occurring. I place a LIMIT SELL order for 1000 CSCO on BATS at $27.90. An HFT decides to front-run me. What exactly happens next?

Please step me through this scenario. Exactly what do you think is happening, in what order and how?

Further, some whales (pension funds) might get hurt and that hurts your median citizen. Many hedge funds seem to have a strategy of whale-hunting. Thus dark pools and all that.

I'm not aware of a correspondingly simple way of acquiring base domain knowledge of the topics we are discussing here. Perhaps you can suggest some books?

The Complete Guide to Capital Markets for Quantitative Professionals (McGraw-Hill Library of Investment & Finance) Hardcover – 1 Dec 2006 by Alex Kuznetsov

A book on trading is prob more useful. There are many public specs of exchanges. For example:

http://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3...

The main thing to learn is orderbook building mechanics

Note: despite the fact that retail investors virtually never trade directly on exchanges, let's stipulate that they always do.

I don't think it's a good thing that the NYSE's feeds were this crappy; that's not what I'm debating.

There is no expectation in the markets that everyone is working with the same information, quite the contrary, the markets wouldn't provide value if everyone was working with the same data. One of the chief reasons the markets are valuable is that they give people incentives to surface (in the form of market activities) pricing information they might otherwise keep secret.

> I have to think that at some level, HFT will result in higher prices paid by brokers and their clients

Quite the contrary, it has led to a race to the bottom such that I can trade for free on my phone from anywhere with internet access at spreads that are nearly nothing. Robinhood is only possible because HFT has made it so.

> and directly or indirectly by mutual funds.

Vanguard, the gold standard for low priced mutual funds, completely disagrees with you.

https://www.sec.gov/comments/s7-02-10/s70210-122.pdf

From the Vanguard comments: This roughly 25% decrease in the end value of the investment demonstrates the impact of reduced transaction costs on longterm investors.

So perhaps you didn't state a position, but you posed a thought (I have to think that at some level) or question and Vanguard rather decisively answers that question.