Bestselling author and former Wall Street trader Michael Lewis called for data analytics to be much-better harnessed by large organisations and governments in a Q&A session at Citrix Synergy.

In an event marking the final day of Citrix’s annual Synergy conference, this year hosted in Anaheim, California, the author of Moneyball, and The Big Short, discussed how technology has affected the financial sector, how data has changed sport, and the role AI plays in human decision-making.

He cited data belonging to the US government, «the biggest company on the planet», on how data analytics can be better-harnessed to not only gain insights, but develop smarter public policy.

«The federal government is a trove of enormous databases that people have only scratched the surface of,» he said, adding we are only starting to see this being better harnessed.

«The only reason we know about the opioid crisis is the national institute for health data on distribution of prescription drugs – which was made available to the public under the Obama administration in a way it was accessible.»

He told the audience «we wouldn’t even know» about the burgeoning opioid crisis affecting parts of the United States if not for analysts at Propublica, a nonprofit newsroom based in New York, taking and analysing this data.

Lewis also spoke about how technology had fundamentally changed the financial sector and how incentives to make short-term profits led to the financial crisis – issues at the heart of The Big Short and Flash Boys.

He explained how traders began to learn that their physical distance and location had an effect on how fast they were getting information about markets, and how high-frequency traders at BATS Global Market built faster networks to the other 12 exchanges scattered around New Jersey so they could detect big sell alerts and sell in front of rivals.

«What happened is, the stock market, instead of being about fundamental investment decisions, starts to become just about speed; how fast can you get from one exchange to the other, or how fast you can assemble a picture of the markets that is more accurate.

«If you can see prices before everybody else – it’s the ultimate form of insider trading.»

Lewis explained how traders began building faster networks to connect markets together, by trying to lay the straightest fibre-optic lines possible – a process that involved blowing up mountains, and hundreds of millions of dollars of investment, to gain only a few milliseconds of advantage, adding «there’s a kind of madness to it».

«There’s a point where the speed – it’s not actually adding anything to the economic efficiency – it’s all about gaming the market; all about finding out what people are doing and and getting ahead, or assembling a picture of the market that’s a millisecond faster than the market itself,» he said. 

The other side of the equation, he noted, was about slowing down the New York Stock Exchange, or Nasdaq, as another way of widening the gap: «The incentives are to widen the gap of time between when an ordinary investor gets a piece of information, and when a high-frequency trader gets a piece of information.»

«This is a malign use of technology; I don’t think this adds anything to the wealth of the society. It does make a high percentage of high-frequency traders very rich.»