Will AI Make Data Science Jobs Obsolete in Finance?

Theoretically, rational machines should be able to make superior judgements to humans, who are susceptible to emotion, biases and external influences, as renowned psychologist and economist Daniel Kahneman and colleagues argue in Noise: A Flaw in Human Judgment.

That, however, does not necessarily mean that recent advances in artificial intelligence (AI) are set to put investment professionals out of work. Everything AI knows is based on the data it is trained on. So, if that data is corrupt, incomplete or even just sorted in an inappropriate way, the model will struggle to see the bigger picture, explained John Elder, Founder of long-established data science consultancy Elder Research.

That’s why the human element still plays an integral role in the machine learning age. “It's very important to bring to bear knowledge and information and caution from outside of the data to bear. Robotic techniques can help you save a lot of time, but they don't get the concept of information from outside the data. If you don't have low-cap stocks in the data, for example, they don't know low-cap stocks even exist,” said Elder.

Perhaps ironically, the more systems – and the testing of those systems – are automated, the more important it will be to exercise human judgement in monitoring them, said Julia Bonafede, Co-Founder of Rosetta Analytics.

"Whenever you're automating something, if you don't have human oversight of the output and the product that you're trying to deliver, the more likely you're going to have some issues in terms of how that model develops over time,” she said.

But even before embarking on building a model, it is vital to clean up the data. Financial data sourced from market information vendors need to be cleansed carefully to minimize the risk of errors, points out Dan Philps, an AI researcher and Head of Rothko Investment Strategies. All data vendors are liable to misreporting data through fat-finger errors or the way they import new data, and data scientists often spend the bulk of their time weeding out any inaccuracies. (See Figure 1.)

“It's a very expensive process. It's probably one of the most complex areas of an artificial intelligence pipeline. And if you haven't bothered to do it, or you're using a single vendor data source, or a legacy data validation approach that >fit for afactor-based approach, you've got a major problem,” he said.

The approach to data cleansing advocated by Philps is to use technology to identify the most accuratepockets of information across many data sources. Certain industries may be more accurately reported in certain data sets, where it can even come down to the strength of the underlying team that is generating the information in that pocket of the dataset.

“Then you can essentially prioritize data from that pocket before you cross validate,” he said. “Using AI to validate data is a key competitive advantage. More than adding alpha, it’s about avoiding big screw-ups.”

This particular use case gels with the widely accepted use of AI in the industry to serve as a “co-pilot” – a notion popularized by the rapid rise of ChatGPT – providing valuable insights, support and guidance, while leaving final decisions to humans.

“ChatGPT cannot directly provide you with the winning stocks that will go up 300% in the next month. But it can help you summarize information, from which you can extract important insights,” said Isaac Wong, Assistant Fund Manager at eFusion Capital.

Wong believes most people underestimate the current capabilities of AI.

“AI at this level is not flawless, of course, but it can already handle maybe 30% to 40% of our day-to-day work, saving a lot of time. It's a great tool to boost productivity at a very low cost. If we do not incorporate AI into our current workflows, our competitors who actively embrace it, will be able to outcompete us through significant cost savings,” he said.

One of the biggest challenges for data scientists is the explosion of data. The volume of data created each year — including data that is generated, captured, copied or consumed — is on track to reach 120 zettabytes in 2023. That is 120 billion terabytes, and a 60-fold increase from 2010. (See Figure 2.) The rapid expansion in our ability to create data has outstripped our ability to support, filter and manage it.

Around two-thirds of companies report having too much data to analyze and nearly three quarters of data within an organization goes unused for analytics. With the amount of data created set to continue skyrocketing in coming years and data science talent in short supply, the only way to address this issue is by empowering the wider business with AI tools to share the load.

But while the march of AI will help level the playing field in leveraging the power of data, it will certainty not render data scientists redundant – if anything, it could make them even more critical.

AI will, however, fundamentally alter the nature of data science jobs within the investment industry. Data scientists could become less involved with writing basic code and assisting stakeholders with data interpretation. But they will continue to play an indispensable role in providing context and additional quality assurance checks to shield organizations from risk.

Crucially, AI will free data scientists to broaden and improve their knowledge and skills – including learning more about the businesses they support. This could lead to an evolution of the role of data scientists in the investment industry.

In fact, by lightening the load for data scientists, AI could well raise expectations for them to be more creative by giving them the opportunity “to just try out different ideas,” said Krishnamurthy. “You have to step up in bringing value, otherwise you could be replaced by someone else who could potentially add more value at a lower cost.” 

Data scientists will also be essential to keeping up with the latest possibilities created by the incessant advance of technology – especially in relation to investment managers’ ceaseless quest for alpha.

“There are always new developments within the field. Especially now, new things are coming out at an almost exponential pace,” said Augustine Backer, Vice President, Lead Analyst of Investment Portfolio at Wells Fargo. 

“We need people who are dedicated solely to developing data science and pushing it responsibly to the next level, while also helping develop classes and ways to teach those things at different levels,” said Backer.

By prioritising data upskilling, firms will derive a significant competitive edge. After all, data has always been at the heart of the investment industry, with managers seeking to beat the market by uncovering unique data or novel ways to process it more quickly or intelligently than their rivals.

Though it may never be possible to extract the full value from all that data, AI will be key to narrowing the gap. Human creativity, judgement and oversight, along with the expertise of dedicated data practitioners, remain imperative – at least for the foreseeable future.

View more Data & Technology stories