The possibility of intelligent, automated cyber attacks is the one that most worries John Launchbury, who directs one of the offices at the U.S.'s Defense Advanced Research Projects Agency, and Kathleen Fisher, chairwoman of the computer science department at Tufts University, who led that session. What happens if someone constructs a cyber weapon designed to hide itself and evade all attempts to dismantle it? Now imagine it spreads beyond its intended target to the broader internet. Think Stuxnet, the computer virus created to attack the Iranian nuclear program that got out in the wild, but stealthier and more autonomous.

"We're talking about malware on steroids that is AI-enabled," said Fisher, who is an expert in programming languages. Fisher presented her scenario under a slide bearing the words "What could possibly go wrong?" which could have also served as a tagline for the whole event.

How did the defending blue team fare on that one? Not well, said Launchbury. They argued that advanced AI needed for an attack would require a lot of computing power and communication, so it would be easier to detect. But the red team felt that it would be easy to hide behind innocuous activities, Fisher said. For example, attackers could get innocent users to play an addictive video game to cover up their work.

To prevent a stock-market manipulation scenario dreamed up by University of Michigan computer science professor Michael Wellman, blue team members suggested treating attackers like malware by trying to recognize them via a database on known types of hacks. Wellman, who has been in AI for more than 30 years and calls himself an old-timer on the subject, said that approach could be useful in finance.

David Fuller's view

Having read the article I am more certain than ever that scientists will be unable to control AI when it is capable of programming itself.  We have not yet reached that stage to any significant degree but we have had predatory software in action for the better part of a decade, particularly in more liquid financial markets.  I have written about it many times although not much recently.

Consider what I described as a version of sonar fishing, where high-speed computers could introduce and then pull orders in nanoseconds, enabling them to identify both large holdings in a narrow price range, or also vacuums of supply or demand. Once identified they were able to pressure large positions or front-run orders in a trending market. 

Smart, young Brad Fukuyama from the Royal Bank of Canada discovered what was going on and eventually created systems for slowing down High Frequency Trading (HFT).  He became a hero in the financial community and the star of Michael Lewis’ best seller, Flash Boys.

HFT is still around, of course, in various different forms.  Consider precious metals (PMs) and the Dollar Index (DXY) (weekly & daily).  For many months a large chunk of money has been buying precious metals when DXY weakened and selling PMs when DXY strengthened.  You can see this from the action of Gold (weekly & daily), Silver (weekly & daily) and Platinum (weekly & daily).