AR T I F I C I A L I N T E L L I G E N C E
AND CRIME
Image: Mike mackenzie via www.vpnsrus.com
R E P O R T OF TH E AN U C Y B E R C R I M E O B S E R V A TO RY
FO R TH E K O R E A N I N S T I T U T E OF C R I M I N O L O G Y
ANU Cybercrime
Observatory
School of Regulation
& Global Governance
ANU College of
Asia & the Pacific
Electronic copy available at: https://ssrn.com/abstract=3407779
Artificial Intelligence and Crime A Research Report for the Korean Institute of Criminology December 2018 Australian National University Cybercrime Observatory Electronic copy available at: https://ssrn.com/abstract=3407779
Cybercrime Observatory, School of Global Governance & Regulation (RegNet) Australian National University, Canberra, Australia Director Professor Roderic Broadhurst Email address: roderic.broadhurst@anu.edu.au URL -­‐ http://regnet.anu.edu.au/research/centres/cybercrime-­‐observatory Researchers Matthew Ball Donald Maxim Paige Brown Alexander Niven Interns Elizabeth McCredie Katrina Pinn Jack Foye Benjamin Donald-­‐Wilson Joy Wang C. Kelly Bugler Corey Johnston Shanmuk Kandukuri Laboratory Co-­‐ordinator Harshit Trivedi Associates Dr. Lennon Chang Dr. Steve Chon Prof. Peter Grabosky Dr. Kee Siong Ng Dr. Ramesh Sankaranarayana Dr. Russell Smith Dr. Khoi-­‐Nguyen Tran Dr. Gregor Urbas ii
Electronic copy available at: https://ssrn.com/abstract=3407779
Acknowledgements We gratefully acknowledge the assistance of the Dr. Soung Jea Hyen and staff of the International Cooperation Center of the Korean Institute of Criminology and the support of the Australian National University, School of Regulation and Global Governance, College of Asia and the Pacific. We also thank Peter Grabosky, Matthew Ball, Corey Johnston, Elizabeth McCredie, Jill Mawby-­‐Tsutsumi, and Alexander Niven, for their assistance and contributions to this report. Suggested citation: Roderic Broadhurst, Paige Brown, Donald Maxim, Harshit Trivedi, and Joy Wang,(2019). Artificial Intelligence and Crime, Research Paper, Korean Institute of Criminology and Australian National University Cybercrime Observatory, College of Asia and the Pacific, Canberra, June 2019. iii
Electronic copy available at: https://ssrn.com/abstract=3407779
Preface Artificial Intelligence and Crime addresses the pressing challenges to cyber safety and security that automated crime-­‐ware or malware pose to increasingly common automated processes in many domains of modern life. The research paper also highlights the role artificial intelligence programs may play in preventing cybercrime, particularly in detecting and mitigating the interference and manipulation of the data relied upon for automated decision making or guidance. The research paper is the third in a series of papers addressing crime in cyberspace and follows earlier reports on Cyber Terrorism (2017; available at SSRN: https://ssrn.com/abstract=2984101) and Malware Trends on ‘Darknet’ Crypto-­‐Markets (2018; available at SSRN: https://ssrn.com/abstract=3226758). This series is the result of a fruitful collaboration between the Korean Institute of Criminology and the Australian National University’s Cybercrime Observatory. In a world increasingly linked by rapidly evolving technologies these research topics draw on common concerns of Korean-­‐Australian criminologists and underline the importance of cross-­‐national cooperation in the detection and prevention of cybercrime. Artificial Intelligence and Crime provides a brief entrée to the increasing importance of artificial intelligence programs and the machine learning methods that underpin automation in many civil, military and industrial applications. The review, however, focuses on the criminal risks of artificial intelligence (AI) programs and does not cover all the emerging theoretical and practical implications of the rapid ascent of machine learning technologies. This research paper describes broadly the potential offensive and defensive uses of AI programs and discusses some of the regulatory and legal implications. Jurisdictions with comprehensive cybercrime laws crafted with technological neutral assumptions seem best placed to respond to the potential risks of crime designed to exploit AI programs. At present artificial intelligence programs are essentially narrow rather than the ‘strong’ or super-­‐
intelligent forms often assumed to be in play and the potential risks to human rights, e-­‐commerce and communications accordingly less dire. All forms of technology including AI are open to misuse and offer unanticipated criminal opportunities. Law enforcement agencies will need to scale up AI programs designed to evaluate and analyse criminal intelligence and develop countermeasures that can neutralise the criminal use of AI. Although increasingly potent in many domains of modern life AI programs are unlikely to lead to either the utopian or dystopian futures anticipated by some. More likely will be a continuous struggle to understand the risks that AI programs may generate and how best to utilize these powerful technologies in the furtherance of humanities increasing environmental, health, governance and security challenges. Professor Roderic Broadhurst Australian National University Canberra iv
Electronic copy available at: https://ssrn.com/abstract=3407779
Foreword It is my great pleasure to present Artificial Intelligence and Crime, the third joint-­‐research project conducted by the Korean Institute of Criminology and the Australian National University. On behalf of the Korean Institute of Criminology, I would like to gratefully acknowledge the significant efforts put in by Professor Roderic Broadhurst and the researchers at the Australian National University, School of Regulation and Global Governance, College of Asia and the Pacific. Conceptually, AI is the ability of a machine to perceive and respond to its environment independently and perform tasks that would typically require human intelligence and decision-­‐making processes, but without direct human intervention. There is no denying that such AI systems are indeed present today and playing critical roles in the lives of people. There is a growing demand for expert data scientists to build this AI architecture. Also, a better understanding of such data is required to help both policymakers and companies make better decisions and empower the AI model. Especially, AI is also quickly becoming an important part of the criminal justice system. In other words, Artificial Intelligence is everywhere these days. In that sense, this joint research of KIC and ANU is of great importance in helping to lead the way in applying AI to address criminal justice needs. I have no doubt that this publication will provide the valuable step in helping scholars and professionals around the world interested in AI in criminal justice. It is my hope that this publication receives the widespread readership that it deserves, and that criminological partnership between Korea and Australia continues to thrive. Once again, I would like to express my appreciation for the hard work of all the researchers and members in the KIC and ANU who made this publication possible. Deputy director Soung, Jea Hyen
Korean Institute of Criminology
v
Electronic copy available at: https://ssrn.com/abstract=3407779
Table of Contents Acknowledgements iii Preface iv Foreword v Summary 1 Chapter 1 Introduction 4 Chapter 2 Algorithm and Threat Assessment Tools 11 Chapter 3 Defensive Applications of Artificial Intelligence 21 Chapter 4 Offensive Applications of Artificial Intelligence 28 Chapter 5 Regulating Artificial Intelligence 35 Chapter 6 Conclusion 43 References 46 Appendix 58 vi
Electronic copy available at: https://ssrn.com/abstract=3407779
Summary This review of the rapidly growing literature on the impact of artificial intelligence (AI) on crime and crime prevention outlines some of the likely limits and potential of AI applications as a crime enabler or crime fighter. In short, our focus is on the criminogenic potential as well as the preventative role of AI. Investment and interest in developing machine learning (ML) technologies that underpin AI capabilities across industry, civil and military applications have grown exponentially in the past decade. This investment in AI research and development has boosted innovation and productivity as well as intensifying competition between states in the race to gain technological advantage. The scale of data acquisition and aggregation essential for the ‘training’ of effective AI via ML pattern recognition processes also poses broader associated risks to privacy and fairness as such technology shifts from a niche to a general purpose technology. The potential weaponization of AI applications such as computational marketing, ‘deep’ fake images or news and enhanced surveillance, however, are pressing challenges to democracies and human rights. Poorly implemented ethical, accountability and regulatory measures will also provide opportunities for criminal activity as well as the potential of accidents, unreliability and other unintended consequences. We start in Chapter 1 by defining what AI means and how it has evolved from the early days of computing. AI is generally described as weak, medium or strong and it is widely understood that the AI currently in use can be classified as weak. This is because the underlying ML iterative processes are based on domain specific or limited ‘training’ particular to the problem or activity addressed and thus offer only narrow or highly focused AI applications. Five major functional areas have stimulated the civil development of AI and these are: ●
Voice and Speech Recognition is an interdisciplinary subfield of AI that blends acoustics, natural language processing and linguistics, in order to recognise the speaker and/or the meaning associated with a spoken input. ●
Computer Vision builds AI systems that obtain and interpret information from images. ●
Extended Reality (XR) is the amalgamation of AI and big data with Virtual Reality to create a user experience that allows interaction with and real-­‐time insights from data. ●
Game Playing is a stable of AI research. Among the first programs written in this area, were programs designed to play games like chess and checkers against human players. ●
Natural Language Processing (NLP) is a field of AI that acts as an interface between human languages and computer systems, often devising programs that process and analyse natural language data. Chapter two explores some of the issues surrounding AI as currently realised and argues that automated decision-­‐making technology should not be relied upon as a sole decision-­‐making tool and that scepticism is currently necessary when evaluating an AI’s performance. The United States White House’s Office of Science and Technology Policy has warned of the potential threat that AI poses to privacy, civil rights, and individual freedoms through the “Potential of encoding discrimination in automated decisions” (Executive Office of the US President, 2016, p. 45). Similar concerns have also been raised in Australia and other jurisdictions about the ethical use of AI. The growth of Investment and the anticipated role of AI in technological driven productivity are however changing society and ways of working (Dawson, Schleiger, Horton, et al. 2019). As with most new technologies AI is expected to generate benefits to society and do no harm, however, transparency, contestability and safeguarding privacy will be key elements in ensuring that potential criminal use can be thwarted. Some researchers have suggested that AI can be used as a method for predicting the places and times 1
Electronic copy available at: https://ssrn.com/abstract=3407779
that crimes may occur (e.g. hot-­‐spotting and geographical crime mapping); predicting and identifying individuals likely to commit crimes, and the most likely victims of crimes. Criminal justice databases have for over a century (cf. the ‘moral’ statisticians Adolphe Quetelet, and Andre-­‐Michel Guerry) been analysed to describe the patterns of crime and victimisation. These inductive generalisations about patterns of crime form the basis for attempts to develop the prediction capacity of AI in criminal justice settings. Attempts to estimate the recidivism of offenders or prisoners have been undertaken f o r nearly half a century based on the past history, and characteristics of known offenders as well as other factors via various means of analysis (e.g. statistical regression). However, such statistical or actuarial methods have yet to yield sufficient accuracy at the individual level and are often associated with high-­‐ levels of false positives and negatives. AI depends on using available (often institutional) data sets to train associated predictive models will also risk reproducing and amplifying inherited bias from incomplete or inaccurate data sources. This bias can be attributed to the explicit and/or implicit biases that may arise from the potentially selective nature of crime reporting and investigations. Consequently, such AI models may incorporate biases about the role of race or ethnicity (or some other arbitrary trait – mental illness, substance abuse, educational status etc.) as a cause of crime. Case studies, such as COMPAS, PredPol and Coded Gaze are described and demonstrate that the application of AI to crime has been highly problematic. The challenge of an AI algorithm designed to predict crime is that it may undermine the jurisprudential principle that one may only be punished for offences that have been committed rather than offences an AI application may predict are likely to occur in the future. Chapters 3 and 4 explore the various ways in which AI can be used as a defensive or an offensive tool in respect to cybercrime. Here the current and predicted uses of defensive and offensive AI are described. We evaluate some of the opportunities and risks that arise when AI is employed either to defend and detect attacks against computer systems or to attack computer systems. Chapter five summarises the legal and regulatory dimensions of AI and discusses the extent to which existing laws are sufficiently technologically neutral or ‘future proof’ to incorporate the use and potential ‘misconduct’ or misuse of AI in decision-­‐making. This chapter describes the legislative status of AI in several jurisdictions, which also illustrates the differences in the crime versus means approach to the regulation of AI. We also discuss how errors and mistakes that may occur with the application of AI decision-­‐making may play out in contest with established rights to privacy, natural justice and fairness. Many advanced economies have extensive laws prohibiting computer trespass, including the stealing, altering or manipulation of data that in principle criminalise tampering with data or the computer programs that use AI. Thus, criminal manipulation or interference of the data and program upon which an AI product relies is generally criminalised and laws protecting the data used by AI program already exist. However, the question of how liability for mistakes or harm arising from an erring AI program give rise to new questions which may not be adequately covered by traditional legal remedies, such as tort actions. Finally, chapter six briefly summarises the review by suggesting that despite the rapid development of AI the limited application of such systems in the criminal justice system is soundly based on the inherent difficulty of predicting individual behaviour with sufficient certainty. We do see an important role for transparent uses of AI to assist in the decision making of the judiciary, and correctional services as well as important functions in forensic analysis and intelligence led policing models. It is also evident that AI will be increasingly used to defend and attack computer systems and that the data relied on by AI systems will also become important targets. AI applications will also be used to enhance many existing surveillance systems incorporating and aggregating data from different sources including fusing individual biometric data with expenditure and activity transaction records to create ‘whole of life’ behavioural classifiers. PR China’s newly 2
Electronic copy available at: https://ssrn.com/abstract=3407779
piloted social responsibility system is such a system that ‘scores’ an individual citizen’s conduct (e.g. by rating the ‘social worthiness’ of consumption and activities) so as to prioritize access to state-­‐
controlled activities or services (e.g. state offices/jobs, travel, educational or medical services). The ubiquitous melding of data to classify conduct and rate individuals driven by AI algorithms may emerge as the key method of the new surveillance state or corporation. In this sense, AI may also evolve as a self-­‐preserving autonomous system. These new forms of total surveillance will also be a key target for manipulation by criminal actors and others. 3
Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 1: Introduction What is the future of artificial intelligence (AI) for criminology? How criminals can use it and how might it be used to detect criminal activity? AI is the endeavour to artificially create cognitive machines that act intelligently and can perceive as well as adapt to new situations. AI researchers believe that the cognitive decision making and automation potential of these machines can be on par with human intelligence. It is for this reason that AI has been a rapidly re-­‐emerging field in computer science for the last two decades (Kurzweil, 2005). AI has traditionally been applied only within the context of the digital world. However, we are now experiencing a growing trend for applications of AI outside the digital domain. For example, in the physical world, AI programs have become more common especially as industry and services move toward further automation of work. Applications that are used for medical diagnosis, robotics, and finance, are amongst the most widely used. Research in the field of AI has surpassed many milestones, with the development of technologies like mobile virtual assistance (Siri, Google Assistant, Cortana), driverless cars (Tesla, General Motors, Uber), game playing (Deep Blue, AlphaGo, Watson) and so on. However, it has yet to deliver significant developments in areas such as Artificial General Intelligence (AGI). There is a strong connection between the human psychological functioning and the rapidly expanding field of AI. In psychology, intelligence is defined as the capability of a being to apply its knowledge, in order to change its environment. Human intelligence can be defined as the ability to perceive the environment and to be able to adapt to it by learning from past experience and gained knowledge (Coward & Sackett, 1990, p. 297-­‐300). Effective adaptation to one’s environment requires perception, learning, memory, logical reasoning and problem solving (Gardener, 2008). This suggests that intelligence is not solely a mental process; it is rather a summation of these processes toward effective adaptation to the environment. AI is the field of computer science dedicated to developing machines that will be able to mimic and perform the same tasks just as a human would. So, the study of AI (and Deep Learning in particular) is the endeavour to develop a mechanized, simplified version of human neural networks and cognitive processing. Kurzweil simply defined AI as “the art of creating machines that perform functions that require intelligence when performed by people” (Kurzweil, 1990). Kaplan and Haenlein recently offered a more encompassing definition of AI that stated, “a system’s ability to correctly interpret external data, to learn from such data, and to use those learning’s to achieve specific goals and tasks through flexible adaptation” (Kaplan & Haenlein, 2018). This new paradigm of AI relies heavily on the acquisition, and use, of (large sets of) data in order to learn and gain experience from similar past events. As Clive Humby observed circa 2006, “data is the new oil” (cited in Arthur, 2013) and “big data” will continue to drive the development and application of AI to many new problems. Types of AI AI can be categorized in many different ways to distinguish both its ability and purpose. The most well-­‐ known categorization provides three levels: Weak AI, Medium AI and Strong AI. Weak AI, also sometimes called ‘Artificial Narrow Intelligence (ANI)’, is an AI system that is programmed to perform a single task, or combination of tasks in a narrow area of expertise. In other words, a machine’s intelligence is said to be narrow or weak, if it is restricted to a specific domain. Any system encompassing the concepts of machine learning, pattern recognition and natural language processing to make autonomous decisions can be classified as a weak AI (Bello, 2017). Most of the AI systems1 today, however advanced they may seem, are confined to a fixed set of tasks that they 1
The field of robotics has seen great progress in recent times, and modern service robots are more than capable of handling multiple tasks like cleaning a house, cooking a meal and providing emotional company to humans, 4 Electronic copy available at: https://ssrn.com/abstract=3407779
specialize in solving. Therefore, the AI in use today is largely weak and cannot be categorized as Medium or Strong AI (AI Index, 2017). Medium AI, also called ‘Artificial General Intelligence’ (AGI), is the intelligence with which a machine would be as capable of understanding the world as any human would, and has the capacity to carry out numerous tasks autonomously. AGI currently does not exist in real world applications, and has only been imagined within the realm of science-­‐fiction. In theory, an AGI should be able to combine human-­‐like intelligence and reasoning with the computational advantages near-­‐instant recall and split-­‐second number crunching (Heath, 2018). There have been several tests used by researchers over the years, to gauge the intelligence of a system, and to determine whether it falls under the category of AGI. The oldest and most popular of these tests is the Turing Test or the Imitation Game, devised by Alan Turing in 1950 (Turing, 1950). In this test, an interrogator, by means of judging the written responses to their questions, is given the task to determine which out of the two players is human and which is a computer. If the computer is successful in duping the interrogator into believing that it is a human, the computer has successfully passed the Turing Test. Most experts and AI specialists believe that we can achieve AGI within the next 30-­‐40 years2 as there are still unresolved problems arising from advancing AI to this level (Kurzweil, 2005). Strong AI, or Artificial Super Intelligence (ASI), is referred to as the systems that would not only equal, but also surpass human intelligence in all domains. This, of course, has also not been realized, and we can only speculate how powerful it may be. It would surpass all humans in all tasks -­‐ prescribing medicines to patients, creating fantastical literature, delivering pizza and much more (Yu, 2019). This also gives rise to the concept of ‘singularity’. British mathematician and cryptologist Irving J. Good, in 1965 speculated: “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control” (Good, 1965, p. 2). This would trigger a quantum leap of self-­‐improvement (autopoiesis) among machines, which can be called ‘Singularity’, where each improvement cycle would produce a better class of machines. There are contrasting schools of thought on what might take place when the singularity stage is reached. Some scholars believe that this can lead to numerous benefits such as raised standards of living, accelerated technological advancements and explosive economic growth (Kurzweil, 2005). However, some experts believe that this age of super-­‐intelligence would have a devastating impact on our environment, which may lead to World War 3 and possibly end humanity (Kurzweil, 2005). Perhaps, to understand these future implications, we should first step back and understand the origins of AI. but they would still be considered as Weak AI as they can’t replace the versatile expertise of humans or be perfect assistants to them. 2
Experiments performed in 2013 revealed that the Fujitsu K computer (then the world’s fourth fastest computer) took 40 minutes to compute the human brain’s 1 second of neuronal activity in real time utilizing the power of more than 82,000 processors. So, there’s still a long way to go before we achieve that, which is termed singularity. 5 Electronic copy available at: https://ssrn.com/abstract=3407779
History of AI The modern idea of artificial intelligence has its beginnings with philosophers attempting to describe the process of (human) thinking as nothing more than a mechanical manipulation of symbols (Turing, 1950). Though, it wasn’t until 1956, at the Dartmouth College Summer Research Project on Artificial Intelligence, that the phrase “Artificial Intelligence” was officially coined by American computer scientist John McCarthy (Moor, 2006)3. British researcher and cryptanalyst Alan Turing planted the seeds of modern AI during World War II through his work in deciphering the German Enigma codes (Copeland & Proudfoot, 1999). The Dartmouth Conference is considered by many as the formative event and foundation stone of AI as a distinct field of research. The proposal for the conference stated: “The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it” (McCarthy et. al., 1955). John McCarthy, who is widely considered as the father of AI and fellow academics Allen Newell and Herbert Simon were instrumental in promoting AI during its formative years (Ray, 2018). McCarthy also developed the LISP (LISt Processor – a pioneer mathematical notation system for computer programs) programming language in the 1950s, which was widely used during the 1950s and 1960s in finding proofs for geometrical theorems and solving other mathematical problems (McCarthy, 1978). Owing to these efforts, the coming decades were filled with optimistic predictions about the growth and potential of AI. From the mid 1970s through to the mid 1990s, there was relatively little research and development conducted on AI. This period is widely referred to as the ‘AI Winter’ (Buchanan, 2006). During this period, researchers realised that creating intelligence based on some AI concepts like machine learning and computer vision would require processing of an enormous amount of data. AI researchers had failed to gauge the magnitude and pitfalls of the problems they were facing (Russell & Norvig, 2003). The failure to show any results led to an acute shortage in funding from governments as well as big corporate firms in the coming years. There was resurgence in the field with the emergence of Expert Systems4 in the 1980s. However, this was short-­‐lived, as even expert systems couldn’t save the field from developing and R&D into AI declined in the late 1980s, owing to the systems’ inherent specificity and brittleness. Despite the absence of funding and public interest, AI still thrived, achieving many landmark goals during the 1990s and 2000s. In May 1997, IBM’s Deep Blue became the world’s first chess computer to beat a world chess champion, when it emerged victorious over world champion Garry Kasparov (3.5 -­‐ 2.5 after six games) (Schultebraucks, 2018). In the late 90s there was a renewed vigour and interest in AI, and governments started to fund new projects thanks to the improvement in computer hardware and growing popularity of machine learning. ‘Big Data’ became the new buzzword when McKinsey Global Institute estimated in their oft-­‐cited research paper that "by 2009, nearly all sectors in the US economy had at least an average of 200 terabytes of stored data" (McKinsey, 2011). In the following years, companies such as Google, Amazon and Baidu have leveraged machine learning to their advantage in order to store and process large volumes of data (Ray, 2018) or what is now often referred as “Big Data” (Lohr, 2013). 3 In 1955 McCarthy then assistant professor at Dartmouth and colleagues (Bell mathematician C.E. Shannon, IBM’s N. Rochester, and Harvard’s M. Minsky) proposed a revolutionary agenda “…that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves” (see http://www-­‐ formal.stanford.edu/jmc/history/dartmouth/dartmouth.html). 4
An expert system is a system that solves a domain-­‐specific problem using the logical rules derived from the knowledge of experts in that domain (Russell & Norvig, 2003). 6 Electronic copy available at: https://ssrn.com/abstract=3407779
Neural Networks have now tackled and simplified the problems relating to text analysis, video processing and speech recognition. Despite this rapid progress and evolution into a constellation of technologies that are collectively known as Artificial Intelligence, there is still considerable scope for development in the field of AI. As noted above, at the time of its conception, there was little if any relationship between AI and psychology or human factors aspect of machine-­‐human interaction. This was largely due to AI (and programming) being born from the mathematics field whereas psychology arose from the social and medical sciences. But as technology rapidly advances, we may be approaching a situation where we have to engage with AI as if it were a vague entity that requires us to analyse what it ‘wants’. Experts in the field have theorized about the impact of Artificial General Intelligence and Super Intelligence on human jobs and human lives, suggesting that economic growth associated with AI may only favour a few wealthy individuals or even pose an existential risk to humanity (Bostrom, 2013). As Hawking et al. (2014) stated, “Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.” Currently, we are at the cusp of a new era in AI as every major economic superpower is investing enormous resources into its development in order to gain a technological advantage over their rivals. With constantly evolving attacks and new forms of malware, Cybersecurity presents a significant opportunity for AI, especially given the potential to scale up pattern recognition. The global market is rapidly growing with AI focused cybersecurity firms, raising nearly USD3.6 billion in the past 5 years (Chaturvedi, 2018). Since 2010, funding for AI start-­‐ups have grown at a compound annual growth rate of nearly 60% (Accenture, 2017). So, it is safe to say that AI is steadily etching a mark in our daily lives and beginning to become mainstream. Predictive Policing with Big Data In Policing the Risk Society, Ericson & Haggerty argue that Policing is “not only a matter of intervention in the lives of individual citizens but also a response to institutional demands for knowledge of risk” (Ericson & Haggerty, 1997). ‘Big Data’ refers to large volumes of complex and variable data that requires advanced techniques for capture, storage and management of information (Gandomi & Haider, 2015). An important application of big data analytics is the use of algorithms as predictive tools for risk analysis or crime prevention. This gives rise to the concept of Predictive Policing, which involves predicting the time and place of future criminal activity through data analytics. This may also include identifying “individuals who are at a risk of committing a crime in the near future”, or creating “profiles that accurately match likely offenders based on data from past crimes” (Perry et al., 2013). Many police organizations around the world report using predictive policing tools or intelligence-­‐led policing to identify places where the likely frequency of crime is higher. The Police Executive Research Forum recorded that 38% of agencies use predictive policing methods (Police Executive Research Forum, 2014). There are two key elements to be addressed when adopting such a framework -­‐ the uptake or adoption of technology, and the impact of that technology upon adoption (Chan & Moses, 2019). There can be varying d egrees of uptake of a new technology. Similarly, the impact of a technology can be defined and assessed as experienced in the environment in which it is adopted. So, the effective implementation and u se of such a framework plays a huge role in deciding its potency. 7 Electronic copy available at: https://ssrn.com/abstract=3407779
Symbolic AI A key problem in the field of AI is to develop an in-­‐built layer of reasoning, logic and learning capabilities. Current systems have either the learning capabilities or the reasoning capabilities, and rarely have the capability to combine both (D’souza, 2018). Symbolic AI, or GOFAI (Good Old-­‐ Fashioned AI) was a concept developed in the 1950s that focuses on the symbolic reasoning and logic to represent and solve a problem. The advantage of symbolic representation is that the reasoning behind the solution of a problem can be logically explained (Bhatia, 2017). Although, to do this, the rules to guide the learning process have to be hand-­‐coded, which is time-­‐consuming and restrictive. Conversely, non-­‐symbolic systems do not manipulate a symbolic representation to find solutions (Bhatia, 2017). It is inherently difficult to understand how a system came to a conclusion, and so, it is very difficult to apply to problems like autonomous driving and medical diagnosis. Therefore, it is effective to use a symbolic representation approach to a problem, when we have sufficient information about the players and environment of a specialized intelligent system (D’souza 2018). Current Applications of AI Weak AI has many applications in today’s tech-­‐centric world. It has been incorporated into a wide variety of technologies that we either use everyday or that have an impact on our daily lives. These include smartphone technologies through to corporate decision-­‐making tools. Some of the typical problems to which AI is applied are as follows: ●
Voice and Speech Recognition -­‐ is an interdisciplinary subfield of AI that blends acoustics, natural language processing and linguistics, in order to recognize the speaker and/or the meaning associated with a spoken input (Russell & Norvig, 2003). Many voice and speech recognition systems today are considerably advanced due to the use of big data and deep learning (Russell & Norvig, 2003). The industry is segmented into sub-­‐fields specializing in speaker identification process (employed as voice biometrics in Immigration Services, Criminal Investigations5 etc.), speaker verification systems (Amazon Echo, Google Home etc.), speech recognition systems (Apple’s Siri, Google Now, Microsoft Cortana, Dragon Dictate among others) and text-­‐to-­‐speech software (TextAloud, Acapela and Amazon’s Ivona). The global market value of the speech and audio recognition industry reached USD 9.12 billion at the end of 2017 (Grandview Research, 2018). ●
Computer Vision -­‐ is a scientific discipline that is concerned with building artificial systems that obtains information from images and related multi-­‐dimensional data. It seeks to develop algorithms for automated visual understanding of images and videos. Computer Vision can be further divided in sub-­‐domains like object recognition, motion detection and estimation, scene reconstruction and event detection (Morris, 2004). Applications of computer vision for autonomous vehicles, and food quality evaluation for example incorporate deep learning and neural (convolutional) networks with increasing accuracy (Sun, 2016; Janie et al., 2017) Over the past two decades, computer vision has achieved significant advancements, thereby benefiting many scientific and manufacturing disciplines. Computer vision is being heavily used for visual surveillance, autonomous vehicle navigation, automated inspections in manufacturing and animal species identification among other jobs (Morris, 2004; Chen 2015). ●
Extended Reality (XR) -­‐ is the amalgamation of AI and big data with Virtual Reality to create a user experience where we can interact with and gain real-­‐time insights from data, rather than 5
In September 2014, British Intelligence invested heavily in voice recognition technology to try and identify ISIS militant ‘Jihadi John’ from the videos depicting the killing of US journalists Steven Sotloff and James Foley (The Guardian, 2014). The killer was one of the approximately 500 Britons recruited by the militant group in 2014. 8 Electronic copy available at: https://ssrn.com/abstract=3407779
conventional analysis of data. Using XR, participants become part of a virtual ecosystem and dissect data within their real-­‐world field of view (Culp, 2018). In essence, XR is an umbrella term that brings together Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) to enable data powered interactions between humans and machines. At present, this technology is mostly used in the virtual entertainment and gaming industry, but, may find numerous uses in other fields like healthcare, real estate and engineering in the near future (SaM Solutions, 2018). ●
Game Playing -­‐ has been an area of research among AI enthusiasts, since the very advent of AI. Among the first programs written in this area, were programs designed to play games like chess and checkers against human players. A game-­‐playing program exploits the superior computational speed of the system to run a brute-­‐force algorithm to generate as many alternatives, moves, and consequences as possible (UToronto, 1999). As the speed of modern computers increases, machines gain an edge over the human champions, accounting for triumphs like IBM Deep Blue's win over world champion Garry Kasparov (1997) at chess, Google AlphaGo’s win over 18-­‐time world champion Lee Sedol at Go (2016) and IBM Watson’s triumph over past Jeopardy! Champions, Brad Rutter and Ken Jennings (2011). ●
Natural Language Processing (NLP) -­‐ NLP is a field of AI that acts as an interface between human languages and computer systems, more precisely dealing with programs that process and analyse huge amounts of natural language data. NLP is fast becoming as popular as machine learning and computer vision as the majority of activities performed by humans today involve human languages (Expert System, 2016). Some applications of NLP include automatic summarization of articles and reports, sentiment analysis, question answering,6 and machine translation of a variety of documents (Davydova, 2017). The Evolution of Chatbots ‘Chatbots’ may seem like a recent buzzword; they have been around for several decades now. Chatbots are computer programs built to engage with and respond to messages received. Due to the nature of these programs, there are multiple AI concepts (Natural Language Processing, Machine Learning and Big Data Analytics) involved in their development. The first chatbot -­‐ ELIZA was created at MIT in the 1960s (Ireland, 2012). ELIZA gained immense popularity when it impersonated the responses of a non-­‐directional psychotherapist in an initial psychiatric interview. A.L.I.C.E. (Artificial Linguistic Internet Computer Entity), designed in 1995, was one of the most famous chatbots of the 20th century. It performed significantly better than other chatbots owing to its heuristic functions (AbuShawar & Atwell, 2015). A.L.I.C.E. was also the inspiration behind Apple’s “virtual assistant” Siri. Other chatbots that have gained popularity over the years are Parry (1972), Jabberwacky (1988), SmarterChild (2001) and Tay (2016). Today, all major tech companies have integrated chatbots and virtual assistants into their products to provide users with a more personalised service. Several chatbots have been designed with the sole p urpose of passing the Turing Test, such as the ‘Eugene Goostman’7 (2001) example, modelled as a 13-­‐year old Ukrainian boy and 6
Question Answering (QA) has become popular, with the advent of applications such as Siri, Google Now, Alexa and other virtual assistants. Chatbots are also becoming increasingly available with growing applications in toys, messaging applications and customer support. 7
In 2014, it was reported that Eugene Goostman had become the first chatbot to pass the Turing Test after it managed to fool 33% of the judges into thinking that it was human. However, the decision was fiercely debated 9 Electronic copy available at: https://ssrn.com/abstract=3407779
‘Mitsuku’ (2005), modelled as an 18-­‐year old female (BotList, 2018). Others, such as ‘Izzy’, a fluffy women’s health birdbot and ‘Casper’, a conversational agent aimed at helping insomniacs get through sleepless nights, are designed as a means to provide a human-­‐like conversation experience across different services or uses (BotList, 2018). In a recent survey by Oracle, 80% of respondents stated that they were already using or planned to use chatbots in their businesses by the 2020 (BI Intelligence, 2016). Currently, Chatbots and personal assistants are a growing part of our lives, and as the demand for them increases, they are constantly restructured to do more tasks, such as process payments, diagnose illnesses, manage finances and ultimately understand and empathise with how you feel. AI and Criminology The application of AI to matters of criminal justice is a recent phenomenon and is yet fully understood or explored. However, no longer confined to logical or technological questions AI has begun to be applied to social and ethical problems that engage a range of criminological problems from digital forensics to intelligence led policing. Google, like many technology companies have begun to examine the intended and unintended consequences of AI by forming the DeepMind Ethics & Society8 to investigate the ethical and impact of AI interacting with society. AI that purports to predict recidivism or the advent of intelligent sex dolls inevitably raises moral questions that need to be explored (Cooper, 2016). The advent of armed robots or ‘warbots’ also give rise to concerns about such killer-­‐ bots failing human commands and whether tomorrow's robo-­‐surgeons will honour the Hippocratic Oath (Tarantola, 2017) are discussed in Chapter 5 which address regulatory and legal challenges of AI, including faulty or rogue AI. Although restrictions can be hard-­‐coded into an AI operating system, just that additional nuance is needed to keep their integrity in check. During the 100th anniversary celebration of MIT’s Department of Aeronautics and Astronautics in 2014, Elon Musk, during his address, remarked: “I’m increasingly inclined to think that there should be some regulatory oversight, maybe at the national and international level, just to make sure that we don’t do something very foolish. I mean with artificial intelligence we’re summoning the demon.” (McFarland, 2014). In addition, many AI experts have been vocal about their concerns regarding the perils of uncontrolled and self-­‐aware AI. But only time will tell whether such predictions are ultimately worth worrying about or not. These issues will be explored in more depth in the following chapters. and criticised, partly due to the very short duration of the test, and also due to the fact that it was very cleverly given the personality of a 13-­‐year-­‐old boy, which would constrain the complexity and depth of conversation (BBC, 2014). 8 DeepMind Ethics & Society carrying out interdisciplinary research that asserts that all “…AI applications should remain under meaningful human control, and be used for socially beneficial purposes. Understanding what this means in practice requires rigorous scientific inquiry into the most sensitive challenges we face, and the inclusion of many voices throughout…Scientific values, such as openness, transparency, freedom of thought and equality of access guide every project we undertake’. See: https://deepmind.com/applied/deepmind-­‐ethics-­‐society/ 10 Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 2: Algorithm and Threat Assessment tools AI as an automated decision making tool offers many advantages to government and industry alike. As discussed in the previous chapter, it is already being widely used in both the public and private spheres as a powerful decision making tool, including by legal practitioners, judges and police (Moses & Chan, 2014). As the technology advances, it has the ability to transform the world. However, the technology is not yet perfect and still is developing. The belief that “Big Data” processing and automated decision-­‐making is always superior to the product that a human can create is an assumption widely held. This view that machines can solve everything has become the established narrative among technology firms as well as some academics (Degeling & Berendt, 2017). The term applied to this belief is solutionism (Morozov, 2013). The downside of this belief is that it often ignores the “socio-­‐technological contexts to which the technology is being applied” (Degeling & Berendt, 2017). The White House’s Office of Science and Technology Policy has warned of the potential threat that Artificial Intelligence poses to the privacy, civil rights, and individual freedoms through the “Potential of encoding discrimination in automated decisions” (Executive Office of the President, 2016, p. 45). Therefore, it is imperative that AI is properly scrutinised so that it does not become a liability rather than the asset it is marketed as. This chapter seeks to explore a selection of the most pressing issues surrounding modern AI and argues that automated decision making technology should not be relied upon as a sole decision making tool. It explores the potential for the negative impact that AI can cause to the social fabric and why understanding the technologies decision making limits must be thoroughly understood and scrutinised. This chapter also explores how government may regulate AI in order to ensure that the potential errors (the presence of false positives and false negatives) that unduly affect lives and institutions are limited. Big Data and AI AI that is trained on big data sets (also referred to as Machine Learning) can only produce a judgement equal to the quality of the data that it has been trained on. If the data that AI is trained on is incomplete or of a poor quality then the product produced will be equally flawed (Price & Ball, 2014). Data incompleteness is not a direct criticism of the data itself as it is very difficult to gather fully representational data set of certain populations or events (Price & Ball, 2014), but rather a qualifier of AI’s potential for inaccuracy. The issue is further exacerbated when data collection conditions are suboptimal as is the case within conflict zones (Price & Ball, 2014). The imperfect data collected and subsequently analysed has the potential to result in a large number of false positives or negatives as too few, and often imperfect, data points are accessible (Price & Ball, 2014, p.11). However, without a representative sample for AI to use and train a biased or inaccurate product is the likely outcome. The ‘precrime’ models adopted by some law enforcement agencies (LEAs) in the United States of America (USA) may offer flawed cost and efficiency measures when used to assess risk for different populations. Predictive policing is one of these measures. Predictive policing through the use of big data sets and AI has been rapidly adopted as a crime fighting strategy by many policing agencies. Predictive policing refers to the application of a combination of analytical methods in order to produce an estimate of the statistical probability of where and when a crime will occur (Degeling & Berendt, 2017). According to Perry et al (2013), these functions are routinely carried out by software packages, which utilise previously recorded data to make these futuristic predictions. Examples of predictive policing software that are popularly used include PredPol, Hunchlab: Risk Terrain Modelling, Chicago’s heat list, and Beware (Degeling & Berendt, 2017, pp. 348 -­‐ 351). A popular version of predictive policing has been the subject of science fiction film, Minority Report, where “precogs” were able to predict 11 Electronic copy available at: https://ssrn.com/abstract=3407779
future crimes for police (Gent, 2018). The approaches of predictive policing have been categorised by Perry et al (2013) as follows: 1.
2.
3.
4.
Methods for predicting places and times of crimes. Methods for predicting offenders and identifying individuals likely to commit crimes. Methods for predicting perpetrators’ identities. Methods for predicting victims of crimes. The first two approaches identified by Perry et al (2013) are also the most commonly automated by Western policing agencies, particularly in the USA. However, all four employ the use of ‘predictive analytics’ to produce ‘intelligence’ (Degeling & Berendt, 2017). Predictive analytics are the methods that software algorithms use in order to predict future trends or events based on the historical machine learned data sets. This data is then processed through ‘classifiers’9 (discussed below) that can be person or machine determined through ‘machine-­‐learning’ or ‘data-­‐mining’ (Degeling & Berendt, 2017). The data that a machine learns from can also be imperfect, and therefore cause the wrong classifiers or decision data making points to be learned in order to determine an optimal decision. The AI that learned from this data may be technologically perfect, however, if the data it learned from is incomplete or biased then the product created will reflect this. We next discuss several of the ways that an AI can produce a suboptimal result based on the data that has been used to teach it. The discussion is illustrative of the ways which machines can make the same mistakes as humans and therefore not be relied upon as the sole decision making tool for any government or business. The Data: How data can affect the learning of AI The data that AI uses to learn from is susceptible to several forms of contamination that can affect the quality and outcome of the analytical product it produces. These can be due to inductive (or context) bias, or a limited sample of specific populations or events and a one size fits all approach. Discussed below are several ways in which imperfect design or data sources can adversely affect the quality of an AI’s evaluation. Classifiers – limits and problems Classifiers comprise the basic decision rules for AI programs. They can be best described as informing the decision making tree. For example, if X is present in the data then perform function Y (Degeling & Berendt, 2017). Classifiers in data can be decided by an individual or through a machine’s learning process. These classifiers can be arbitrary, as those simply based on a hunch, or they can be decided through the statistical analysis of data in order to determine a causation effect (Degeling & Berendt, 2017). In order to produce an evaluation, classifiers are issued a weighting or value by either a person (i.e.-­‐ The FBI’s ‘Four-­‐Pronged Assessment Model’ for school shooters [Critical Incident Response Group, 1999]) or through the iterative process of machine learning (Derrac, Triguero, Garcia & Herrera, 2012). These weightings impact the evaluation and can produce “bad” results even if the training data is accurate and everything has been correctly classified (Derrac et al, 2012). It is therefore necessary to understand how classifiers are formulated and utilised in an AI’s decision making process in order ascertain the accuracy of the AI’s judgements. Classifiers are built from the analysis of “statistical regularities” derived from previous case studies (Degeling & Berendt, 2017, p. 351). The quality of ‘classifiers’ depends on the quality of the data as 9
A classifier is a decision rule that can be based on any criteria such as ‘intuition, prejudice, past human experience, or statistics’ (Degeling & Berendt, 2017, p. 351). 12 Electronic copy available at: https://ssrn.com/abstract=3407779
well as the opinions of the operators managing the cases or sources of data upon which the AI relies. If a machine applies these classifiers then it is subject to the programmers’ beliefs about what are the important categories. For example, the COMPAS threat assessment tool (below) attributes an inordinate amount of weight to the subject’s ethnicity when determining parole (Angwin, Larson, Mattu & Kirchner, 2016; Duwe & Kim, 2015). This resulted in individuals of African American heritage being denied bail when compared to Caucasians convicted of similar crimes (Berk, 2017). AI Image Caption Models (ICM) and CEM AI can also be used for Image Caption Models (ICM) in various fields. ICM uses a neural network, which can be implemented to encode images such that the output is a deterministic sequence of numbers (vectorization) that can be used by the model for analysis. The language-­‐based modelling neural network then uses the output from this model. This process is illustrated in Figure 1. This ICM can then be employed to categorize image captions, and by extension the images, into different categories by providing output into a Natural Language Processing (NLP) algorithm. Objects within the image may also be classified, such as plants, rocks, or a distinctive hide pattern. Extending this method to a specific object, face, or emotion is consequently viable. One potential application for this process is in the area of Child Exploitation Material (CEM). This process can be used in order to assess the content of potential CEM on a large scale without requiring the high level of human interaction that is currently required. This is beneficial both for consistency and also for the wellbeing of professionals in this space, reducing the amount of potentially distressing and harmful content that they need to manually review. Figure 1: Image Captioning Process Note: representation of the process of Image Captioning that identifies the input image as “a group of giraffes standing in a grassy area” (Automatic Image Captioning using Deep Learning (CNN and LSTM) in PyTorch). 13 Electronic copy available at: https://ssrn.com/abstract=3407779
These types of classification errors are not only the mistakes that humans make. A machine, while learning from a data set that places more emphasis on an offender’s ethnicity or gender can create similar sorts of classifiers. These classifiers, as seen in the COMPAS case study can then result in disproportionate and biased sentencing simply based on an individual's categorical variables. Although the promise of numerical guidance for judicial decision-­‐making may offer an expedient solution to problems such as an overloaded court system it also creates the possibility of producing false risk assessments and ultimately convictions. COMPAS Risk Assessment Tool AI based Risk Assessment tools, such as Northpointe’s the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), are becoming increasingly common across courtrooms and prison systems within the USA. These tools boast that they are able to accurately determine the General Recidivism Risk and the Violent Recidivism Risk of criminals (Northpointe, 2012, p. 1). Risk Assessments are produced as an integer out of a possible 10 determine if an offender is a “Low” (1-­‐4), “Medium” (5-­‐7), or “High” (8-­‐10) risk (Larson, Mattu, Kirchner & Angwin, 2016). However, in an examination of the cases where COMPAS was used, it found that the software was only 65% accurate in its recidivism assessments (Yong, 2018). Worse still, COMPAS was found in a separate study to only predict acts of violent recidivism in 20% of cases (Larson, Mattu, Kirchner & Angwin, 2016). This is little better than a random guess and is therefore at high risk of producing a false positive (conviction) or false negative (acquittal) result. It was further found in a controversial 2016 study by ProPublica that COMPAS was significantly biased against African Americans. The study reviewed assessments of white and black offenders in the USA, and found black offenders were routinely assessed with a higher risk of recidivism than white offenders that committed similar or more severe crimes regardless of previous convictions (Angwin, Larson, Mattu, & Kirchner, 2016). White defendants were more likely to be incorrectly assessed as low risk even with similar or more serious offences (Larson, Mattu, Kirchner & Angwin, 2016). Prediction Fails for Black and White Defendants White African American Labelled Higher Risk, But Didn’t Re-­‐Offend 23.5% 44.9% Labelled Lower Risk, Yet Did Re-­‐Offend 47.7% 28.0% Table 1: (Replicated from Angwin, Larson, Mattu & Kirchner, [2016, May 23]) This inherent bias has resulted in skewed sentencing based on race. For example, in 2014, 18 year old, African American Brisha Borden was assessed as ‘High Risk’ (a score of 8/10) of recidivism when she was arrested for petty theft after stealing a child’s scooter and bike worth USD80. Her record indicated that she had 4 prior juvenile misdemeanours and no convictions as an adult. Her risk assessment differs strikingly when compared to Vernon Prater, a male Caucasian, who had a similar charge for petty theft when he was arrested for stealing USD86.35 worth of tools from a Home Depot. Unlike Borden, Prater had a more serious criminal history. His prior convictions included two armed robberies and one attempted armed robbery. His threat assessment was computed to be ‘low’ (3/10). It was found that two years later when followed up that both assessments were completely incorrect as Borden had 14 Electronic copy available at: https://ssrn.com/abstract=3407779
not reoffended while Prater had been convicted of a grand theft offence (Angwin, Larson, Mattu, & Kirchner, 2016). So why does the COMPAS Threat Assessment tool inaccurately predict that African Americans are more likely to reoffend than Caucasian offenders? The reason for this disparity in risk prediction has been attributed the classifiers used in the COMPAS algorithm. Although race is not a classifier, the AI does make use of socio-­‐economic status, job status, income, associations with deviants and ultimately, prior convictions which are associated with racial class (Corbett-­‐Davies, Pierson, Feller & Goel, 2016). This trend will likely be exacerbated by the data collected through AI based predictive policing models (see Predpol Case Study below). However, it is difficult to accurately determine the actual cause of the AI’s perceived bias against African Americans as Northpointe maintains the secrecy of their AI’s algorithms. It is therefore, until the AI itself is examined independently by a third party, unknown if COMPAS is biased or is simply perceived to be. Figure 2: (Graph from Corbett-­‐Davies, Pierson, Feller & Goel, 2016) As a direct result of the questionable efficacy of the COMPAS Risk Assessment AI produced risk assessments warrant scepticism according to the Wisconsin Supreme Court (see State v. Loomis, 2017). Due to this uncertainty in AI’s capability to produce a fair and accurate prediction, other jurisdictions that use COMPAS or similar Risk Assessment tools may need to regulate their use and encourage a healthy level of scepticism in order to ensure a fair and transparent criminal justice system (see Broadhurst, Maller, Maller & Bouhours 2017 for a discussion about the use or limits the prediction of recidivism in Australia). The potential for bias caused by either human input or through machine-­‐learned bias should require the classifiers used by any AI should be questioned and not assumed to be infallible (Osoba & Welser IV, 2017). A degree of healthy scepticism and scrutiny should be levelled against the product, which an AI produces, as the data and classifiers used may not be accurate and may even be biased. The frequent absence of transparency about the specific form of an algorithm or decision-­‐making process also reinforces the need for oversight. This is particularly relevant in the decision-­‐making about a person’s criminal risk to a community and the quantum of their sentence. 15 Electronic copy available at: https://ssrn.com/abstract=3407779
A recommended method of avoiding classification bias is to employ regular testing of classifiers by measuring the quality of their predictions (Degeling & Berendt, 2017). The ultimate goal of this audit would be to verify the classifiers reliability in producing accurate predictions in order to justify its use (Degeling & Berendt, 2017). If the audit finds that the classifier produces too many false positives or negatives then it should be removed from the decision-­‐making tree and have its prior judgements thoroughly scrutinised and re-­‐evaluated. In order to facilitate the auditing process algorithms and training data should be available for scrutiny, and potential replication. This is important because the validity of any (AI) system can be best proven through the replication of findings by independent researchers (Degeling & Berendt, 2017; Osoba & Welser IV, 2017). This transparency and associated scrutiny by independent organisations or researchers would allow for a more open justice system therefore increasing the overall validity of the policing and prediction software and the justice system overall (Osoba & Welser IV, 2017). In order to facilitate this process, the producers of the AI software and custodians of the data would need to allow for a reasonable level of transparency for auditors to examine and evaluate their product. The Data Diet and Machine Learning The ‘one size fits all approach’ in machine learning has resulted in suboptimal analytical products and can lead to inductive bias. In algorithms, an inductive bias refers to the learned assumptions that a program makes when training with a set of data in order to create classifiers for its predictions (Degeling & Berendt, 2017). The bias is created through the learning examples that are created from its training data and these favour the simplest hypotheses for the product analysis (Moses & Chan, 2014). The bias then manifests in real-­‐world scenarios when data is presented that falls outside of the AI’s learning parameters. This in turn creates a weak analysis that performs no better than a random guess (Degeling & Berendt, 2017; Mitchell, 1988, p. 178). Inductive bias is present in machine learning models in the criminal justice system as shown in the PredPol case study (see below)10. PredPol PredPol is an AI based predictive analytics tool that is designed for the purpose of anticipating where and when gun violence will occur. The methodology leverages a city’s crime data, computer learning and cloud computing to produce crime forecasts. Of particular note is that the p rogram does not place a strong emphasis on past homicides as a classifier. Rather, the AI places more emphasis on precursor crimes such as “weapons violations, assaults, and batteries” to act as the predictors to deduce geospatial criminal activities (“PredPol releases gun violence prediction technology”, 2013). The program then produces a heatmap of where and what type of crime will occur, which will inform police officers where to focus during patrols. This is effectively using Situational Crime Prevention techniques to increase the chances of a criminal being caught and therefore deterring the crime from occurring at all (Juvenal, 2016). PredPol has not been designed with the intention of replacing veteran police officers and crime analysts but to be used as a tool to assist their decision making. The effectiveness of AI tools such as PredPol has proven to be questionable at best. Early adopters of the tool in the United Kingdom have cancelled their £100,000 p er year contract due to a lack of results from the program (Nilsson, 2018). Kent’s police Superintendent, John Phillips stated: “Predpol had a good record of predicting where crimes are likely to take 10
Falsified data fed to AI or data that has been purposefully tampered with by malicious actors to alter
machine learning outcomes, referred to as “Behavioural Drift,” is discussed in Chapter 3.
16 Electronic copy available at: https://ssrn.com/abstract=3407779
place… What is more challenging is to show that we have been able to reduce crime with that information” (Nilsson, 2018). The reluctance of early adopters to continue with the technology brings into question its effectiveness as a crime fighting technology. However, technologies like PredPol have other issues apart from demonstrating a lack of effectiveness in reducing crime. Data can also be skewed through over-­‐reporting or under reporting of crime (Lum et al 2016 p. 15). Due to community perceptions of police, a lack of a willingness to report crimes to police can cause a knowledge gap in predictive policing software. As a result, this may see some areas over represented in policing hot-­‐spot reports and others unrepresented. The AI may be functioning perfectly but if the data is flawed it will produce an incorrect or biased evaluation. This is evident in cities such as Chicago, where there has been a long history of racial bias in policing the African American community (“Serve and Predict”, 2018). Due to this, there has been an overabundance of police resources targeting African American communities, which results in more data points, reinforcing these communities as problematic (Saunders, Hunt & Hollywood, 2017). This creates a vicious cycle of targeting and retargeting, while ultimately ignoring other crime trends. When applied to an AI such as PredPol, it has been found that the technology does not remain neutral and unbiased but rather amplifies the effect (“Serve and Predict”, 2018). Effectively, incomplete and poorly implemented tools such as PredPol are “more accurate at predicting policing than predicting crime” (Edwards, 2016). A solution to these problems would be to train an AI using a synthetic population modelled from federal census data rather than policing data (Lum et al 2016 p. 16). This would help establish better classifiers and frameworks for an AI to use rather than transferring the implicit biases inherent in practice to the new system. However, like Microsoft’s chatbot Tay, a learning algorithm may still learn these biases if the ongoing reporting is biased and unrepresentative of real-­‐world crimes. As mentioned above, the data used to train AI is particularly important as it determines how specific classifiers are decided in a machine learning process. Ultimately, the AI can only be reliably effective in analysing a topic if the choice of classifiers is reviewed extensively within its trained data set (Degeling & Berendt, 2017; Osoba & Welser IV, 2017; Short, D’Orsogna, Brantingham, & Tita, 2009). If this data is incomplete and lacks crucial details, is historically biased or the input of data points lack a precise definition, then this can pollute the quality of training data being fed to a decision-­‐making tool (Degeling & Berendt, 2017; Osoba & Welser IV, 2017). This ultimately affects the overall veracity of the produced evaluation. Data sources can also be manipulated before they are introduced to the AI. In the infancy of AI, American Airlines sponsored the Semi-­‐Automated Business Reservations Environment (SABRE) tool for travel agents (Osoba & Welser IV, 2017). This program was designed to revolutionise the travel industry by providing the first algorithmic systems that would populate domestic USA flight listings and route information on a single platform. However, the standard sorting behaviour packaged into the software meant that the software would favour the sponsoring airlines over its competitors cheaper and more direct flights. It was discovered in antitrust proceedings that American Airlines created an unfair advantage for itself and were ordered to make SABRE a more transparent and fair system 11 (Osoba & Welser IV, 2017, p. 8). Another example of how AI can be misused through altering data can be found in the falsifying of police reporting. Eterno et al (2014) reported that some level of 11
For more information see Friedman & Nissenbaum, (1996). 17 Electronic copy available at: https://ssrn.com/abstract=3407779
corruption is present within police forces in the form of falsifying or manipulating crime reports for personal gain. These biased and falsified reports subsequently, whether intentionally or not used by the producers of the AI, are then part of the historical learning data set and the process undermined (Degeling & Berendt, 2017, p. 353). This means that human bias is transmitted to the learning algorithms of the AI and therefore corrupts the evaluation produced by the machine-­‐learning process. This bias can be accomplished through the design of the data output, or through the corruption of the training data. As a result, real world bias is able to transfer into a machine-­‐learned form of bias. A study conducted by Kristian Lum and William Isaac (2016) found that police-­‐recorded data sets ‘are rife with systemic bias.’ This bias has contributed to an overrepresentation of certain groups or neighbourhoods in police reporting as law enforcement allocate more resources to these areas which in turn lead to more arrests (Lum & Isaac, 2016). Furthermore, it was found that when AI learned from these data sets to train its predictive models it would reproduce and even amplify the inherited bias. This inherited bias has been attributed to the explicit and implicit biases that may arise from a reporting officer who considered race or ethnicity a cause of crime (Lum & Isaac, 2016). This has led to a large number of false positives in predictive policing models that could cause police to incorrectly identify and accuse certain categories of persons of being prone to committing crimes. For example, in the USA there is an over representation of African Americans and Latinos being charged with drug related offences compared to their Caucasian counterparts (Edwards, 2016; Mooney et al, 2018). This has been attributed to the explicit and implicit bias in LEAs in policing non-­‐ White communities. This data processed via a learning algorithm could determine, even if designers scrubbed racial data or was ignored by the AI, that non-­‐White communities coincide with geospatial data collected from police reports that identify their localities as drug-­‐use hot-­‐spots. The biased data would then be transformed into an assessment that would see an increase in targeting policing resources and result in more arrests and charges against non-­‐White communities. This would create a bias in policing priorities, as overall drug consumption in the USA is similar regardless of demographic group, although police arrest data overwhelming suggests drug consumption is an under-­‐class phenomena (Degeling & Berendt, 2017; Mooney et al, 2018). When challenged about bias by community members, the LEA could simply state that ‘the data doesn’t lie’ as a justification of these policing practices. Without scrutiny of the training data and an understanding of how the algorithm came to produce its assessment then it is very likely that the many systemic biases would be continued through the use of AI. It is therefore necessary that any agency or person that uses an AI’s product in any law enforcement capacity understand how the assessment was produced. The Coded Gaze One of the most popular applications of AI powered biometrics is facial recognition (for a more in depth discussion on biometrics and facial recognition see chapter 3). The ability to quickly identify and verify an individual's identity and to cross check it against a criminal database or white list of acceptable individuals makes the technology an incredibly potent tool in any security system. However, the technology is largely flawed and has not been reliably able to deliver an effective or accurate product. The London Metropolitan Police forces has also experimented with the use of facial recognition technology but have labelled the technology “not yet fit for use” (Sharman, 2018). A freedom of information request revealed the technology returned 98% false positives. A trial by the South Wales Police yielded similar results with less than 10% of their matches 18 Electronic copy available at: https://ssrn.com/abstract=3407779
being correct (Sharman, 2018). Effectively, the facial recognition technology currently employed by these policing agencies is useless. Figure 3: How facial technology works (Fox, 2018) So why is facial recognition technology unable to deliver on this basic matching function? One of the theorised factors is the ‘Coded Gaze’. The coded gaze denotes a form of racial bias (explicit or implicit) that comes from a lack of adequate representation within the data diet or its sources. For example, not enough women or people of colours’ unique facial features have been used in order to create effective classifiers beyond the established norms. According to a recent study by the Massachusetts Institute of Technology, this results in a high number of false positives if the assessed face falls too far outside of the programs norms (Hardesty, 2018). In short, an AI does not have enough adequate classifiers to properly identify the individual and thus make its facial match at best a guess. Overestimating the ability of modern biometrics or mismanaging the construction of these technologies can be a costly venture. Australian use of biometric matching technologies has yielded similar results to the UK case noted above with its attempts to employ facial 19 Electronic copy available at: https://ssrn.com/abstract=3407779
recognition and biometric technology. The Australian Criminal Intelligence Commission (ACIC) was forced to abandon its AUD $52 million Biometric Identification Services (BIS) project in the wake of a damning audit review that found the project was “deficient in almost every single respect” (Auditor General, 2019). The Auditor General of Australia found that the project had been financially mismanaged and some of the technological outcomes were currently impossible to deliver. Australia’s BIS highlights the financial and reputational hazards of poorly implemented biometrics and should serve as a lesson to policy makers. The data diet of a machine-­‐learning algorithm can be as noted severely affected by the quality and quantity of the data captured. In a study by Price and Ball (2014) it was found that algorithms are unable to accurately predict instances of large-­‐scale violence accurately in conflict zones such as in Syria and Iraq. This was due to the sporadic nature of data collection in these areas. The data was collected through what are described as ‘snapshots of violence’, i.e. -­‐ public killing videos on YouTube, truth commission reports, domestic and international media coverage, geo-­‐located SMS messages during protests and victim’s testimonies to non-­‐governmental human rights organizations (Price & Ball, 2014, p. 9). This data disparity can be further complicated by the relative geographical remoteness of these events. For example, crimes that occur in an urban region are generally well recorded (though as mentioned above, potentially inaccurately). However, when compared to rural regions, this is not the case, as many crimes are not reported. On analysis AI would produce a product that would conclude that violence is primarily an urban phenomenon and therefore does not occur at the same rate as its regional counterpart (Price & Ball, 2014, p. 11). AI also would struggle to make an effective judgement from the data when it is either too porous or poorly constructed and would therefore not be able to create a reliable or even plausible prediction. AI is a powerful decision making tool and is capable of revolutionising every aspect of human life. However, it is currently not an infallible tool for automated decision-­‐making and processes. This chapter has served as a brief overview of the current issues in AI and how these issues can be overcome through transparency and thorough scrutiny in order to create an effective tool that will help address the complex problems faced by humanity rather than exacerbate many of our existing societal flaws. 20 Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 3 -­‐ Defensive applications of Artificial Intelligence Defensive applications of AI in both the cyber and real world are becoming increasingly common. In the real world, there has been the implementation of risk assessments and predictive policing. Within the cyber-­‐world we have seen the rise of AI powered anti-­‐viruses, firewalls and threat categorization tools. As the uses of artificial intelligence in the defensive space are further explored, there will be an associated increase in the practical applications of such systems. There is a growing market for defensive uses of AI and, in particular, machine learning techniques to combat cybercrime (Microsoft, 2018). These uses include; malware detection, biometric security, vulnerability detection, and threat and intrusion detection, all of which will be explored further in this chapter. Defensive AI can be considered to be any system, which is driven by AI and acts in a manner to either react to threats o r to reduce access from unauthorised sources. With increasingly sophisticated offensive attacks, many businesses, companies and other actors are now required to respond with significant defensive capabilities in order to counteract these developments. This is a notable risk with the increase in offensive systems, as explored in Chapter 4: Offensive Applications of Artificial Intelligence, which make use of AI capabilities in order to execute more effective attacks. This has led to an increase in defensive systems that also apply AI and attempt to address the scale of attacks supported by AI. Routine Activity Theory is a useful model to gain a broad understanding of the strengths and weaknesses of AI when it is used in a defensive manner. Routine Activity Theory (RAT) is based on the presence of at least three conditions that are required in order for an offense to be committed. These are: a motivated offender; an accessible target; and a lack of a capable guardian (Felson, 1994; NSW Attorney General's Department, 2011). AI can be beneficial to preventing offenses by increasing the instance in which there is a capable guardian. As AI based systems can be used to monitor the defences of a system, they are able to act like a capable guardian even during times where there is limited human interaction (Bock et. al., 2017). This is particularly useful as it ensures that there are significantly fewer situations in which there is not a capable guardian, causing a reduction in the opportunities for attack. AI can also reduce the instances in which there is an accessible target by reducing the downtime of offensive systems and ensuring that resources are used in a more effective manner. However, the use of AI is not entirely beneficial, since this can lead to ‘solutionism’ as discussed in Chapter 2: Algorithm and Threat Assessment Tools. The dependence on the technology resulting from ‘solutionism’ can in turn reduce the effectiveness of the defensive system with an over-­‐ reliance in the AI systems infallibility leading to reduced human measures, increasing the accessibility of the target and reducing the instance of a capable guardian or proxy such as AI malware detection system protecting the system. Defensive AI is important not only for current high-­‐risk systems but also for other applications of AI. As AI becomes more prominent and is more widely understood, there is an increase in attacks that aim to damage AI systems. Malicious actors are able to exploit the “learning” process of the AI activity and teach the system that unusual behaviour is now normal, an occurrence which is known as “behavioural drift” (Bond, 2017). This is done through the introduction of data that has been designed to “confuse” the system for the purpose of misclassifying data points. Malware Detection Malware has become increasingly effective and sophisticated in recent history. These include examples such as Stuxnet, WannaCry, Industroyer, and DeepLocker (these are explored further in Chapter 4). The rapidly developing malware landscape requires a defensive tool that can detect and deter any attack. As a direct result, AI is the best defensive tool available in the cyber security arsenal 21 Electronic copy available at: https://ssrn.com/abstract=3407779
as it is able to learn, adapt and overcome new forms of malware and attack vectors. Crucially AI defensive systems can cope with large scale multiple source attacks. Cyber solutions are often the other side of cyber threats: the same technology can be used to develop an increasingly effective attack vector and also can be used to defend against these threats (Panda Security, 2016). For this reason, the same features that provide significant benefits to adversarial AI (as explored in Chapter 4) can also provide advantages to defensive systems. It is predicted that the changes made to cyber security by AI will fit into three types: scalability, adaptability, and variety (Dixon, 2018). While these were considered to be amplifications to the threat landscape from an adversarial perspective, the two-­‐sided nature of AI means that they can also be used from a defensive perspective. Scalability in system design refers to how well a system can handle a continuously growing amount of capacity in line with demand (Hasan, Baqai, Butt, Ausaf, & Zaman, 2018). Scalability is useful in a defensive context because it allows for systems to monitor and analyse a greater variety of potential attack vectors. When human actors are required to manually handle cases, the number of cases that can be handled is significantly lower than if AI is used (Vasudevan, 2018). The use of AI allows for the system to be scaled up to handle a greater number of potential attack vectors. As a result, cyber security professionals are able to apply more time and resources solving problems, rather than detecting them. Therefore, as the effectiveness of defensive AI develops manual intervention can be prioritised and the number of cases that require human involvement reduced. Adaptability is another element that AI has to offer in malware detection. Adaptability refers to the system’s ability to “learn” from past experiences and develop more effective future responses (Dixon, 2018). This is particularly useful in the field of malware detection due to the evolutionary nature of malware (both human and machine made). This ability to learn from past experiences provides AI systems with the ability to detect nuances or patterns that may go undetected by human actors. This is because the AI will have access to all data that it has previously encountered as well as additional data it has been provided with as training data. This will also allow AI to develop increasingly complex rules for the detection of malware as it learns from this data. Variety refers to the ability to introduce a wider style of methods for completing actions. In the area of malware detection, this can lead to the ability to develop new methods for detection or management of malware (Dixon, 2018). One example of this is the use of a good-­‐behaviour model that can be used to detect potentially malicious programs (Vigna, 2018). The current model for malware detection is based on a bad-­‐behaviour model (Vigna, 2018). The bad-­‐behavioural model works by detecting occurrences, which match previous incidents that have been classified as “bad” (Vigna, 2018). This model is flawed because this approach requires the system to have witnessed an attack, or a very similar attack, previously. Classifying an incident as bad is particularly challenging to achieve due to the rate at which malware is already evolving, a trend that is likely to be further increased as AI is incorporated into malware distribution more frequently. With the use of AI fuelled malware, there will be greater adaptability and variety in the malware that exists and a significantly lower chance that a single system will be able to keep up with the rapid changes. The use of a good-­‐ behaviour model addresses this challenge by alternatively modelling what is considered to be “good” and assuming that all other instances (of behaviour) is potentially “bad” (Vigna, 2018). By using a good-­‐behaviour model, malware detection systems are able to detect anything that does not qualify as “good” and then handle the potentially “bad” instances as required. This system requires the use of AI due to the large amounts of dynamic data and analysis in order to remain effective and not become obsolete (Vigna, 2018). 22 Electronic copy available at: https://ssrn.com/abstract=3407779
Automated Malware Analysis Malware is continuously evolving in sophistication and evasiveness. The financial cost of malware attacks is reported to be USD$10 billion a year (Kumar, 2018), and the cost is constantly increasing. Artificial intelligence, through big data and machine learning, has evolved and emerged as the most promising approach to detecting never-­‐seen-­‐before malware. Although it is next-­‐to-­‐impossible to defend against malware in real-­‐time, threat analysis tools that reveal malware behaviour have been d eveloped. SNDBOX is one such platform which performs analysis of files against different attributes by monitoring the file’s behaviour, but also allows users to search the (growing) online malware analysis database (Kumar, 2018). The process of automating malware detection is comprised of three main categories: -­‐ Analysing malicious files and websites -­‐ Recording and analysing network traffic -­‐ Analysing system memory and operating system application – program interface (API) calls Biometric Security AI is becoming increasingly popular in the field of biometric security. Biometric security refers to the use of an individual’s biometric data in order to determine their access privileges to a secured system or location (IFSEC Global, 2016). These systems will use AI to determine if the sample that is provided to the system matches one that has been stored. There are two basic types of biometric identifiers: physical and behavioural (Biometrics Institute, 2018; Rouse, 2015). Physical biometrics refers to identifiers such as iris scans, fingerprint recognition, and voice recognition (Biometrics Institute, 2018). Behavioural biometrics refers to identifiers such as keystroke dynamics and gait analysis (Rouse, 2015). AI is particularly useful as a tool for biometrics as it is able to detect and analyse small discrepancies present between valid samples. Facial recognition, for example, is a form of biometric recognition, which often may not appear exactly the same between all the relevant samples available (Li, 2004). Emotions, angles, lighting, and other factors can alter the way that a face appears and cause a literal comparison to fail. Through the use of AI matching or pattern recognition systems facial features that are indicative of the unique individual can be focused upon allowing small discrepancies such as those note above may be ignored. A method that is more prominent in biometric security is keystroke analysis. Keystroke analysis is based on keystroke identifiers such as the speed and error rate of an individual’s typing (Rouse, 2015). This method is increasingly used in combination with other security measures such as passwords as a two-­‐factor authentication system to ensure that the individual entering the password is not an imposter. This is also sometimes used continually throughout a user’s session in order to ensure that the current user is the same as the one who logged in to the system. Methods such as these are becoming increasingly popular due to the authorised user not being able to forget them, a common problem with passwords, and being difficult for unauthorised users to forge in order to gain access to the system (IFSEC Global, 2016). While it is becoming increasingly difficult for unauthorised users to gain access to these systems, the technology is not always fully reliable, 23 Electronic copy available at: https://ssrn.com/abstract=3407779
making some systems of biometric security more suited to being one half of two-­‐factor authentication than as the sole factor in determining authentication (IFSEC Global, 2016). Biometric Security Biometric security refers to the use of biometric identifiers to allow access to a particular system. There are two main types of biometrics: -­‐ Physical Biometrics: These are comprised of unique physical characteristics that an individual can be identified by. Examples of physical characteristics include fingerprints, voice identifiers, veins, and iris patterns. -­‐ Behavioural Biometrics: These are identifiers, which are developed around the behaviour that individuals routinely exhibit. This can include identifiers such as keystroke analysis. Despite the use of sophisticated biometrics, stored reference features are unlikely to precisely match features drawn from current measurements (for example as people age). For this reason, it is important that security measures have a level of flexibility in their algorithms. Too tight and a positive match is unlikely to be obtained even on the correct individual. There is also a risk that a false positive will occur and an illegitimate user may gain access. The challenge in biometric security is to develop rules flexible enough to cater for expected changes in legitimate users, without allowing in illegitimate users. A false match rate (FMR) calculates the proportion of false positives (Thakkar, D., n.d). It is the rate at which a biometric process mismatches biometric signals from two distinct individuals as coming from the same individual, and is given by: FMR = number of successful false matches÷number of attempted false matches A false non-­‐match rate (FNMR) calculates the false negatives. It is the rate at which a biometric matcher miscategorises two signals from the same individual as being from different individuals, and is given by: FNMR = number of rejected genuine matches ÷number of attempted genuine matches There is a trade-­‐off between a lower FMR and a higher FNMR (and vice versa): making the criteria stricter reduces the false match rate but also increases the false non-­‐match rate. Finding the right balance between the two depends on the application. The false positive identification rate increases as the number of entries in a database increases. The equal error rate (EER) is given by setting the FMR and FNMR to an equal value. Error rates of current matching schemes: • Facial matching: an FNMR of 1.1%; • Fingerprint matching: 1 finger print has an FNMR of 0.6% and an FMR of 0.01%, 4 finger prints have an FNMR of 0.1% and an FMR of 0.01%; • Iris pattern recognition: an FNMR of 1% and an FMR of 0.1%; • Palm vein matching: • Voice matching: an FNMR of 10% and an FMR of 2%. Vulnerability Detection Vulnerability detection has become an increasingly popular use of defensive AI. Vulnerability detection is a more proactive defensive measure in which a computer system is scanned for the presence of vulnerabilities (Fadelli, 2018). Effectively, a form of automated penetration testing. AI is particularly useful in this area due to the adaptability that it offers. An AI based system designed for 24 Electronic copy available at: https://ssrn.com/abstract=3407779
vulnerability testing and trained on models that have been successful at exploiting vulnerabilities in other systems could be used as a preliminary measure to determine what weaknesses exist. A defensive AI trained by an offensive AI would be better prepared to thwart attacks by similar or even less complex techniques therefore making it a more effective tool (Fadelli, 2018). This also increases the likelihood that the forms of defence available will be able to combat a sophisticated AI based attack. Such measures will not remove all risk of an attack being successful but will decrease the likelihood of a successful compromise of a computer system. A possible advantage of AI-­‐based vulnerability detection is the ability to automatically patch identified flaws (Condliffe, 2017). If a system is trained to understand the types of patches that were applied when a similar weakness was discovered in another system, it could then apply a similar patch to the system that it is working on (Bush, 2017). In the short term, this would not replace human actors who would still be required to perform further tests, ensure that the patches are effective, and follow up on weaknesses that the system is able to identify but not resolve. However, this would reduce the human workload involved in identifying and patching common weaknesses as well as identifying weaknesses. The latter is of particular importance when it is considered that a similar AI system may be used to perform an attack. Automated Application Security Platforms Vulnerability detection tools have been long been suggested to help developers detect and repair different types of vulnerabilities in their software. However, these tools require manually generated repair templates, and are developed ad-­‐hoc to focus on a single type of vulnerability. Manually generating the repair template is laborious and time consuming, and a user is unlikely to capture specific repair templates for all possible vulnerabilities. By learning from a training set of repair examples, the tool VuRLE (Vulnerability Repair by Learning from Examples) is able to automatically identify vulnerabilities in software, generate a group of possible repair templates to those vulnerabilities, simulate the repaired software, and conclude which repair template is the most appropriate given the simulation data. This enables VuRLE to repair multiple types of vulnerabilities. The researchers ran the VuRLE tool on 279 vulnerabilities from 48 real-­‐world applications using 10-­‐fold cross validation setting. Their results were the successful detection of 183 (65.59%) out of the 279 vulnerabilities, and the automated repair of 101 vulnerabilities (Ma, Thung, Lo, Sun, 2017). Threat and Intrusion Detection Another use of AI in the defence of a computer system is that of threat and intrusion detection. Threat and intrusion detection refers to a system that is able to use AI to detect threats and malicious activities. This is accomplished through methods similar to the malware detection and vulnerability protection explored above. Threat and intrusion detection is important, particularly for organisations such as government departments and multinational companies, to determine if a breach has occurred within their system and the extent to which the system has been compromised. A report by Logrhythm (2018) states that “…in 2016, the average time it took for an organisation to realise it had been breached was 99 days.” This is a particularly concerning statistic when considering the extent of penetration and access that an attacker could gain. Rapid detection is becoming increasingly important with the increasing 25 Electronic copy available at: https://ssrn.com/abstract=3407779
potence of attacks against computer systems (Logrhythm, 2018). The longer an attack goes unnoticed, the more damage it has the potential to cause to the target system and the longer it has access to potentially compromising data (Logrhythm, 2018). AI is particularly beneficial in the field of threat and intrusion detection for two main reasons: it has the ability to effectively parse very large amounts of data and is able to analyse a myriad of potential attack vectors faster than any human. This capability is particularly useful for assessing actions, as the now growing mass of data would be insurmountable without the assistance of AI. By using AI, this data can be more rapidly assessed in a meaningful manner leaving only more complex assessments to be handled manually by humans. This would effectively allow for a greater quality of scrutiny by cyber security professionals on more complex and pressing issues, drastically increasing a system’s overall security. AI is also useful in the analysis and detection of disguised threats. As threats become increasingly sophisticated, their methods of disguise or obfuscation are also becoming increasingly challenging for computer systems administrators to identify. By posing in a way that imitates ordinary behaviour and avoiding a response from a bad-­‐behaviour detection model, computer intrusions are able to move through the system undetected. However, “artificial intelligence can recognize significant changes in user behaviour that suggest a security risk” (Logrhythm, 2018). The way that AI systems do this is by using a good-­‐behaviour model, as explored above, to detect actions and access that are not deemed as “good”. This is done by analysing vast amounts of data on what individuals who have authorised access to the system should be able to do and what actions they regularly perform to gain an understanding of what is “good” behaviour in the system (AIAA, 2018). The data is then used to check actions that are performed within the system, searching for any actions that do not meet the “good” behaviour criteria. By using the good-­‐behaviour method, threat and intrusion detection systems are able to more accurately determine which actions should be further verified or followed up by human actors, reducing the time taken to respond to potential threats and the damage which can be achieved should a genuine threat occur. With the rapid developments in the field of AI, there is a growing trend towards the use of AI in a defensive manner. As the field further develops, the methods of defence and the attacks that must be defended against will also develop, resulting in an evolving defensive landscape. By embracing the use of AI, defensive systems can be significantly improved and be better prepared for future attacks that may involve the use of AI. This will contribute to levelling the playing field between offensive and defensive systems and reducing the overall effectiveness of malicious actors who would otherwise gain from leveraging this technology. Artificial Intelligence and Cyber Threat Detection Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory have demonstrated an AI-­‐powered platform, named AI², which predicts cyber-­‐attacks. In benchmark testing, run over a data-­‐base consisting of 3.6 billion data entries of log-­‐in activity generated by millions of users over a three-­‐month period, the platform successfully predicted 85% of attacks (Connor-­‐Simons, 2016). To predict attacks, AI² combs through data and detects suspicious activity by grouping the data into meaningful patterns using unsupervised machine-­‐learning. The platform then presents this activity to human analysts who confirm which events are actual attacks, and incorporates that feedback into its models for the next set of data. The platform continuously generates new models, improving its d etection rates significantly and rapidly (Connor-­‐Simons, 2016). 26 Electronic copy available at: https://ssrn.com/abstract=3407779
AI is also being designed to respond to the burgeoning threats to and created by ‘ Internet of Things’. Botnets providing disruption services, hackers breaking into (and taking control) of infrastructure and medical equipment, and smart gadgets spying on individuals. Combining smart cameras, motion sensors, and utilising a predictive AI processing hub, the Deep Sentinel Smart Home Surveillance System creates an intelligent surveillance zone that monitors devices for unusual behaviour (Deep Sentinel, n.d.). Cloudfare’s Orbit Security Solution provides a virtual network for IoT devices and monitors the traffic coming from and going to each device (Newman, 2017). If suspicious activity is detected, the platform will temporarily disconnect the device from the network. However, every approach to security has its trade-­‐offs: in the case of Cloudfare’s solution, the trade-­‐ off is a lack of centralized oversight of IoT because any smart-­‐home device could be compromised and data sent to a third party. Platforms such as Cloudfare can ensure effective baseline security for monitoring devices in order to protect against less sophisticated attacks. Intelligent cloud services, such as Microsoft’s upcoming AI powered Azure IoT Edge, will be instrumental in not only better securing IoT devices but also optimising their performance (Brunkard, 2018). Microsoft claims the AI will be able to effectively analyse vast amounts of historical data in order to identify patterns through the use of predictive analytics. Any event that falls outside the normal operating patterns of a device will be quickly identified, analysed and responded to in machine speed. With the coming advent of 5G networks and the ever growing myriad of new threat vectors to the cybersecurity environment by the IoT, the need for sophisticated defensive AI is now more pertinent than ever (Shoebridge, 2018). The sheer amount of data produced by billions of users (and even more machines!) requires real-­‐time analysis in order to protect both people and devices, which can only be accomplished through the implementation of defensive AI. 27 Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 4 -­‐ Offensive applications of Artificial Intelligence: The investment and growth in defensive AI systems has sparked the demand for more sophisticated offensive methods by Black, Grey and White hat actors. This demand has resulted in the development of sophisticated offensive AI technologies (Filar & Anderson, 2018). Offensive AI is the specialised counterpart to defensive AI. Unlike reactive defensive systems, offensive AI actively and aggressively seeks out system vulnerabilities and attack vectors. Furthermore, offensive AI is proactive in preparing and coordinating attacks against identified targets through the vulnerabilities it has identified (Rouse, 2012). This section will explore in greater detail both the proactive and adversarial components of offensive AI. Defensive AI an increasingly prominent feature in network security. This is due to the dynamic way the AI learns and therefore continuously developing and improving its malware detection abilities. Despite the trend towards the use of AI in the defensive sphere, the majority of currently known offensive AI use has been by researchers rather than industry or state actors (Brundage et al, 2018). While there is little visibility about the development of offensive AI, some researchers believe that an ‘arms race’ will occur in the near future between those employing offensive systems and those employing defensive systems ("Five key trends to watch in 2018 as cybercriminals continue to innovate", 2017; Filar & Anderson, 2018). With the potential for an AI arms race, and an increased demand and funding for sophisticated cyber defence platforms, defensive AI developers currently have the advantage (Ashford, 2018). With the high demand for professionals who have an AI background, there are few who would be unable to find a professional use for these skills, resulting in a lack of criminally motivated AI specialists -­‐ at least until such capacity becomes more widely available the number of actors developing offensive measures is presumed limited (Ashford, 2018). This can be largely explained by Rational Choice Theory (RCT). RCT is based on the premise that humans are rational actors who will make decisions that benefit them based on a rational analysis of costs and benefits (Boudon, 2003). This rationality explains why those with the skills required would select legitimate and rewarding work that has a very low risk compared to illegitimate work, which may provide high reward but at a much higher risk. As the development of offensive capabilities for industry and state actors continues apace, it is probable that more criminal actors will emerge. The offensive us of AI is explored further throughout this section. Adversarial AI Adversarial AI is a subset of offensive AI that is designed to imitate human learning, reasoning and problem solving in order to be an effective malicious actor (Dixon, 2018). According to Dixon (2018), adversarial AI has significant capacity to alter the cyber threat landscape through three major changes commonly predicted (Dixon, 2018). These changes are outlined below. The first change outlined by Dixon (2018) is the exponentially increasing variety of attack vectors available to malicious actors against a plethora of viable targets. Historically, cyber offending has been limited in scale due to the necessity of human interaction in their development and deployment. The limited scalability of attacks has increased the investment by malicious actors to focus on specialised attacks against specific systems rather than all potential systems.12 With the implementation of adversarial AI these limitations would be greatly reduced. Therefore, potential actors would be able 12
An example of this in practice is the Wannacry ransomware attack in 2017. Despite its notoriety Wannacry only affected an estimated 200 000 computers in 150 countries through an unpatched vulnerability known as EternalBlue (Broadhurst et al, 2018). 28 Electronic copy available at: https://ssrn.com/abstract=3407779
to increase the range of their attacks and, in combination with the following two factors, to increase the effectiveness of these attacks. The second theorised change is to the adaptability of attacks. Instead of the static attack that is limited by the instructions it possesses when released, future attacks driven by an AI would have the capacity to evolve and adapt, drastically increasing the overall effectiveness of the attacks (Dixon, 2018). These systems may have the ability to adapt in the field to improve their attack method and vector. This capacity increases further challenges for defensive systems, which would require the capacity to defend against evolving attacks in real time. This capacity to learn may also provide attacks with the ability to develop defences against prevention measures. The third change is a greater variety of new attack modus operandi. With less human interaction required to develop and execute attacks that are powered by AI, there is an increased opportunity for attacks to develop in new ways that have not previously been seen (Dixon, 2018). This poses a challenge for defensive AI systems as a greater variety of attack vectors and methods require greater capacity and more sophisticated responses from defensive systems. Defensive systems would therefore need to continuously improve and evolve, as adversarial AI becomes a feature of future cyber offences. While adversarial measures largely have a focus on attack-­‐based actions, there is also a positive benefit to the evolution of defensive capability. Attacks on an actor’s own system presents a unique opportunity to better understand the strengths and weaknesses of a given system, effectively a form of automated penetration testing (Filar & Anderson, 2018). The main benefit of AI conducting automated penetration testing is that a broader and more comprehensive survey of system vulnerabilities can be undertaken. By using an attack system that is able to evolve and adapt to the security measures it encounters, a wider variety of genuine attacks can be tested and prepared for. This can include attacks made manually by human actors, attacks made by malware, which does not make use of AI, and attacks that are powered by AI. These proactive measures are not guaranteed to develop a fully secure system but would allow for heightened security given the appropriate AI algorithms. Malware risks One of the potential uses of offensive AI are targeted malware attacks. Largely due to the technology that has been available, targeted attacks have focused on larger groups such as “businesses, government agencies, or political groups” (“Targeted Attacks”, n.d.). With the development of AI driven malware, it will become possible for offensive actors to more effectively target multiple individuals or organisations with the same form of malware, as shown in malware such as Stuxnet. With the evolving changes to the threat landscape that will occur with the advent of adversarial AI, the potential for adaptive, specialised and ultimately more effective malware will also become a reality. The development of malware that has the capacity to evolve in the field and to have greater target specificity will pose new challenges for security and open up new opportunities to malicious actors. A popular example of a targeted malware is the Stuxnet cyber weapon. The cyber weapon was deployed as Operation Olympic Games against the Iranian Natanz Nuclear enrichment facility, believed to be supplying weapons grade plutonium to the Iranian government (Zetter, 2014). The weapon was designed to covertly cripple the facilities’ Siemens Programmable Logic Controller centrifuges (Fruhlinger, 2018; Zetter, 2014). Sophisticated cyber-­‐attacks such as Operation Olympic Games would be greatly improved through the addition of AI. Recently IBM’s DeepLocker malware, an adaptation of the WannaCry cryptoworm, was demonstrated to use “deep neural networks to hide its malicious payload […] and only activate it when it detected its target” (Brown, 2018). This proof of concept malware demonstrates the effectiveness of incorporating AI into malware to further 29 Electronic copy available at: https://ssrn.com/abstract=3407779
streamline attacks and allow for both concealment and attack precision. DeepLocker performed this targeted attack by being embedded into a video conferencing application. As this application would be granted access to the device’s camera feed, the malware is able to use a neural network trained for facial recognition in order to scan the camera’s view for the intended target. The malware is then designed to encrypt all files on the device of the victim in the same manner as the WannaCry ransomware (Brown, 2018). The development of DeepLocker shows the impact that AI is capable of having upon the development of malware. As an early incorporation of AI into malware, it shows that there will be significant changes to the threat landscape as this technology further develops. At present, this form of AI technology is usually only accessible to large corporations such as IBM. However, researchers predict that the technology will develop to the point where it is more streamlined and accessible to a wider variety of actors in the future (Filar & Anderson, 2018). Such accessibility would further open the use of the technology for criminal use with algorithms and platforms becoming more mainstream and available to a wider variety of actors – including criminals. Biometric Recognition Biometric recognition is becoming an increasingly popular security measure due to the highly individual nature of biometric data (Thakkar, n.d.). AI is used in biometric recognition in order to determine if a sample matches against the stored data. AI is particularly important as biometric data often contains small discrepancies between each sample. For example, voice recognition systems make use of certain biometric markers that data can be used to identify an individual. This is done by “measuring the characteristics of a person’s speech as air is expelled through their lungs, across their larynx and out through their nose and mouth” (IFSEC Global, 2016). However, this may not be exactly identical each day. Therefore, AI is required to determine which characteristics of the biometric data must remain the same at each instance, and those that can change. This is trained into the algorithm to allow for the minute detection that is required in order to correctly identify individuals. There are two ways in which biometric recognition is a component of offensive AI: for enhanced security and as a component of an attack. Biometric recognition is useful for security as a proactive measure often with the purpose of granting or denying access based on an individual’s existence in a database. This can be considered an offensive system, particularly when coupled with predictive policing measures, explored below, and similar strategies. One such measure is the use of keystroke dynamics. Keystroke dynamics are considered to be a behavioural measure and can be implemented on devices as a form of two-­‐factor authentication (Violino, 2018). A ptypical use of this is in combination with a password. By assessing a user’s keystroke dynamics while a password is entered, a system can determine if the user is actually the one that the password corresponds to (Violino, 2018). This can be considered to be an offensive use of a defensive measure, particularly where keystroke dynamics are in constant use throughout a user’s session to ensure proactively that the authenticated user is the same throughout the session. As shown by the development of DeepLocker, biometric recognition also poses an offensive risk. The incorporation of biometric AI into attacks such as malware produces the capacity to select individual targets based on known biometric data. This allows for highly personalised attacks that can be tailored down to the individual level. This personalisation can be designed to target a specific individual, or a variety of individuals, based on the data that is provided to the algorithm. Another aspect of this is that these attacks are easily concealed within seemingly safe files and programs, such as the one used for DeepLocker. This enables the attack component to be well concealed, presenting only when the target is identified. Biometric based attacks do have limits, as the offensive actor must know sufficient detail about the target’s biometric data in order to provide the algorithm with sufficient data to 30 Electronic copy available at: https://ssrn.com/abstract=3407779
identify the target. This will then require actors to have detailed knowledge about their target or to broaden their attack scope to demographics or general features in order to reach the target. Predictive Systems Predictive systems are an increasingly popular use of offensive AI that has been implemented in a number of markets and countries. These systems comprise of AI that possesses the data to make predictions about the likelihood of future events. Two of the most prominent predictive systems are predictive policing and the Chinese Sesame Credit system. While these systems are potentially highly beneficial to society, they do possess many high-­‐level risks, which present challenges to the effectiveness of such measures. Such systems, due to their basis in AI algorithms, are only able to be as effective as the data that they are presented with. As explored in Chapter 2, the limitations of AI are based largely on the data that the algorithm is initially trained with, the way that the algorithm itself is originally constructed, and the weightings that are applied to each categorisation. For this reason, there is significant error risk in existing predictive systems. Chinese Social Credit In 2014, China released a social credit system for voluntary citizen participation from (Rollet, 2018). This system will be mandatory for all citizens by 2020 and aims to reshape Chinese society. The system is based around developing a trust system for Chinese society in order to reward those who are deemed trustworthy according to the relevant societal values and to punish and re-­‐educate those who are deemed to be untrustworthy. The Sesame Credit system, co-­‐developed by online shopping platform Alibaba, has a credit system which “constantly scores people from 350 to 950” (Harris, 2018). The system has been designed to monitor both private citizens through means such as monitoring “transgressions such as dodging transport fares and not caring sufficiently for your parents” (Harris, 2018) and also businesses and officials in order to reduce corruption. The rewards for being a trustworthy citizen are many and varied, including being able to rent items and stay at hotels without putting down a deposit (Rollet, 2018). Baihe, the largest matchmaking service in China, has also collaborated with Sesame Credit and will “promote clients with good credit scores, giving them prominent spots on the company’s website” (Hatton, 2015). (continued overleaf) 31 Electronic copy available at: https://ssrn.com/abstract=3407779
Figure 4: The Sesame Credit Score displayed in the Alipay Wallet App (Xiang, 2015). The social credit system has been criticised by many who feel that such a system is “nightmarish” and “Orwellian” in nature, training or socializing a society in which individuals are constantly watched and every interaction can reap real-­‐world consequences (Hatton, 2015). This poses particular concern to individuals facing debt as “frequent debtors could eventually be barred from attempting to by breakfast, take a bus, and look for jobs” (Rollet, 2018). While there are many who are opposed to the idea of social credit, there are also many who feel it is a n ecessary change in order to fix societal issues. Venezuela, a country currently in crisis, has adopted China’s social credit system in order to make its population more supportive of the current ruling government (Berwick, 2018). It is very likely that repressive regimes will adopt this technology to ensure a population’s subservience. 32 Electronic copy available at: https://ssrn.com/abstract=3407779
Figure 5: Carnet de la Patria/ Fatherland card. A joint venture of China’s ZTE and the Venezuelan government to create a social credit system based on a citizen’s loyalty. Source: Vidal, 2018. The technology is also useful in banking as “the central bank has the data from 800 million people, b ut only 320 million have a traditional credit history” (Hatton, 2015). However, this massive gap in credit history can instead be replaced by a social credit score. The use of this new system will have the potential to significantly alter the way in which Chinese society operates. In turn it may stimulate measures to avoid, game, forge or fake social credit scores Predictive policing has become increasingly popular in recent years with the trend towards a preventative rather than reactionary strategy. Preventative policing has been explored in greater detail in Chapter 2: Algorithm & Threat Assessment Tools Critique. Preventative policing is considered an offensive use of AI due to the focus upon proactive strategies. The Chinese Sesame Credit system is an example of an alternative use of a predictive system. Based on strict models of what is considered to be a “good” citizen, the actions of those within the system are ranked. This provides an indication of how “good” a citizen any individual is, a measurement that could be used to determine how individuals fit or may be aggregated into a predictive policing model. Due to the proactive nature and potential incorporation with predictive policing concepts, this system can be considered to be an example of a current dual offensive and defensive application of artificial intelligence. Conclusion Offensive AI has many applications for both legitimate and criminal uses. As the field continues to develop, more benefits and risks will become apparent. AI fuelled malware is a threat that will become more prominent and poses a significant risk to many potential targets. Current developments such as Stuxnet show the challenges that can be created even with limited usage of AI. Further, DeepLocker illustrated the risks should these systems be able to interface with the rich data that is available 33 Electronic copy available at: https://ssrn.com/abstract=3407779
through Internet of Things devices. Biometric systems are becoming increasingly popular, introducing additional risks as technology further incorporates these measures and as they become a component of attacks. Predictive systems, while capable of being beneficial with the correct data, present a risk should they be provided with poor data or attacked in a manner that compromises their decision-­‐ making. For these reasons, offensive AI poses not only significant opportunity for legitimate uses that can alter the threat landscape, but also the capacity to enable increasingly complex and effective acts by illegitimate actors. 34 Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 5: Regulating Artificial Intelligence in Law and Policy AI has become a powerful driving force of industry and government alike. It is increasingly recognised as a prominent automation and decision-­‐making tool that is reshaping the world. As technology continues to develop, the use of AI is likely to become a more ubiquitous part of society and everyday life (West, 2018). However, the more prevalent use of AI also increases the risk and fear of it being misused to cause harm to people and society. This is because AI can potentially be used to facilitate criminal acts, or commit offences as a result of its independent, autonomous actions (Čerka et al., 2015; King et al., 2019). To effectively address these concerns would require the regulation of AI, and in a way that does not hinder the development of such technology (Guihot et al., 2017). This chapter will outline the challenges of regulating AI and examine possible solutions to strengthening its regulatory framework. It will mainly focus on the solution of applying and adapting existing liability laws to the technology, in order to address situations where an AI has been used to facilitate a crime, or where a mistake has resulted in a crime being committed. The Challenges of Regulating Artificial Intelligence Rapid advances in disruptive technology such as AI have prompted concerns about the potential for their misuse and calls for the government to regulate it. There are currently few laws and regulations addressing the specific challenges of AI outside of the regulation of autonomous vehicles and drones in several states (Guihot et al., 2017). As argued by Matthew Scherer, the lack of development in AI regulation can be attributed to the fact that traditional methods of regulation, such as ethical oversight of research and development, would seem ineffective to manage the potential risks of AI (Scherer, 2016). This is because AI research and development can require little physical infrastructure (‘discreet’), have different components of the AI system designed without conscious coordination (‘discrete’), engage contributors from various locations around the world (‘diffuse’) and be subject to secrecy in order to protect intellectual property interests (‘opaque’), thus making its regulation difficult and problematic (Scherer, 2016, p. 369). The delays in AI regulation can also lead to uncertainty among developers and investors resulting in the withdrawal of resources and interest in AI projects. Such ambiguity about regulatory costs could hinder potential innovation, as developers seek to avoid the risk of costly regulatory impositions or condition, including prohibition by regulators in the future (Guihot et al., 2017). There are also specific challenges to regulating AI. Like many new technologies, the development of AI outpaces attempts to regulate it. While ‘future-­‐proofing’ legislation could address this issue, it can result in the laws being ‘…too general or vague to effectively serve their intended purpose or to provide meaningful guidance regarding any specific technology’ (Guihot et al., 2017, p. 421). Furthermore, AI research and development are largely conducted by private companies with legitimate interests in keeping their information confidential. This secrecy in turn creates information asymmetry and makes it difficult for regulators to fully understand the subject of regulation. The social and ethical implications of systems using AI are also unclear, especially with the development of sex robots and killer drones. In addition, AI development is an inter-­‐disciplinary field and would require a coordinated approach across different regulatory bodies (e.g. melding machine safety, and privacy regulations). Lastly, regulatory failure or ‘capture’ can occur in circumstances where the regulators become overly sympathetic to the industry they are regulating. Due to the information asymmetry noted, employees of AI developers are particularly valuable informant for regulatory bodies due their knowledge and expertise, which can therefore result in ‘agency capture’ (Guihot et al., 2017). Such challenges can result in delays regulating AI especially when novel technologies merge creating new unregulated products. For example, child sex dolls with augmented speech recognition and other AI driven robotic functions could breach laws against child sex exploitation materials in many 35 Electronic copy available at: https://ssrn.com/abstract=3407779
jurisdictions but for clarity amendments to relevant laws in several countries including Australia (Brown & Shelling, 2019). Artificial Intelligence and Crime – Redefining Responsibility In an age where AI are capable of performing complex tasks that were once only capable of being performed by human beings, there is concern that AI can be used outside of their intended purpose – for example, to commit a crime. A way to address the regulatory challenges of AI and crime is to apply and adapt existing liability schemes to the technology (Holder et al., 2016). By legal norms, a person who commits an offence, or is responsible for the actions of someone who commits an offence, must compensate for their actions (Čerka et al., 2015). However, the entity that is liable can be less clear in a situation where AI is involved in the commission of a crime. Unlike human beings and corporations, AI does not have a formal legal status under national or international law. As such, liability is likely to be transferred to the user of the AI or its creator (Hallevy, 2010). What complicates this issue of liability is the future possibility of fully autonomous AI committing an offence, in an act that was unforeseeable by its creator (Scherer, 2016). Such as development invites the argument that AI could be given a legal status and therefore be held liable for its actions (Čerka et al., 2017). The situations in which criminal liability of AI entities might arise can be examined through the three legal models proposed by Gabriel Hallevy. The first is the ‘Perpetration-­‐via-­‐Another’ liability model, which considers AI to be an innocent agent and attributes liability to the entity that instructs it to commit an offence. The second is the ‘Natural-­‐Probable-­‐Consequence’ liability model, which holds users and the developers of the AI liable if they knew that the commission of a criminal offence was a natural, probable consequence of their use or programming. The third is the ‘Direct Liability’ model, which directly attributes liability to the AI system, as long as it satisfies the physical and mental elements of the offence (Hallevy, 2010). In this way, users can be liable for using AI as a tool to commit a crime, developers and programmers can be liable for being an accomplice to the crime, and the AI system itself could arguably be liable if it commits a crime autonomously (Kingston, 2016). The Elements of Criminal Liability For an AI entity to be criminally liable, there are two elements of the offence that must be proven: a physical element, and a mental element (Ashworth & Horder, 2013, p. 83). The physical element, also known as the actus reus (‘guilty act’), requires an individual to have voluntarily performed an act, or omitted to perform an act, which is criminal in the eyes of the law (Sullivan, 2009). The mental element, or the mens rea (‘guilty mind’), requires an individual to have had criminal intent and knowledge that their act was wrongful at the time they committed the offence (Colman, 2015). As such, the actus reus is fault-­‐based while the mens rea refers to the mental state of the perpetrator, and both elements must be satisfied in order for a criminal liability to occur. An exception to this would be a ‘strict liability’ offence, (e.g. drink driving offences) which is an offence that does not require proof of intent or knowledge of wrongfulness (Australian Law Reform Commission, 2014, pp. 124-­‐ 125). Making AI crimes strict liability offences can be problematic in situations where an AI system has autonomously committed a crime, and its otherwise faultless owner would consequently be liable (King et al., 2019). Deliberate Use of AI as a Tool for Crime One of the ways that AI can be involved in a crime is if it is used as a tool to facilitate the crime, as outlined in Hallevy’s ‘Perpetration-­‐via-­‐Another’ liability model. This can occur when an offence is committed by an agent who is innocent because they are mentally incompetent or lack a criminal state of mind. In that situation, the agent is regarded as a ‘mere instrument’ whereas the real 36 Electronic copy available at: https://ssrn.com/abstract=3407779
perpetrator is the individual who orchestrated the offence (Hallevy, 2010, p. 179). For example, if a person kills someone with a gun, that person would be criminally liable, and not the gun. As currently most AI programs today are still considered ‘weak AI’ (as discussed in Chapter 1), it is highly unlikely that such AI are mentally capable of satisfying the mens rea of the offence, or the requirement of volition for the actus reus. Criminal liability would therefore be attributed to the real perpetrator as a principal in the first degree for the conduct of the AI, since the AI would be considered an ‘innocent agent’ (Hallevy, 2010, p. 179). This complies with the general rule that ‘the principal of a tool is responsible for the results obtained by the use of that tool’ (Čerka et al., 2015, p. 384). As such, the user or programmer of an AI entity could be criminally liable if they use or program an AI entity to commit an offence. Crime as a Consequence of Coding In situations where the AI independently commits an offence, and its user or programmer had no knowledge or intent to use the AI entity to commit that offence, the ‘Perpetration-­‐via-­‐Another’ liability model would not be appropriate. On the other hand, the ‘Natural-­‐Probable-­‐Consequence’ liability model could impose liability upon users or programmers as accomplices to the offence. This model attributes liability upon negligence, through the question of whether the user or programmer ‘should have foreseen the offence, and prevented it from being committed by the AI entity’ (Hallevy, 2010, p. 184). If the user or programmer was negligent in using or programming the AI entity, which resulted in the commission of an offence despite not having criminal intent, it would therefore be a case of negligence. However, the question of liability becomes more complicated in the situation where a programmer knowingly and wilfully programs an AI entity to commit an offence, but the AI entity commits an additional, unplanned offence. The example Hallevy gives is of an AI entity that has been programmed to commit a bank robbery, which then deviates from the plan by killing someone. In that case, the programmer would be criminally liable for the violent robbery, as they had used the AI entity to commit the crime. The manufacturer or programmer would also be liable for the killing, even in the absence of criminal intent, because the killing would be a natural, probable consequence of the violent robbery (Hallevy, 2010). Self-­‐Driving Car Kills Pedestrian as a Result of Coding In March 2018, an Uber self-­‐driving car in Arizona car hit and killed 49-­‐year-­‐old Elaine Herzberg while she was pushing a bicycle across the road. It was later found that the AI driven vehicle had chosen to ignore her movement across its path, as it mistakenly recognised her as a ‘false positive’ that did not need to be avoided. It was also reported that the human ‘safety driver’ of the car had not been paying adequate attention to the road to intervene (Gibbs, 2018). This case demonstrates a situation where a programmer’s coding has inadvertently led to the death of an innocent party, and prompted debate about where culpability should lie. On one hand, Uber was the creator of the AI system, which had made a mistake in hitting the pedestrian. On the other hand, the role of the safety driver as the operator of the vehicle was to watch the road and intervene in situations where the AI might malfunction (Burkitt, 2018). In March 2019, the Yavapai County Attorney’s Office determined no basis for criminal liability for Uber (Zaveri, 2019). However, the prosecutors also added that there should be an investigation into what the safety driver “would or should have seen that night given the vehicle’s speed, lighting conditions, and other relevant factors” (Zaveri, 2019). 37 Electronic copy available at: https://ssrn.com/abstract=3407779
By applying the ‘Natural-­‐Probable-­‐Consequence’ liability model, it can be argued that the safety driver would be liable as she had been negligent in using the AI entity. In this case, it is very likely that hitting the pedestrian was foreseeable and preventable by the safety drivers, whose purpose was to intervene and take control in situations where the AI might make a mistake. Command Responsibility Another way to attribute liability for a crime caused by an AI entity is through the model of command responsibility. Under Article 28 of the Rome Statute of the International Criminal Court, a military commander is criminally responsible if they knew about, or should have known owing to the circumstances at the time, and failed to take reasonable steps to prevent crimes committed by their forces. Although the term forces seem to refer to people, they can arguably include AI entities such as robots that purpose is to replace or augment the role of soldiers (McAllister, 2011). The application of the command responsibility model requires the military commander to satisfy the mens rea of having knowledge that the offence would occur (King et al., 2019). Military commanders are required to understand the means and methods of warfare they deploy (Sassóli, 2014), however, AI systems may not always respond as they have been programmed and it might not be possible to predict when a deviation might occur. Nonetheless, like the ‘Perpetration-­‐via-­‐Another’ and ‘Natural-­‐Probable-­‐ Consequence’ liability models, the command responsibility model also ultimately attributes criminal responsibility to human actors. The Use of Autonomous Weapons Systems in International Humanitarian Law Autonomous weapons systems, including killer drones and robots, are increasingly being used by the military as a means of warfare in all kinds of armed conflicts. While fully autonomous weapons systems that use AI have not yet been developed, there has been debate on which entity is to be held accountable if these weapons commit crimes (Sassóli, 2014). According to Sassóli, a military commander’s responsibility for deploying autonomous weapons is not one of command responsibility, but a case of direct responsibility. He argues that it is fair to hold a military commander deploying autonomous weapons responsible, because the commander ‘must understand how they function, just as for any other means and method of warfare’ (Sassóli, 2014, p. 324). Furthermore, new weapons must firstly be assessed and approved according to Protocol I, Article 36 of the Geneva Conventions. Military commanders and operators would also be given clear instructions on how, when and under what circumstances the device can be used (Sassóli, 2014). As such, it has been argued that humans will always be liable for crimes committed by autonomous weapons systems because ‘the responsibility for the appropriate use of the systems will nevertheless remain with the human operators and commanders’ (Schmitt and Thurnher, 2013). On the other hand, if a military commander or human operator deploys the autonomous weapon, which they did not know was defective, and the device results in the loss of life, then the manufacturer and programmer of the device are likely to be held criminally liable under most domestic laws. In this way, it has been argued that International Humanitarian Law can adequately apply to autonomous weapons using AI during armed conflict. Furthermore, domestic criminal laws are likely to hold the manufacturer and programmer accountable if they had deliberately, recklessly or negligently created a device that causes the loss of life (Sassóli, 2014). 38 Electronic copy available at: https://ssrn.com/abstract=3407779
AI as a Perpetrator of Crime Although liability for damage caused by an AI entity is likely to be attributed to its user, manufacturer or programmer, it has been argued that an AI entity could be directly liable. The problem with such an argument is that AI is not recognised as a legal subject under national or international law (Čerka et al., 2015). However, an AI entity could arguably be accorded some formal legal status akin to that applied to corporations, if they are ‘truly autonomous machines’, which would make them a legal entity under the law and thus subject to being sued or prosecuted (Vladeck, 2014, p. 124). If an AI entity obtains this legal status in the future and commits a crime, then it is assumed that both the actus reus and mens rea of the offence must be satisfied. Beyond meeting the evidentiary requirements for establishing an offence was committed, applying the physical element of criminal liability to an AI entity is relatively uncomplicated. Provided it can be shown that the AI entity committed an illegal act or omission, then the requirement would be satisfied (Kingston, 2016). However, it is extremely complicated to prove that AI has the mental capacity to demonstrate criminal intent, or to comprehend the wrongfulness of its action. At this stage of development, AI systems are not capable of experiencing emotions or indeed capable of acting as a completely free agent. The mens rea could arguably be imputed to an AI entity, in the same way it is argued that corporations could have imputed the mental state of its employees but this could only be the case if the AI is recognised as a legal entity (King et al., 2019). Exceptions to the requirement of the mens rea are ‘strict liability’ offences, which only require the physical element of the offence to be proven (Australian Law Reform Commission, 2014, pp. 124-­‐125). The difficulty with using a strict liability model is that it is difficult to pinpoint who exactly should be held responsible, as there can be a multitude of actors who worked on the development of the AI entity (King et al., 2019). Furthermore, holding the manufacturer or programmer of the AI liable is problematic as ‘certain technologies used in the development of AI may date back to years before such AI is developed’ (Gurkaynak et al., 2016, p. 754). Developers of such technology may not have anticipated that their technology would be incorporated into an AI system, which would result in the commission of a crime. In this way, making AI-­‐related crime a strict liability offence can be unfair to manufacturers and programmers due to the diversity of individuals and firms that are involved in creating an AI, and the challenge of identifying who exactly is liable (Gurkaynak et al., 2016). While it has been argued that AI could have directly liability, imposing liability on AI would not be effective in deterring crime because AI cannot be punished in the same way as human beings, thus potentially resulting in the ‘de-­‐responsibilisation’ of the human actors behind it (Kingston, 2016). While individuals might be deterred from committing a crime due to the possibility of imprisonment and fines, AI cannot to be deterred. However, it would be feasible to alter the AI system by modifying the machine learning process to account for potential criminal consequences of an errant AI function. AI could not understand that it is being punished, and would not be capable of changing accordingly. Thus, it would not be effective to hold AI liable, and legislators should instead focus on the liability of individuals using AI to commit crimes, and the liability of its programmers. Balancing the Interests of Regulation and Innovation A challenge for regulators is to balance the need for regulation with the interest of encouraging the continued development of AI technology that can result in useful and profitable innovation. For example, holding programmers as negligently culpable raises an important policy consideration: would this be setting legal standards too high? After all, bugs and glitches in software are arguably an inherent part of the creation process. If programmers became easy targets for criminal negligence actions, they may be deterred from creating innovative and unexplored areas of AI at the risk of being 39 Electronic copy available at: https://ssrn.com/abstract=3407779
held criminally liable for an unintended action of their creation (Čerka et al., 2017). As such, legislators must consider how to best balance accountability on one side, and innovation on the other. Firstly, a clear legal definition for liability would be useful. The possible approach would be that only actions for gross negligence committed by the manufacturer or programmer would be pursued. In other words, situations where the ‘reasonable person’ with knowledge of coding would have easily noticed a flaw in the AI’s programming and identify that it would almost certainly lead to an unintended consequence. As such, the issues of foreseeability and causation must also be addressed (Scherer, 2016). This would protect programmers who make a small and unintentional programming error that could not have been reasonably foreseen to lead to a criminal action by the AI, while also holding them accountable for grossly negligent and easily avoidable mistakes. A clear definition of liability would also indicate which projects AI developers should or should not work on, which increases certainty and encourages innovation (Guihot et al., 2017). Furthermore, the research and development of AI could be made more transparent through legislation requiring the publication of code. Testing could also be strengthened so that the AI programmes conform to safety standards. As argued by Scherer, discreteness and opacity are two of the main challenges of regulating AI, as corporations are likely to wish to keep their research and development a secret from their competitors. However, these challenges can be addressed by legislation requiring AI developers to share their code, to ensure that it is safe. To prevent the hindering of innovation, incentives such as limited liability and tax incentives could be given to cooperating entities. In addition, the challenge of AI research and development being discreet and diffuse does not necessarily reflect reality. While small private actors are capable of developing AI, it appears that AI research and development is largely led by commercial and government entities (Scherer, 2016). In this way, despite the challenges, it is possible to strengthen the regulatory framework of AI while taking into consideration the need to protect innovation. Regulating Artificial Intelligence in International Law One of the significant legal challenges which the international community will face as AI becomes more capable is consistency. Current laws regarding AI are largely inconsistent, and can vary substantially: some countries have laws, which are well-­‐equipped to address the challenges brought by AI, while others have no legislation at all. Such international discrepancy could raise jurisdictional issues in a number of circumstances. For example, how can the nationality of an AI be determined for legal purposes? As AIs develop more human-­‐like tendencies and capabilities, it may become much more important to be able to identify an AI according to its country of origin. In 2017, Sophia, an English-­‐speaking robot, was awarded Saudi Arabian Citizenship (Maza, 2017), highlighting the very possibility of computers and AI being granted legal titles usually exclusive to human beings. A possible solution to the question above would be that the AI possesses the nationality of the country in which it was created. However, with the modern-­‐day efficiency of the Internet, it would not be unusual for an AI to be created by multiple developers living in multiple countries, each with vastly differing laws on computer-­‐related crimes. In this circumstance, how would its nationality be determined? This is a question that, without a treaty to consolidate laws between countries, would be near impossible to answer. Thus, it is essential that international actors begin to consider the consolidation of laws regarding AI capabilities, particularly as we approach the potential advent of medium-­‐AI. Another important consideration for international law is how extradition of criminally involved AI would operate. For example, if an AI is physically present in Australia (although identifying the physical location of an AI presents its own challenges), but commits a fraudulent crime in the United States, 40 Electronic copy available at: https://ssrn.com/abstract=3407779
can it be extradited to the US for prosecution? A possible answer is that extradition should occur -­‐ as AI approaches more human-­‐like capabilities, they could arguably be treated in the same way. However, problems may arise with the way in which extradition takes place. For example, if an AI physically exists in one country, and another country requests extradition, would it be required that all of the AI’s physical components (which could be numerous) be transported? Furthermore, if the AI has multiple physical components in different countries, would extradition have to be approved by each of the countries housing the AI? Further complicating the matter would be the possibility of one country approving extradition, but another refusing it, leaving prosecuting country with only part of the AI. In such a circumstance, would they still be able to prosecute the AI, despite only having a portion of its physical existence? Each of these pose unique problems to legislators, which can only be solved by the consolidation, harmonization, and cooperation of laws, further highlighting the importance of international law being developed to cater for the challenges posed by AI. Cloud Data Stored in Multiple Countries and the CLOUD Act In a recent United States Supreme Court case, United States v. Microsoft Corporation, Microsoft was asked to produce information, including emails, about a customer thought to be involved in illegal drug trafficking (Justia, 2018). Microsoft refused to disclose the sought-­‐ after information as it were stored in Dublin, thus arguing that it was subject to Irish jurisdiction. In response, US legislators enacted the Clarifying Lawful Overseas Use of Data Act (known more widely as the ‘CLOUD Act’), which allows US-­‐based companies to be subpoenaed to provide data stored on their servers, regardless of whether those servers are held overseas (Mocheva, n .d.). Although this advancement is not directly related to the regulation of AI, it provides a potential platform for the development of laws regarding the extradition of AI. If the creator of an AI which has committed a crime could be subpoenaed to at least provide access to an AI’s physically based component, even if it were held overseas, it would overcome some of the issues which have been outlined. Nonetheless, for such legislation to operate properly, there would have to be significant cooperation in the international community, likely by way of treaties and ratification thereof. International law enforcement co-­‐operation and AI misuse Legal definitions of computer offences and the evidence that must be assessed to establish an offence differ across jurisdictions. International cooperation is often essential if criminal interference or misuse of an AI program (or indeed many cybercrimes) is to be effectively prosecuted. This is because the offender, victim and/or affected computer may be located in different jurisdictions. Cross-­‐national and international efforts to suppress cybercrime may be onerous because many states require legal equivalence before providing mutual legal assistance (MLA) to another state. The mitigation of cybercrime with an international or cross-­‐national dimension is partially addressed by the 2001 Council of Europe (CoE) Convention on Cybercrime (Budapest Convention). The Cybercrime Convention has three aims: first, to lay down common definitions of certain criminal offences – thus enabling relevant legislation to be harmonized at the national level; second, to define common investigative powers enabling criminal procedures to be brought into line between countries; and to foster international cooperation, thus enabling cooperating countries to implement joint investigations and prosecutions of cybercrimes. 41 Electronic copy available at: https://ssrn.com/abstract=3407779
The CoE Convention on Cybercrime outlawed computer trespass by requiring member states to criminalize acts that undermine the confidentiality, integrity and availability of computer data and so in principle covers the misuse of AI (by interference with its data and/or its programming) but not necessarily matters that arise due to the faulty design of an AI product or process. Article 1 requires states to criminalize: (1) illegal access of a computer system; (2) interception of non-­‐public transmissions of computer data to, from, or within a computer system; (3) interference with computer data; (4) interference with computer systems, such as computer sabotage; and (5) the misuse of computer-­‐related devices (e.g. ‘hacker tools’), including the production, sale, procurement for use, import or distribution of such devices. The Convention provides for the extradition of suspects, the disclosure and preservation of computer and traffic data, real time traffic data collection, trans-­‐border access to stored computer data and the interception of hacking tools. In addition illicit content, such as child pornography is criminalized (Broadhurst, 2006). Criminalizing illegal access, that is, ‘hacking’, cracking ‘or’ computer trespass address many of the risks associated with criminal misuse of AI. Computer intrusions may also give access to confidential data (including passwords, information about the targeted system) and might encourage hackers to commit more dangerous forms of computer-­‐related offences, such as the manipulation of an AI program. The criminalization of illegal interception protects the privacy rights of data communication and seeks to deter the recording of communications between persons (or indeed autonomous programs) and applies this principle to all forms of electronic data transfer. The provision on data interference aims at providing computer data and computer programs with protection similar to that enjoyed by corporeal objects against intentional infliction of damage. Conduct, such as damaging, deteriorating or deleting computer data, reduces the integrity or content of data and programs also captures malicious codes, and viruses (e.g. Trojans). The Convention importantly in respect to AI criminalizes acts of computer sabotage and covers the intentional hindering of the lawful use of computer systems, including telecommunications, by using or influencing computer data (system interference). The section covering misuse of devices establishes a separate criminal offence including conduct involving the production, distribution, sale, or access to devices, which were primarily designed or adapted for misuse. The procedural part of the Convention aims to enable the prosecution of computer crime by establishing common procedural rules and adapting traditional measures such as search and seizure and creating new measures, such as expedited preservation of data, to remain effective in the volatile technological environment. As data in the IT environment is dynamic, other evidence collection relevant to telecommunications (such as real-­‐time collection of traffic data and interception of content data) has also been adapted to permit the collection of electronic data in the process of communication. It enables data to be seized, or obliges those who possess the relevant data to disclose, or preserve data for investigation, but the Convention does not require or justify the surveillance of personal communications or contacts, by either service providers or police, unless there is an official criminal investigation (Broadhurst, 2006). The Budapest Convention has been adopted by 67 states around the world including all 47 states in Europe. However, China, Brazil, the Russian Federation and India among others have not signed the Convention limiting its scope in assisting with cybercrime offences that may involve offenders and victims from these jurisdictions. 42 Electronic copy available at: https://ssrn.com/abstract=3407779
Chapter 6: Conclusion In the preceding chapters we have broadly outlined the potential roles of AI programs to prevent crime or enhance the conduct of crime. Despite the hyperbole it is generally understood that AI applications currently in use can be classified as weak or narrow. An oft-­‐stated end goal of AI research and development is to produce strong AI that will revolutionise humanity through mass automation and knowledge dissemination. The limitations of the iterative or pattern recognition learning processes of machine learning (ML) technologies restricts the potential of AI applications to generalise and imitate human capacities such as intuition, self-­‐learning and empathic listening. Strong AI or Artificial General Intelligence (GAI) could nevertheless develop as self-­‐generating technologies with the potential for autopoietic machine based decision-­‐making. Thus in some futurist scenarios AI machines control or serve a dystopian or utopian world.1 Autopoietic or self-­‐replicating systems provide a framework for understanding complex, non-­‐linear or living systems that filter and process information from the wider environment. The nature of self-­‐
learning and replicating systems (for example in nature) is the basis for theorising about how AI could develop as a self-­‐actualizing or self-­‐regulating communication system. However, strong or GAI is still firmly in the realm of science fiction, but evolving forms of networked AI applications in tandem with complex human machine interaction are transforming human futures and societies. The speed and impact of these technological changes have invited speculation about how human societies may adapt, and we briefly touch on some ways to understand late modernity. Several controversial sociological theories of the emergence of human machine networks seek to describe and explain their significance and potential for transforming modern societies. One strand includes the materiel-­‐semiotic or Agent-­‐Network approaches (see Bruno Latour, 1987, 2005) that describe the agency of non-­‐human communications such as machines and other identities in interdependent communications networks in which speech is but one form. Another approach exemplified by the sociologist and system theorist Niklas Luhmann applied the notion of autopoiesis to social systems (i.e. legal systems) as meaning processing systems that distinguishes them from other self-­‐replicating systems such as biological systems. Assuming that ML technologies develop alongside the continued accumulation of mass or big data aggregation driven by smart machines then it is feasible that self-­‐preserving or autonomous versions of communication systems may evolve to mimic social systems. “A social system comes into being whenever an autopoietic connection of communications occurs and distinguishes itself against an environment by restricting the appropriate communications. Accordingly, social systems are not comprised of persons and actions but of communications" (Luhmann 1989, p. 145). Thus, social systems in Luhmann’s theory are comprised of networks of communication that instigate further communication but only communication that are inherently self-­‐referential (see Schatten 2008, p. 839). This allows social systems to distinguish between the system itself and its environment (see Schatten 2008, p. 839). Maleković and Schatten (2008), argue that analysing information systems as subsystems of social or organizational systems, enable autopoiesis to be defined in information systems “as communicative events that reproduce new communicative events based on previous (stored) communication”. The organization of this system is comprised of the relations between communicative events described through their semantics (meaning). The structure of the system is made up of the means that are used to produce communication described through syntax. However, applying notions of self or independent creation to AI systems underpinned by ML is both novel and controversial because communicative machines are added to the overall system of communications and a component of the social environment as much as people. 1
Examples of these in literature are Skynet from Terminator or Data from Star Trek. Electronic copy available at: https://ssrn.com/abstract=3407779
43 Bača et al. (2007, p. 148), describe potential for autopoiesis in information systems as “…the ability of an information system to continuously adapt to the needs of its current users and also to keep all the characteristics that make it unique and recognizable as an information system" especially if it "continuously adapt[s] to its users and the surroundings in which it operates, by collecting, storing, keeping, processing and disseminating information important for the organization and society, in order to make it accessible to everyone who wants to use it." Thus, every information system could be more or less autopoietic (Bača et al., 2007; Schatten & Bača 2010, p. 859; Varaždin & Maturana, 1975) and thus also a likely beneficiary of ML technologies and AI applications. The risk is that AI programs could lead to the reification of the data relied upon and the outcomes produced by AI but nevertheless replicate typical problems of context and confirmation bias. In short, self-­‐reproducing information and associated decision making systems also magnify ethical problems. Enthusiasm for the promise of AI has outstripped the well-­‐known challenge of implementing even weak AI. However, weak AI can inflict considerable cost and inconvenience when deployed inappropriately or incorrectly. For example, via a botnet that delivers malware or spam or when an automated prediction system misclassifies attributes of a decision-­‐making domain. Given developments now being made in the field of AI, there is a trend towards the use of AI as both an offensive and a defensive tool. As AI further develops, the methods of defence and the attacks that must be defended against will also develop, resulting in an evolving risk landscape. By embracing the use of AI, defensive systems can be significantly improved and be better prepared for future attacks that may involve the misuse of AI. AI fuelled malware is a threat that will become more prominent and poses a significant risk to a variety of potential targets. Current developments, such as the highly sophisticated Stuxnet, show the challenges that can be created even with weak AI. The example of DeepLocker also shows the risks should these systems be able to interface with the rich data that is available through devices connected to the Internet of Things. Aggregation of data, especially behavioural metrics will increasingly power AI applications. Biometric systems are becoming widely employed, introducing additional risks as the technology further incorporates these measures and as they also become a component of attacks. Predictive systems present a risk should they be provided with poor data or attacked in a manner that compromises AI decision-­‐making or indeed change its data sources and authenticity. For these reasons, offensive AI poses both a significant opportunity for legitimate uses and the capacity to enable increasingly complex and effective attacks by illegitimate actors (e.g. especially against banking/finance, transport and other industries which already use weak AI applications). Defensive AI is important not only for the protection of high-­‐risk infrastructure systems (e.g. energy, transport, information systems) but also for other commonplace applications of AI. As AI becomes more prominent and is more widely understood, there is a commensurate increase in attacks that aim to damage AI systems. Malicious actors are able to exploit the “learning” process of the artificial intelligence process and re-­‐train or teach the system, and induce “behavioural drift” (Bond, 2017). Introducing data that has been designed to ‘confuse’ the system into misclassification can achieve disruption and error in automation. A foremost challenge is how best to overcome the possibility that real world bias is able to transfer into a machine-­‐learned form of bias. In short the old adage ‘garbage in, garbage out’ applies to all data sets including those that are used to develop AI programs. It is necessary that a sufficient level of transparency is provided about AI algorithms, training sets and data sources that can be independently verified in order to reduce unintended bias and to ensure the efficacy of the 44 Electronic copy available at: https://ssrn.com/abstract=3407779
evaluations or decisions produced by AI’s. The approach of lawmakers has generally focused on a crime-­‐based or a means-­‐based approach. The crime-­‐based approach, which focuses on specific crimes such as fraud, child exploitation materials, or computer trespass, is limited to the specific evidentiary requirements of the offence. The means-­‐ based approach focuses more on the misuse or modification of computer data as an offence on its own, or an aspect of the commission of another offence. This approach is more readily able to adapt to changing technologies and new uses for technology, because it has more general terms that allows it to be applied to a broader range of actions. Legislation, which takes a means-­‐based approach, is better able to cover potential crimes targeting AI in the varying capacities and uses that AI applications, may have in the future. Many jurisdictions have adopted the means based approach by criminalising any trespass or interference with a computer, its programs and data (see Appendix for examples). The theft or alteration of data and/or programs upon which an AI program relies upon is thus already an offence in many jurisdictions. Hallevy’s (2010) account of AI’s potential liability for criminal actions, addresses issues of indirect or direct liability associated with the autonomous actions of AI. Despite significant advances, No AI application has developed a ‘consciousness’ or independent motivation remotely approaching Artificial General Intelligence (AGI), and thus its creators may still be liable for criminal misuse. AI programs have not advanced enough to warrant moral considerations as a unique identity. However, the unintended consequences of AI applications as they evolve as a general purpose technology will set new ethical challenges as well as providing new opportunities for criminal activity and crime prevention. Nevertheless, the integration of AI programs into many manifestations of everyday life from monitoring consumers to enhanced surveillance will continue to be propelled by investments by both commercial enterprises and the state. These developments will also reshape the opportunities for criminal activity. This review briefly highlighted some of the challenges and benefits that AI programs may bring to the crime game but also the need for further research about its impact on the nature of cybercrime. 45 Electronic copy available at: https://ssrn.com/abstract=3407779
References 2017 Annual Report. (2017). Retrieved from Artificial Intelligence Index website: https://aiindex.org/2017-­‐report.pdf AbuShawar, B., & Atwell, E. (2015). ALICE Chatbot: Trials and Outputs. Computación y Sistemas, 19(4). doi:10.13053/cys-­‐19-­‐4-­‐2326 Accenture. (2017). Boost your AIQ -­‐ Transforming into an AI business. Retrieved from G20 Young Entrepreneurs' Alliance website: https://www.accenture.com/t20170614T050454Z w /us-­‐ en/_acnmedia/Accenture/next-­‐gen-­‐5/event-­‐g20-­‐yea-­‐summit/pdfs/Accenture-­‐Boost-­‐Your-­‐ AIQ.pdf#zoom=50 AIAA. (n.d.). Artificial Intelligence for Cybersecurity. Retrieved 11 November, 2018 from The American Institute of Aeronautics and Astronautics website: https://www.aiaa.org/protocolAI/ Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine Bias. ProPublica. Retrieved from www.propublica.org Arthur, C. (2013, August 24). Tech giants may be huge, but nothing matches big data. The Guardian. Retrieved from https://www.theguardian.com Artificial Intelligence: the Future of Fighting Cybercrime. (2016, December 27). Panda Media Center. Retrieved from https://www.pandasecurity.com Ashford, W. (2018, June 4). Offensive AI unlikely in the near future, says Mikko Hypponen. Computer Weekly. Retrieved from www.computerweekly.com Ashworth, A., & Horder, J. (2013). Principles of Criminal Law. New York: Oxford University Press. Auditor General of Australia. (2019). The Australian Criminal Intelligence Commission’s Administration of the Biometric Identification Services Project (Auditor-­‐General Report No. 24 2018-­‐2019 Performance Audit). Retrieved from https://www.anao.gov.au/sites/default/files/Auditor-­‐General_Report_2018-­‐2019_24.pdf Australian Law Reform Commission. (2014). Serious Invasions of Privacy in the Digital Era. Sydney: Commonwealth of Australia. Bača, M., Schatten, M., & Rabuzin, K. (2007). Towards an open biometric ontology. Journal of Information and Organizational Sciences, 31(1), 1-­‐11. Barber, G. (2019, Janurary 31). San Francisco Could Be First to Ban Facial Recognition Tech. Retrieved from Wired: https://www.wired.com/story/san-­‐francisco-­‐could-­‐be-­‐first-­‐ban-­‐facial-­‐ recognition-­‐tech/ Bello, I. (2017, July 17). Beginners guide to Artificial Intelligence 'AI'. Becoming Human. Retrieved from https://becominghuman.ai Berk, R. (2017). An impact assessment of machine learning risk forecasts on parole board decisions and recidivism. Journal of Experimental Criminology, 13(2), 193-­‐216. Doi: 10.1007/s11292-­‐ 017-­‐9286-­‐2 46 Electronic copy available at: https://ssrn.com/abstract=3407779
Berwick, A. (2018, november 18). Venezuela is rolling out a new ID card manufactured in China that can track, reward, and punish citizens. Business Insider. Retrieved from https://www.businessinsider.com Bhatia, R. (2017, December 27). Understanding the difference between Symbolic AI & Non Symbolic AI. Analytics India. Retrieved from https://www.analyticsindiamag.com BI Intelligence. (2016, December 14). 80% of businesses want chatbots by 2020. Business Insider. Retrieved from https://www.businessinsider.com Biometrics Institute. (n.d.). Types of Biometrics. Retrieved 12 December, 2018 from https://www.biometricsinstitute.org/types-­‐of-­‐biometrics Bock, K., Shannon, S., Movahedi, Y., & Cukier, M. (2017). Application of Routine Activity Theory to Cyber Intrusion Location and Time. Paper presented at the 139-­‐146. doi:10.1109/EDCC.2017.24 Bond, S. (2017). Artificial intelligence and quantum computing aid cyber crime fight. Financial Times. Retrieved from https://www.ft.com Bostrom, N. (2013). Existential Risk Prevention as Global Priority. Global Policy, 4(1), 15-­‐31. Doi: 10.1111/1758-­‐5899.12002 Boudon, R. (2003). Beyond Rational Choice Theory. Annual Review Of Sociology, 29(1), 1-­‐21. doi: 10.1146/annurev.soc.29.010202.100213 Broadhurst, R., Lord, D., Maxim, D., Woodford-­‐Smith, H., Johnston, C. R., Carroll, S., Chung, H., Trivedi, H., & Sabol, B. (2018). Malware trends on darknet crypto-­‐markets: research review. The Australian National University Cybercrime Observatory: Canberra. Broadhurst, R., Maller, R., Maller, M. and B. Bouhours (2017). The Recidivism of Homicide Offenders in Western Australia, The Australian and New Zealand Journal of Criminology, First published date: July-­‐27-­‐2017 10.1177/0004865817722393 Broadhurst, R.G. (2006) Developments in the global law enforcement of cyber-­‐crime, Policing: an International Journal of Police Strategies and Management, Vol. 29(3): 408-­‐433. Brown R. & Shelling J. (2019). Exploring the implications of child sex dolls. Trends & issues in crime and criminal justice No. 570. Canberra: Australian Institute of Criminology. Retrieved from https://aic.gov.au/publications/tandi/tandi570 Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., … Amodei, D. (2018). The Malicious Use of Artificial Intelligence: Forecasting, Prevention and Mitigation. Retrieved from the Cornell University website. Retrieved from https://arxiv.org/abs/1802.07228v1 Brunkard, P. (2018, September 3). The future of IOT is AI [Web log post]. Retrieved from https://www.techuk.org/insights/opinions/item/13827-­‐the-­‐future-­‐of-­‐iot-­‐is-­‐ai Buchanan, B. G. (2006). A (Very) Brief History of Artificial Intelligence. AI Magazine, 26(4), 53-­‐60. Burkitt, B. (2018, June 22). Self-­‐driving Uber fatal crash: Experts say prosecution would be precedent setting. AzCentral. Retrieved from https://www.azcentral.com 47 Electronic copy available at: https://ssrn.com/abstract=3407779
Bush, S. (2017). Software bugs fixed automatically with AI and Big Data. Electronics Weekly. Retrieved from https://www.electronicsweekly.com Cambridge Dictionary. (n.d.). State of Mind. Retrieved from https://dictionary.cambridge.org/dictionary/english/state-­‐of-­‐mind Carney, M. (2018, September 18). Leave no dark corner. Retrieved from ABC News: https://www.abc.net.au/news/2018-­‐09-­‐18/china-­‐social-­‐credit-­‐a-­‐model-­‐citizen-­‐in-­‐a-­‐digital-­‐ dictatorship/10200278 Chaturvedi, A. (2018, October 7). 13 major Artificial Intelligence trends to watch for in 2018. Geospatial World. Retrieved from https://www.geospatialworld.net Čerka, P., Grigienė, J., Sirbikytė, G., 2017. Is it possible to grant legal personality to artificial intelligence software systems? Computer Law & Security Review 33, 685–699. https://doi.org/10.1016/j.clsr.2017.03.022 Čerka, P., Grigienė, J., Sirbikytė, G., 2015. Liability for damages caused by artificial intelligence. Computer Law & Security Review 31, 376–389. https://doi.org/10.1016/j.clsr.2015.03.008 Chen, C. H. ed. (2015). Handbook of pattern recognition and computer vision. World Scientific. Cimpanu, C. (2017, January 12). Scientists Extract Fingerprints from Photos Taken From up to Three Meters Away. BleepingComputer. Retrieved from https://www.bleepingcomputer.com Cole, S. A. (2009). Suspect Identities: A History of Fingerprinting and Criminal Identification. Cambridge, MA: Harvard University Press. Colman, A. (2015). Mens Rea. Retrieved from A Dictionary of Psychology: http://www.oxfordreference.com/view/10.1093/acref/9780199657681.001.0001/acref-­‐ 9780199657681-­‐e-­‐4967# Condliffe, J. (2017, October 2). Oracle’s New Database Uses AI to Patch Itself. MIT Technology Review. Retrieved from https://www.technologyreview.com Connor-­‐Simons, A. (2016, April 18). System predicts 85 percent of cyber-­‐attacks using input from human experts. MIT News. Retrieved from http://news.mit.edu Cooper, D. (2016, December 23). We know nothing about the future of sex robots. Engadget. Retrieved from https://www.engadget.com Copeland, B., & Proudfoot, D. (1999). Alan Turing’s Forgotten Ideas in Computer Science. Scientific American, 280(4), 98-­‐103. Doi: 10.1038/scientificamerican0499-­‐98 Corbett-­‐Davies, S., Pierson, E, M., Feller, A., & Goel, S. (2016, October 17). A computer program used for bail and sentencing decisions was labelled biased against blacks. It’s actually not that clear. The Washington Post. Retrieved from www.washingtonpost.com Coward, W. M., & Sackett, P. R. (1990). Linearity of ability-­‐performance relationships: A reconfirmation. Journal of Applied Psychology, 75(3), 297-­‐300. doi:10.1037//0021-­‐ 9010.75.3.297 48 Electronic copy available at: https://ssrn.com/abstract=3407779
Critical Incident Response Group (1999). The School Shoot: A Threat Assessment Perspective. Department of Justice. Quantico, Virginia: Author. Culp, S. (2018, April 30). Virtual Reality Plus Artificial Intelligence Can Transform Risk Management. Forbes. Retrieved from https://www.forbes.com Davydova, O. (2017, August 17). 10 Applications of Artificial Neural Networks in Natural Language Processing. Medium. Retrieved from https://medium.com Dawson D, Schleiger E, Horton J, McLaughlin J, Robinson C, Quezada G, Scowcroft J, and Hajkowicz S (2019) Artificial Intelligence: Australia’s Ethics Framework. Data61 CSIRO, Australia. Deep Sentinel. (n.d.). Smart Home Surveillance System. Retrieved from https://www.deepsentinel.com/ Derrac, J., Triguero, I., Garcia, S., & Herrera, F. (2012). Integrating Instance Selection, Instance Weighting, and Feature Weighting for Nearest Neighbor Classifiers by Coevolutionary Algorithms. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 42(5), 1383-­‐1397. Doi: 10.1109/TSMCB.2012.2191953 Dickson, B. (2018, November 28). The Malware Of The Future Will Have AI Superpowers. Gizmodo. Retrieved from https://www.gizmodo.com.au Dixon, W. (2018, November 21). What is adversarial artificial intelligence and why does it matter? World Economic Forum. Retrieved from www.weforum.org D'souza, R. (2018, October 19). Symbolic AI v/s Non-­‐Symbolic AI, and everything in between? Medium. Retrieved from https://medium.com Duwe, G. & Kim, K. (2015). Out With the Old and in With the New? An Empirical Comparison of Supervised Learning Algorithms to Predict Recidivism, Criminal Justice Policy Review, 28(6), 570-­‐600. Doi: 10.1177/0887403415604899 Edmark, K. (107). Unemployment and Crime: Is There a Connection? The Scandinavian Journal of Economics, 107(2), 353-­‐373. Edwards, E. (2016, August 31). Predictive policing software is more accurate at predicting policing than predicting crime. American Civil Liberties Union. Retrieved from https://www.aclu.org Ericson, R. V., & Haggerty, K. D. (1997). Policing the Risk Society. Toronto: University of Toronto Press. Eterno, J. A., Verma, A., & Silverman, E. B. (2014). Police Manipulations of Crime Reporting: Insiders’ Revelations. Justice Quarterly, 33(5), 811-­‐835. Doi: https://doi.org/10.1080/07418825.2014.980838 Executive Office of the President (2016). Big Data: Seizing Opportunities, Preserving Values. Washington, D.C: The White House. Retrieved from https://obamawhitehouse.archives.gov/sites/default/files/docs/big_data_privacy_report_5.1. 14_final_print.pdf Expert System. (2017, October 5). Natural language processing applications. Retrieved from https://www.expertsystem.com/natural-­‐language-­‐processing-­‐applications/ 49 Electronic copy available at: https://ssrn.com/abstract=3407779
Fadelli, I. (2018). Using machine learning to detect software vulnerabilities. Tech Xplore. Retrieved from https://techxplore.com Felson, M. (1994). Crime and everyday life: Insight and implications for society. Thousand Oaks, CA: Pine. Filar, B. & Anderson, H. (2018, February 20). Malicious Use of Artificial Intelligence in InfoSec [Web Log Post]. Retrieved from https://www.endgame.com/blog/technical-­‐blog/malicious-­‐use-­‐ artificial-­‐intelligence-­‐infosec Five key trends to watch in 2018 as cybercriminals continue to innovate (2017, December 4). HelpNet Security. Retrieved from www.helpnetsecurity.com Fox, C. (2018, May 15). Face recognition police tools ‘staggeringly inaccurate’. BBC News. Retrieved from https://www.bbc.com Friedman, B. & Nissenbaum, H. (1996). Bias in Computer Systems. ACM Transactions on Information Systems, 14(3), 330-­‐347. Doi: 10.1145/230538.230561 Fruhlinger, J. (2017, August 22). What is Stuxnet, who created it and how does it work? CSO Online. Retrieved from https://www.csoonline.com Fussell, S. (2018, July 26). Can We Make Non-­‐Racist Face Recognition? Retrieved from Gizmodo: https://www.gizmodo.com.au/2018/07/can-­‐we-­‐make-­‐non-­‐racist-­‐face-­‐recognition/ Gandomi, A., & Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35(2), 137-­‐144. doi:10.1016/j.ijinfomgt.2014.10.007 Gardner, H. E. (2008). Multiple Intelligences: New Horizons in Theory and Practice. London, England: Hachette UK. Gent, E. (2018, December 3). Britain is developing an AI-­‐powered predictive policing system. Singularity Hub. Retrieved from https://singularityhub.com Gibbs, S. (2018, May 8). Uber’s self-­‐driving car saw the pedestrian but didn’t swerve – report. The Guardian. Retrieved from https://www.theguardian.com Good, I. J. (1966). Speculations Concerning the First Ultraintelligent Machine. Advances in Computers, 6(c), 31-­‐88. doi:10.1016/s0065-­‐2458(08)60418-­‐0 Grandview Research. (2018). Voice and Speech Recognition Market Size, Share & Trends Analysis Report, By Function, By Technology (AI, Non-­‐AI), By Vertical (Healthcare, BFSI, Automotive), And Segment Forecasts. Retrieved from GrandView Research website: https://www.grandviewresearch.com/industry-­‐analysis/voice-­‐recognition-­‐market Guihot, M., Suzor, N.P., Matthew, A.F., 2017. Nudging Robots: Innovative Solutions to Regulate Artificial Intelligence. Vanderbilt Journal of Entertainment and Technology Law 20, 73. Gurkaynak, G., Yilmaz, I., Haksever, G., 2016. Stifling artificial intelligence: Human perils. Computer Law & Security Review 32, 749–758. https://doi.org/10.1016/j.clsr.2016.05.003 50 Electronic copy available at: https://ssrn.com/abstract=3407779
Hannem, S., Sanders, C. B., Schneider, C. J., Doyle, A., & Christensen, T. (2019). Can "Big Data" Analytics Predict Policing Practices. In Security and Risk Technologies in Criminal Justice: Critical Perspectives (pp. 41-­‐82). Toronto, Ontario: Canadian Scholars. Unpublished Manuscript Hallevy, G. (2010). The Criminal Liability of Artificial Intelligence Entities -­‐ from Science Fiction to Legal Social Control. Akron Intellectual Property Journal, 4(2), 171-­‐201. Hardesty, L. (2018, February 11). Study finds gender and skin-­‐type bias in commercial artificial-­‐ intelligence systems. MIT News. Retrieved from www.news.mit.edu Harris, J. (2018, March 5). The tyranny of algorithms is part of our lives: soon they could rate everything we do. The Guardian. Retrieved from https://www.theguardian.com Hasan, S. M., Baqai, A. A., Butt, S. U., Ausaf, M. F., & Zaman, U. K. (2018). Incorporation of part complexity into system scalability for flexible/reconfigurable systems. The International Journal of Advanced Manufacturing Technology, 99(9-­‐12), 2959-­‐2979. doi: https://doi.org/10.1007/s00170-­‐018-­‐2654-­‐x Hatton, C. (2015, October 26). China sets up huge 'social credit' system. BBC News. Retrieved from https://www.bbc.com Holder, C., Khurana, V., Harrison, F., Jacobs, L., 2016. Robotics and law: Key legal and regulatory implications of the robotics age (Part I of II). Computer Law & Security Review 32, 383–402. https://doi.org/10.1016/j.clsr.2016.03.001 Heath, N. (2018, August 22). What is artificial general intelligence? ZDNet. Retrieved from https://www.zdnet.com Hurst, D. (2018, February 6). Japan lays groundwork for boom in robot carers. The Guardian. Retrieved from https://www.theguardian.com Hawking, S., Russell, S., Tegmark, M., & Wilczek, F. (2014, May 1). Transcendence looks at the implications of artificial intelligence -­‐ but are we taking AI seriously enough? The Independent. Retrieved from www.independent.co.uk IFSEC Global [Author]. (2016, October 28). Biometric security systems: a guide to devices, fingerprint scanners, facial recognition, access control. IFSEC GLOBAL. www.ifsecglobal.com Ireland, C. (2012, September 13). Alan Turing at 100. Harvard News. Retrieved from https://news.harvard.edu Janai, J., Güney, F., Behl, A., & Geiger, A. (2017). Computer vision for autonomous vehicles: Problems, datasets and state-­‐of-­‐the-­‐art. arXiv preprint arXiv:1704.05519. Jouvenal, J. (2016, November 23). Is crime prediction software the way forward for modern policing? Or biased against minorities? The Independent. Retrieved from www.independent.co.uk Justia. (2018, April 17). United States v. Microsoft Corp., 584 U.S. (2018). Retrieved from Justia: https://supreme.justia.com/cases/federal/us/584/17-­‐2/ King, T.C., Aggarwal, N., Taddeo, M., Floridi, L., 2019. Artificial Intelligence Crime: An Interdisciplinary Analysis of Foreseeable Threats and Solutions. Science and Engineering Ethics. Retrieved from https://doi.org/10.1007/s11948-­‐018-­‐00081-­‐0 51 Electronic copy available at: https://ssrn.com/abstract=3407779
Kingston, J. (2018). Artificial Intelligence and Legal Liability. Retrieved from https://arxiv.org/ftp/arxiv/papers/1802/1802.07782.pdf Kumar, M. (2018, December 6). SNDBOX: AI-­‐Powered Online Automated Malware Analysis Platform. The Hacker News. Retrieved from https://thehackernews.com Kurzweil, R. (1990). The Age of Intelligent Machines. Cambridge, MA: MIT Press. Kurzweil, R. (2005). The Singularity Is Near: When Humans Transcend Biology. London, England: Penguin. Larson, J., Mattu, S., Kirchner, L. & Angwin, J. (2016, May 23). How We Analyzed the COMPAS Recidivism Algorithm. ProPUBLICA. Retrieved from www.propublica.org Latour, B. (1987) Science in Action: How to follow Scientists and Engineers Through Society, Milton Keynes, Open University Press. Latour, B., (2005). Reassembling the Social: An Introduction to Actor-­‐Network-­‐Theory. Oxford: Oxford UP. Legal Information Institute. (n.d.). Mens Rea. Retrieved from https://www.law.cornell.edu/wex/mens_rea Leyden, J. (2004, May 26). FBI apology for Madrid bomb fingerprint fiasco. The Register. Retrieved from https://www.theregister.co.uk Li, S. Z. (2004). Face Recognition: Technical Challenges and Research Directions. In: Li S.Z., Lai J., Tan T., Feng G., Wang Y. (eds) Advances in Biometric Person Authentication. SINOBIOMETRICS 2004. Lecture Notes in Computer Science, vol 3338. Springer, Berlin, Heidelberg. LogRhythm. (2017). The Rise of AI-­‐enabled Threat Detection. Retrieved from LogRhythm website: https://logrhythm.com/pdfs/brochures/uk-­‐ai-­‐threat-­‐detection-­‐brochure.pdf Lohr, S. (2013, February 1). The Origins of 'Big Data': An Etymological Detective Story [Web log post]. Retrieved from https://bits.blogs.nytimes.com/2013/02/01/the-­‐origins-­‐of-­‐big-­‐data-­‐an-­‐ etymological-­‐detective-­‐story/ Luhmann, N. (1989). Law as a social system. Northwestern University Law Review, 83(1-­‐2), 136-­‐150. Luhmann, N. (1998), Observations on modernity, Stanford: Stanford University Press. Luhmann N. (2003), Organization. In: Bakken, T. and Hernes, T. (eds.), Autopoietic Organization Theory Drawing on Niklas Luhmann's Social Systems Perspective (pp. 31-­‐52), Abstract, Liber, Copenhagen Business School Press, Oslo. Lum, K. & Isaac, W. (2016). To predict and serve? Significance, 13(5), 14-­‐19. Doi: https://doi-­‐ org.virtual.anu.edu.au/10.1111/j.1740-­‐9713.2016.00960.x Ma, A. (2018, October 30). China has started ranking citizens with a creepy 'social credit' system -­‐-­‐ here's what you can do wrong, and the embarrassing, demeaning ways they can punish you. Retrieved from Business Insider Australia: https://www.businessinsider.com.au/china-­‐social-­‐ credit-­‐system-­‐punishments-­‐and-­‐rewards-­‐explained-­‐2018-­‐4?r=US&IR=T Ma, S., Thung, F., Lo, D., Sun, C., & Deng, R. H. (2017). VuRLE: Automatic vulnerability detection and repair by learning from examples. Paper presented at the Conference: European Symposium 52 Electronic copy available at: https://ssrn.com/abstract=3407779
on Research in Computer Security, 10493 229-­‐246. doi:10.1007/978-­‐3-­‐319-­‐66399-­‐9_13 MacAskill, E. (2014, September 2). Did 'Jihadi John' kill Steven Sotloff? The Guardian. Retrieved from www.theguardian.com 53 Electronic copy available at: https://ssrn.com/abstract=3407779
McAllister, A., 2011. Stranger than Science Fiction: The Rise of A.I. Interrogation in the Dawn of Autonomous Robots and the Need for an Additional Protocol to the U.N. Convention Against Torture. Minnesota Law Review 47. Maleković, M., Schatten, M. (2008). Leadership in Team Based Knowledge Management – An Autopoietic Information System's Perspective, In: Aurer, B., Bača, M. (eds.), 19th Central European Conference on Information and Intelligent Systems – Conference Proceedings (pp. 53-­‐56), Faculty of Organization and Informatics. Manyika, J. (2011). Big data: The next frontier for innovation, competition, and productivity. Retrieved from McKinsey Global Institute website: https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/ Our%20Insights/Big%20data%20The%20next%20frontier%20for%20innovation/MGI_big_data _exec_summary.ashx Maturana, H. (1975), The organization of the living: A theory of the living organization. International Journal of Man-­‐Machine Studies, 7(3), 313-­‐332. Doi: https://doi.org/10.1016/S0020-­‐ 7373(75)80015-­‐0 Maturana, H. R., Poerksen, B. (2007). Autopoiesis and Social Theory: A Conversation. Journal of Sociocybernetics, 5 (1/2): 68-­‐73. Maza, C. (2017, October 26). Saudi Arabia Gives Citizenship to a Non-­‐Muslim, English-­‐Speaking Robot. Retrieved from Newsweek: https://www.newsweek.com/saudi-­‐arabia-­‐robot-­‐sophia-­‐ muslim-­‐694152 McCarthy, J. (1978). History of LISP. History of programming languages, 173-­‐185. doi:10.1145/800025.1198360 McCarthy, J., Minsky, M., Rochester, N., & Shannon, C. E. (1955, August). A proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Retrieved from http://raysolomonoff.com/dartmouth/boxa/dart564props.pdf McFarland, M. (2014, October 24). Elon Musk: ‘With artificial intelligence we are summoning the demon.’ The Washington Post. Retrieved from www.washintongpost.com Melick, M. (2003). The Relationship between Crime and Unemployment. The Park Place Economist, 30-­‐36. Microsoft Philippines PR Team. (2017, August 17). AI (Artificial Intelligence) and Cybercrime: The good and bad news. Microsoft News Center Philippines. Retrieved from https://news.microsoft.com Mitchell, T. (1988). Quantifying Inductive Bias: AI Learning Algorithms and Valiant’s Learning Framework. Artificial Intelligence, 36(2), 177-­‐221. Doi: 10.1016/0004-­‐3702(88)90002-­‐1 Moses, L. B., & Chan, J. (2014). Using Big Data For Legal And Law Enforcement Decisions: Testing The New Tools. University of New South Wales Law Journal, The, 37(2), 643-­‐678. Mocheva, Z. (n.d.). The CLOUD Act: What you need to know. Retrieved from CloudSigma: https://www.cloudsigma.com/the-­‐cloud-­‐act-­‐what-­‐you-­‐need-­‐to-­‐know/ 54 Electronic copy available at: https://ssrn.com/abstract=3407779
Mooney, A. C., M.P.H., Giannella, E., PhD., Glymour, M. M., Neilands, T. B., PhD., Morris, M. D., PhD., Tulsky, J., M.D., & Sudhinaraset, M., PhD. (2018). Racial/Ethnic disparities in arrests for drug possession after california proposition 47, 2011–2016. American Journal of Public Health, 108(8), 987-­‐993. doi:http://dx.doi.org.virtual.anu.edu.au/10.2105/AJPH.2018.304445 Moor, J. (2006). The Dartmouth College Artificial Intelligence Conference: The Next Fifty Years. AI Magazine, 27(4), 87-­‐91. Morris, T. (2004). Computer Vision and Image Processing. Palgrave Macmillan. Mouzos, J., & Venditto, J. (2003). Contract Killings in Australia. Canberra: Australian Institute of Criminology. Newman, L. (2017, April 27). A Clever Plan to Secure the Internet of Things Could Still Have Big Drawbacks. Wired. Retrieved from https://www.wired.com NSW Attorney General's Department. (2011). Routine activity theory. Retrieved from the NSW Justice website: http://www.crimeprevention.nsw.gov.au/Documents/routine_activity_factsheet_nov2014.pd f Nilsson, P. (2018, November 26). First UK police force to try predictive policing ends contract. Financial Times. Retrieved from www.ft.com Northpointe. (2012). Practitioners Guide to COMPAS. Ohio, USA: Author. Perry, W. L., McInnis, B., Price, C. C., Smith, S. C., & Hollywood, J. S. (2013). Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations. Santa Monica, CA: RAND. Police Executive Research Forum. (2014). Future Trends in Policing. Washington, DC: Police Executive Research Forum. http://www.policeforum.org/assets/docs/ Free_Online_Documents/Leadership/future%20trends%20in%20policing% 202014.pdf. PredPol releases gun violence prediction technology. (2013). Manufacturing Close -­‐ Up. Retrieved from https://search-­‐proquest-­‐com.virtual.anu.edu.au/docview/1432230934?accountid=8330 Price, M & Ball, P. (2014). Big Data, Selection Bias, and the Statistical Patterns of Mortality in Conflict. SAIS Review of International Affairs, 34(1), 9-­‐20. Doi: 10.1353/sais/2014.0010 Randazzo, R., & Pineda, P. (2019, February 2). Tempe faces $10 million claim in Uber self-­‐driving vehicle fatality. Retrieved from AZ Central: https://www.azcentral.com/story/news/local/tempe/2019/02/02/tempe-­‐faces-­‐10-­‐million-­‐ claim-­‐uber-­‐self-­‐driving-­‐vehicle-­‐fatality/2744423002/ Ray, S. (2018, August 11). History of AI? Towards Data Science. Retrieved from https://towardsdatascience.com Rollet, C. (2018, June 5). The odd reality of life under China's social credit system. Wired. Retrieved from https://www.wired.co.uk Rouse, M. (n.d.). What is behavioral biometrics? TechTarget. Retrieved from https://whatis.techtarget.com 55 Electronic copy available at: https://ssrn.com/abstract=3407779
Rouse, M. (2012). What is offensive security? TechTarget. Retrieved from https://whatis.techtarget.com Rouse, M. (2015). What is keystroke ID (keystroke identification)? TechTarget. Retrieved from https://whatis.techtarget.com Russell, S. J., & Norvig, P. (2003). Artificial Intelligence: A Modern Approach (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Saunders, J., Hunt, P., & Hollywood, J. S. (2017). Predictions Put Into Practice: A Quasi-­‐Experimental Evaluation of Chicago’s Predictive Policing Pilot. Journal Of Experimental Criminology, 12(3), 347-­‐371. doi: 10.1007/s11292-­‐016-­‐9272-­‐0 Sassóli, M., 2014. Autonomous Weapons and International Humanitarian Law: Advantages, Open Technical Questions and Legal Issues to be Clarified 90, 34. Schatten, M., & Bača, M. (2010). A critical review of autopoietic theory and its applications to living, social, organizational and information systems. Društvena istraživanja: časopis za opća društvena pitanja, 19(4-­‐5), 837-­‐852. Scherer, M.U., 2016. Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies. Harvard Journal of Law and Technology 29, 353. Schmitt, M.N., Thurnher, J.S., 2013. “Out of the Loop”: Autonomous Weapon Systems and the Law of Armed Conflict. Harvard National Security Journal 4. Schultebraucks, L. (2018, December 5). A Short History of Artificial Intelligence. Dev. Retrieved from https://dev.to Sharman, J. (2018, May 13). Metropolitan Police’s facial recognition technology 98% inaccurate, figures show. Independent. Retrieved from www.independent.co.uk Shoebridge, M. (2018, July 20). AI and national security: lethal robots or better logistics? The Strategist. Retrieved from www.aspistrategist.org.au Short, M. B., D’Orsogna, M. R., Brantingham, P. J., & Tita, G. E. (2009). Measuring and modeling repeat and near-­‐repeat burglary effects. Journal of Quantitative Criminology, 25(3), 325-­‐339. Doi: https://doi.org/10.1007/s10940-­‐009-­‐9068-­‐8 State v. Loomis. (2017). Harvard Law Review, 130(5), 1530-­‐1530. Retrieved from https://harvardlawreview.org/2017/03/state-­‐v-­‐loomis/ Sullivan, L. (2009). The SAGE Glossary of the Social and Behavioural Sciences. Thousand Oaks: SAGE Publications. Retrieved from The SAGE Glossary of the Social and Behavioural Sciences. Sun, D. W. (Ed.). (2016). Computer vision technology for food quality evaluation. Academic Press. Szal, A. (2017, March 20). Treasury Secretary, Despite Forecasts, Not Worried About AI Taking Jobs. Manufacturing Business Technology. Tarantola, A. (2017, July 25). Will we be able to control the killer robots of tomorrow? Engadget. Retrieved from https://www.engadget.com 56 Electronic copy available at: https://ssrn.com/abstract=3407779
Targeted Attacks. (n.d.). In Trend Micro website. Retrieved from https://www.trendmicro.com/vinfo/us/security/definition/targeted-­‐attacks Thakkar, D. (n.d.). Importance of Biometric Security in Modern Society. Bayometric. Retrieved November 11, 2018, from https://www.bayometric.com Thakkar, D. (n.d.). Top Five Biometrics: Face, Fingerprint, Iris, Palm and Voice. Bayometric. Retrieved November 17, 2018, from https://www.bayometric.com/biometrics-­‐face-­‐finger-­‐iris-­‐palm-­‐ voice/ Tiku, N. (2018, December 6). Microsoft wants to stop AI's 'race to the bottom'. Retrieved from Wired: https://www.wired.com/story/microsoft-­‐wants-­‐stop-­‐ai-­‐facial-­‐recognition-­‐bottom/ Titcomb, J. (2017, April 11). Why your smartphone's fingerprint scanner isn't as secure as you might think. Retrieved from The Telegraph (UK): https://www.telegraph.co.uk/technology/2017/04/11/smartphone-­‐fingerprint-­‐scanners-­‐ could-­‐easily-­‐fooled-­‐fake-­‐prints/ Turing, A. M. (1950). Computing machinery and intelligence. Mind, LIX (236), 433-­‐460. doi:10.1093/mind/lix.236.433 United Nations Office of Drugs and Crime (2018) Cybercrime Repository Database of Legislation. Retrieved from https://sherloc.unodc.org/cld/v3/cybrepo/legdb/index.html?lng=en University of Toronto. (1999). Artificial Intelligence | Game Playing. Retrieved from http://psych.utoronto.ca/users/reingold/courses/ai/games.html Vasudevan, V. (2018, July 24). How AI Is Transforming Cyber Defense. Forbes. Retrieved from https://www.forbes.com Vladeck, D.C., 2014. Machines Without Principals: Liability Rules and Artificial Intelligence. Washington Law Review 89, 117. Vidal, L. (2018, December 29). Venezuelans shudder at news of biometric ID deal with Chinese tech giant. Uncensored. Retrieved from https://www.uncensored.co.nz Vigna, G. (n.d.). How AI will help in the fight against malware. TechBeacon. Retrieved January 5, 2019 from: https://techbeacon.com/ Violino, B. (2015, March 3). Biometric security is on the rise. CSO Online. Retrieved from https://www.csoonline.com West, D. M. (2018). The Future of Work: Robots, AI, and Automation. Washington, D.C.: Brookings Institution Press. What Is XR (Extended Reality) and What Can We Do With It? (2018, August 24). SaM Solutions. Retrieved from https://www.sam-­‐solutions.com Xiang, T. (2015, January 29). Alibaba’s finance arm officially launches credit scoring service sesame credit. Technode. Retrieved from https://technode.com Yu, D. (2019, January 22). PepsiCo Sees Future In Artificial Intelligence After Launching Snack-­‐ Delivery Robot. Forbes. Retrieved from https://www.forbes.com 57 Electronic copy available at: https://ssrn.com/abstract=3407779
Zaveri, M. (2019 March 5). Prosecutors Don’t Plan to Charge Uber in Self-­‐Driving Car’s Fatal Accident. New York Times, Retrieved from https://www.nytimes.com/2019/03/05/technology/uber-­‐self-­‐driving-­‐car-­‐arizona.html Zetter, K. (2014). Countdown to Zero Day. New York, NY: Crown Publishers. 58 Electronic copy available at: https://ssrn.com/abstract=3407779
Appendix: AI in Criminal Law– Overview of Current Legislation in Selected Jurisdictions This appendix summarises selected legislation from 13 countries that may be able to address the misuse of AI. The information used in this appendix is compiled from the United Nations Office on Drugs and Crime Cybercrime Repository Database of Legislation. Table A1: Overview of legislation by country Country Legislation Relevant Sections Australia Criminal Code Act 1995 Schedule 1 ‘The Criminal Code’ Chapter 10, Part 10.7 – Computer Offences s.477.1 – Unauthorised access, modification or impairment with intent to commit a serious offence s.477.2
– Unauthorised modification of data to cause impairment s.477.3
– Unauthorised impairment of electronic communication s.478.1 – Unauthorised access to, or modification of, restricted data s.478.2 Unauthorised impairment of data held on a computer disk, etc. Brazil Codigo Penal Art.154-­‐A Estonia Penal Code s.206 – interference with computer data s.207 – Hindering of operation of computer system s.217 – Unlawful use of a computer system Finland The Criminal Code of Finland s.7a – interference with a computer system s.7b – aggravated interference with a computer system s.8 – Computer break in s.8a – aggravated computer break in 59 Electronic copy available at: https://ssrn.com/abstract=3407779
France Penal Code Chapter III – Unauthorised access to automated data processing systems Art. 323-­‐1 Art. 323-­‐2 Art. 323-­‐3 Germany German Criminal Code s.303a – Data tampering s.303b – Computer sabotage s.253a – Computer fraud Nigeria Cyber Crime Act 2015 Part III – Offences and Penalties Art. 6 -­‐ Unlawful access to a Computer Art. 8 – Unauthorised modification of computer data Art. 9 – System interference Art. 10 – Misuse of Devices People’s Republic China Criminal Law of the People’s Republic of of China* Art. 285 Art. 286 Art. 287 Republic Korea of Criminal Act Art.347-­‐2 – Fraud by Use of Computer Act on the Protection of Information and Communications Infrastructure Art 12 – Prohibition against Intrusion, etc of Critical Information and Communications Infrastructure Act on Promotion of Information and Communications Network Utilisation and Art 48 – Prohibition on Intrusive Data Protection, etc. Acts, etc. on Information and Communications Network 60 Electronic copy available at: https://ssrn.com/abstract=3407779
Singapore Computer Misuse and Cybersecurity Act [date] South Africa Electronic Communications Transactions Act 25 of 2002 United Kingdom Computer Misuse Act 1990 United States USC – Title 18 Crimes and Criminal of America Procedure s.3 – Unauthorised access to computer material s.4 – Access with intent to commit or facilitate an offence s.5 – Unauthorised Modification of computer material s.6 – Unauthorised use or interception of a computer service s.7 – Unauthorised obstruction of use of computer s.8 – Unauthorised disclosure of an access code and s.86 – Unauthorised access to, interception of or interference with data s.87 – computer related fraud, extortion or bribery s.1 – Unauthorised access to computer material s.2 – Unauthorised access with intent to commit or facilitate commission of further offences s.3 – Unauthorised acts with intent to impair, or with recklessness as to impairing operation of computer, etc S1029 – Fraud and related activity in connection with access devices s1030 -­‐ Fraud and related activity in connection with computers s2701 – Unlawful access to stored communications Australia Australia’s legislation is broadly designed, so it may be applicable to a wide range of technology and to a wider range of actions. This is done intentionally so that the legislation is relatively ‘future proofed’ to meet the rapidly evolving technological landscape. For example, the term ‘computer’ is left undefined and can thus adapt to technological advancements. The provisions cover the unauthorised access, modification or impairment of data held in a computer (United Nations Office on Drugs and Crime (UNODC), 2018). Australia’s legislation does not focus on a particular crime as the purpose of the action. However; it may be limited in that it requires the actions to be done with the intent of committing a serious crime (UNODC, 2018). According to the provision of the Australian 61 Electronic copy available at: https://ssrn.com/abstract=3407779
Criminal Code (s477.1), a serious crime is a crime punishable by imprisonment for life or a period of 5 years or more, which potentially leaves any unauthorised access, modification or impairment done with the intent of committing a crime punishable by imprisonment for less than 5 years not expressly covered by the provision (Other countries with a similar legislative framework are Brazil, Estonia, Finland, Nigeria, Singapore and the United Kingdom (UNODC, 2018). France The content of the offences in the French penal code are similar to offences in other countries with regards to unauthorised access, modification or impairment. The articles refer to fraudulently accessing or remaining with all or part of an automated data processing system (Art. 323-­‐1), obstructing or interfering with the functioning of an automated data processing system (Art. 323-­‐2) and fraudulent introduction of data into an automated data processing system (Art. 323-­‐3) (UNODC, 2018). Two points of difference are noted between the French legislation and legislation in the other countries. Firstly, the French legislation refers to ‘fraudulent’ access, etc., whereas many of the other countries refer to ‘unauthorised’ access, etc. (UNODC, 2018). Secondly, the French legislation does not refer to computers, instead using the term ‘automated data processing system’ (UNODC, 2018). In practice however, it is unlikely that these differences in wording would affect the applicability of legislation to actions targeting AI. Germany Though the wording of the German provisions is fairly broad, the offences are categorised under ‘criminal damage’ (UNODC, 2018). This means the provisions require some sort of alteration to the data or computer system, and are not as broad as some of the provisions in other countries, which criminalise unauthorised or unlawful access, without requiring that the access have caused any damage or change. However, the applicability of the provisions is still broader than those that limit the application to actions with a specific motive. People’s Republic of China China’s legislation covers both the approach taken in Australia and South Korea. Article 285 provides for invasion of a computer information system involving the field of state affairs, national defence, construction or science and technology, which places a similar restriction on the provision as exists with the South Korean legislation (UNODC, 2018). However, Article 286 of the PR China penal code provides a much broader set of offences relating too much more general actions against computers and data (UNODC, 2018). The punishment is dependent on serious consequences, which draws a comparison with the requirement in the Australian legislation that the access, modification or impairment be related to the commission of a serious offence. Republic of Korea In the Republic of Korea the criminal provisions are narrow. The first provision is restricted to fraud committed through the use of a computer (similar to the United States of America, see below) (UNODC, 2018). The other relevant provisions are broad in terms of the actions it applies to, but limited in that the provisions are restricted to actions against critical information and communications infrastructure (UNODC, 2018). This raises questions about how, from a legal perspective, actions against AI would be covered in the private sector, or in other areas that are not designated as critical information and communications infrastructure. South Africa South Africa’s general provision (s86) is focused on the access, interception and interference with data, without reference to a computer (UNODC, 2018). However, it is likely that the provision would 62 Electronic copy available at: https://ssrn.com/abstract=3407779
cover the same types of behaviour as the other general provisions in other countries’ legislation. South Africa’s legislation also contains an offence relating specifically to a crime type, being fraud, extortion and bribery that are computer related (s87) (UNODC, 2018). This provision is framed in terms of the actions under s86 as committed for the purpose of fraud, extortion or bribery, and so operates in a narrower field than other offences relating to actions for the purpose of the commission of a crime. It limits the motives for which actions against AI would be captured under s87, but in combination with s86, the overall ability of South Africa’s legislation to cover action against AI is relatively broad. United States of America Legislation in the United States of America does not contain general offences relating to the unlawful access, modification or impairment of data. The legislation is instead framed in terms of particular crimes, namely fraud, and the use of computers or access devices to commit those crimes (UNODC, 2018). The United States do however have a separate offence for unlawful access to stored communication for the purpose of obtaining, altering or preventing authorised access to wire or electronic communication (UNODC, 2018). This offence is broader than the provisions relating to fraud, but still has a narrower scope than some legislation in other countries relating to access, modification or impairment. The fraud provisions would apply to AI as a computer, and depending on what it is programmed to do, the unlawful access provision may also apply. However, the scope of protection offered by these provisions in relation to AI as a target of crime is narrow. The offences are constructed on the basis of the motive for the action, which limits their applicability to these specific factual situations. Actions against AI which may cause damage or do harm, may not be covered by these offences if the motive falls outside those included in the offences. 63 Electronic copy available at: https://ssrn.com/abstract=3407779