February 28, 2019
Big tech companies, with access to huge amounts of data, have driven a renewed attention for machine learning. As more and more types of tasks are being successfully tackled by machine learning algorithms, we need to re-examine the question of how to safely and securely harness their power. As machine-learning components introduce uncertainty, how can the systems that use these components be relied upon?
Written by Alexandru C. Serban, Phd Candidate in the Digital Security group at Radboud University, hosted at Software Improvement Group (SIG) and Joost Visser (photo) Chief Product Officer at SIG and Professor of Large-Scale Software Systems at Radboud University, former member Advisory Board dcypher.
One of the research angles to safe use of AI and ML are the software architecture patterns that are being investigated in a collaboration between Radboud University, Software Improvement Group, and Technical University of Eindhoven. These patterns allow software system designers to assure the safety properties of systems by buffering, mitigating, channelling and reasoning about the uncertainty introduced when using ML components. During the past decades software quickly evolved in the DNA of our information society. It is sometimes easy to forget how deeply embedded in our lives software is. From helping us keep in touch with our friends to running vital business processes and enterprises, software goes mostly unheeded. In fact, it is harder to find a process which was not touched by software. Our transportation systems, supply chains, water management, the energy grid and even our social relationships and habits heavily rely on software.
And it's not hard to say why. From their inception, computers and software systems were thought as universal machines. Instead of building a dedicated machine for each task, we can use the same machine to execute a broad range of tasks. All it is needed is to encode a set of instructions, specific to each task, and execute them on a computer. This process runs in a deterministic fashion: computers will execute the instructions in the order they were asked to, encoded by software developers. While this paradigm has proven prolific in solving a wide range of tasks, we often encounter tasks which cannot be encoded in a set of instructions. For example, recognizing objects, understanding speech or driving on highways are tasks which our brains learn quickly and execute autonomously. However, it is hard to describe what happens when we recognize a cat in a picture or decide to execute a driving manoeuvre. It is even harder to encode these processes in a set of instructions and run them on computers.
However, not being able to deterministically encode complex tasks in a set of instructions did not stop computer scientists from trying to run them on computers. In the end, computers are designed to be universal machines, therefore, able to run any task. With a lack of instructions, researchers have made a change of paradigm: instead of specifying which instructions to execute, computers learn to execute a task by looking at data describing it. This process, known as statistical or machine learning, maximizes the computer’s performance on a task by looking at past experience of it. The main ideas underlying modern machine learning have been developed decades ago. However, for a long time, the lack of data and computational power llowed only small progress on complicated tasks, such as object or speech recognition. With the wide adoption of software in our society, which allowed collection of vast amounts of data, and the increase in computational power, both problems diminished and allowed machine learning to make a come-back. Big tech companies, with business models centred around data and computing, have triggered a new wave of innovation by applying machine learning to a broad range of tasks.
A particular algorithm, inspired by the way the human brain works, seems to perform well on many tasks. Artificial neural networks - a mechanism to simulate the synapses in the human brain have been successfully adapted to process diverse data such as images, text orsounds and achieve state-of-the art results in diverse settings. However, the huge complexity of these systems is often hard to grasp by human actors. In many cases, researchers can only analyse the inputs and the outputs, with little reasoning capacity about the inner workings of the algorithms.
In the narrow sense, the recent success of machine learning allows us to build even more specialized software and automate new tasks. For example, machine learning is explored in building autonomous manufacturing robots, self-driving vehicles, diagnose and understand diseases, design new materials, find new particles, drugs and even stars. In the wider sense, computer scientists dream about building a computer that learns to think and develop intelligence, possible outsmarting human intelligence. Facing commercial deployment in systems with large impact on the society, such as the transportation or medical systems, it is crucial to think about safety and security aspects of machine learning. At first, it is important to make sure people do not misuse such technologies in creating more sophisticated security attacks. Secondly, it is important to design algorithms able to withstand malicious attacks and avoid unintended or harmful behaviour. In this light, new properties emerge. In our research, we look at ways to cope with the uncertainty intrinsic to complex machine learning algorithms. Particularly, complex algorithms seem to perform poorly when faced with slightly different data than trained with. Only small perturbations of the inputs can lead to instability and decreases in performance. The property to cope with changes in the operational environment without suffering any modification has been studied previously studied in software engineering. However, the change from deterministic to probabilistic software raises new interesting challenges.
Theoretical results suggest that more resources are needed to train robust machine learning algorithms. However, these resources are often scarce. We tackle this problem from two different angles: algorithm and software design. At first, we are concerned with designing algorithms able to learn robust models with low resources. Secondly, we investigate how to design software systems able to manage uncertainty by design. We look for solutions in the field of software and system architecture which can enable fast deployment of components which large intrinsic uncertainty, while minimizing their risk. Moreover, they bring closer a wider audience, ranging from machine learning to software designers and engineers. The development of intelligent systems will, most probably, be a multi-disciplinary process. Bringing together people with different backgrounds and offering the right frameworks for communication and reasoning will lead to more robust, safe and secure solutions.