This article was first published in Fire and Security Matters Magazine July 2020: https://www.fsmatters.com/Artificial-Intelligence-The-Transparency-Challenge
Why should we challenge the transparency of AI?
A powerful new general purpose AI technology has emerged from the computer science community. GPT 3.0 contains over 1000 times more learning reference points than its predecessor, resulting in a quantum leap in capability.
The beneficial application this new AI engine could improve the analysis of millions of lines of text in legal documents, medical documents and other academic papers in order to extract meaning and in doing so accelerate human decision making. This could lead to a faster route to a cure for infectious diseases such as COVID, or help to recommend patient specific treatments.
Natural language processing (NLP) may seem fairly straight forward when compared with the AIs needed in the security domain. The structure of language is generally well defined so “all” the algorithm has to do is interpret the form of the characters (using convolutional neural networks/image processing) and assimilate words assuming the data is already in a readable format. It is the context which presents the greatest challenge to any neural network. (NLP uses recurrent neural networks (RNN)). Every time you use predictive text on your phone you are interacting with a form of NLP. But does it always get it right? Usually you have to decide which option is most appropriate. The human, knowing the intended meaning, makes the final decision.
With a brief from a human, GPT 3.0 may be able to create factual text which is indiscernible from human writings. This may alarm those who enjoy spending hours assimilating research data. In stark contrast, replicating the human traits of ideation and creative flair caused by the unique way we each experience life, is thought to be beyond the capability of such technology.
AI terminology may sound complex to those unfamiliar with the nuances of computer science, nevertheless, business leaders are still accountable for the ethical application of AI.
Beneficial AI in the physical security domain
In security, things are not so “easy” for the AI as they seem with NLP. This is due to the inconsistent and changing nature of the environments, the random activities of malevolent actors and the multitude of available data sources from which meaning may be extracted.
The smart city, smart building and smart public place all generate billions of bytes of data which can be analysed automatically by AI. Autonomous IoT building sensor technologies are proliferating, and although legacy systems create drag in the convergence process there is an increasing ability to convert protocols and transcode unstructured data resulting in a harmonised data pool suitable for AI analysis. After all this manipulation for accessibility, it is quite possible that the AIs may be analysing data which is no longer in its original format.
What is transparency?
The purpose of AI in the security domain is simple. It is to enable better human decision making resulting in fast and relevant action to protect people, protect property and reduce loss.
Transparency is one of the building blocks of trust. In the context of security decisions, we need to understand and trust how the AI reached its recommendation. Adoption then follows trust. In a constantly moving environment it is somewhat difficult to test the replication of an AI decision when detecting a one off threat. Where did the AI’s training data come from in this case?
The use of AI must be tested in the context of the use case because scenarios which could not be thought of by the developers due to narrow experience, may arise which are new to the AI. This may seem obvious, but you wouldn’t drop an untrained person into a security and risk environment without contextual and professional training.
On the subject of training, for video analytics to work effectively across a range of scenarios, it has to learn from millions of classified images gathered from many sources. People are invited to contribute to open source projects by looking at images describing what is seen. Emerging AIs can now learn from their own learnings which means that they can recognise familiar objects in a changing scene over time. This new technology reduces the need to label every instance of a type of object in an image and also reduces the anomalies which creep into AI if the background is inadvertently classified too. For example, if a boat and a blue sky become associated because all classified images of boats also have a blue sky in the background, only a boat with a blue sky will be recognised. Boats without blue sky will not be recognised as boats. Some AIs are really that raw and fundamentally untrained.
It is important to understand the distinction between the AI engine and what is does, and the context of the application, in essence, what it is being used for.
If leaders were not using AI we would define the goal and create a process journey from start to finish, usually with traceability throughout.
Transparency is one of the fundamental elements of supply chain management. Component traceability in electronics manufacture is vital for quality control and warranty. Every piece of data is tagged in some way to identify it and its source. (Blockchain is emerging in supply chain verification but does not provide the answer in the security intelligence environment due to the risk of false data being introduced which then becomes the “truth”.)
If you can trace the components in a manufactured electronic device back to their source, it is logical that an AI decision should also be traceable back to the original data sources. It should therefore be possible to determine whether the source data, which may be the prima facie evidence in a legal case, has been altered. This may be beyond current technology, however, when a data problem is clearly defined, a solution is usually found.
It could be argued that generic open source AIs which are released to the world for experimentation, become prone to bias due to the fact that the specific applications are not defined and therefore the training data sources are uncontrolled.
This has happened in practice recently when automatic facial recognition technology was released into law enforcement use in the USA. It was discovered that the AI had been developed using a narrow demographic and started to show inaccuracy and bias in its recommendations. This can be corrected by widening the data set. If it had not been highlighted through this use case, some of the endemic biases in the organisations may have been left unchallenged too. AI may help to remove bias.
Why are transparency and explainability so important in the physical security industry?
The industry generates more data than it currently utilises. Even so, AI is focused around generating data about the data. This is because the unstructured data produced by video surveillance cameras is impossible to analyse alongside other relevant data sources without some kind of change of format from the original. Integrations which store alarm data in the image header aid the text search for video associated with an alarm but do not tell the analyst what appeared in the video. Every image has to be reviewed by a human. This is time consuming and inefficient.
AIs in the form of video analytics (convolutional neural networks) provide effective filter and search functions for video data. The search results can trigger the start point for another analytical AI process such as picking up a related social media tweet. Sadly, not only do those whom mean harm tweet their intent, members of the public seem to believe that tweeting about a serious incident may be a good enough substitute for calling the emergency services (the socially responsible thing to do). What is buzzing on the social media networks could be highly relevant to an incident captured on a video surveillance camera.
The source data and the resulting decision can become far removed and much harder to trace back to the original. The digital evidence bag may contain multiple data formats gathered from disparate sources and may be full of untraceable data, potentially reducing its value in the criminal justice system.
Verifying the validity of digital forensic evidence is becoming an ever-increasing problem, even more so now than it was in the 2000s when digital recording took off as a means of storing video.
Without challenging the roots of the AIs embedded in security systems, we may inadvertently propagate flaws right through to the final decisions. There is a school of thought in leadership which suggests that any decision is better than no decision. But when the decision is about life safety, security and respect for the privacy and dignity of the individual, and is based on a biased process it is worse than no decision due to the potential harm it may cause.
Decisions based on bias still occur every day in society, but with the increased level of awareness, now is a good time to scrutinise the AIs used in security decision making and ensure that there is a continuing education and a drive to make things better.
To challenge AI, leaders have to gain a deeper understanding of how these technologies work. This is not a matter for the data science department in isolation, it is a Board level consideration.
Robust measures need to be put in place to enable traceability back to source to verify whether the data has been altered along the way. It is only a matter of time before a deep fake video or audio file carries weight in a prosecution. Let’s ensure that AI does not magnify injustice and instead propagates fairness.
We need AI which is free from biased training data. We need AI whose decisions can be replicated and the type and use of AI to be understood and transparent. Where the source data and the decision become dissociated in some way lets be mindful of this and apply AI in a way which allows the source data to be identified and verified. Most of all AI should be applied to achieve the maximum benefit, ethically, within the legal, regulatory and standards frameworks and without causing harm. This is why we need to challenge the transparency of AI.
Or visit our downloads page, and access your copy.
Would you like to understand more about AI and how it can benefit society, industry and solve your business problem?