AI – Artificial Ignorance?
· Artificial Intelligence · 7 min read · By Dr Jason I Broch
I've been intrigued by the explosion Artificial Intelligence products on the market, especially in Healthcare. First, I have to say that the technology is impressive. The ability to analyse an image and spot abnormalities suggesting potential cancers can only be a good thing. Even deploying speech recognition into a consultation or online meeting system seems to have scope for improving digital dictation systems and in some settings may help make clinicians more efficient.
I suppose the area I am most apprehensive about is where this technology could be used to change or potentially summarise the information. My thoughts on this do not necessarily just apply to applications in healthcare, but are probably more philosophical about the nature of intelligence itself and the assumptions people make about what a computer may tell them.
In general, people are quite simple beings. We are not very good at being critical of all the information we receive. If something seems plausible, then we will accept it as the working truth, until proved otherwise. We have had to evolve this way to survive, but aren't always aware of the risks in the modern world when we are bombarded with so much information every moment. It's like the Little Britain 'Computer says no!' phenomenon. If we are faced with a suggestion from a trusted device, we will just accept it, alternatively we get pop up fatigue, where if there are too many warnings, then we get used to them and ignore them after a while, with clicking through them becoming almost automated. Another evolutionary trait is to think that others are like us. We see that in the numerous security failures when in retrospect peoples behaviour could have been suspect in planning a terrorist attack, but went unnoticed, as security services or surrounding people did not 'see the signs'. Con artists have long taken advantage of this trait, whether through speaking with confidence about a potential investment or a fake email or phone call to obtain your banking information. Often victims feel this is their fault, but in fact it is part of being human to accept what we are told in general, unless we are really ready for the signs. This same evolutionary trait is also a risk when it comes to generative AI.
Generative AI and the Large Language Models (LLM) that power them have taken the world by storm in the last few years. Most people have played with Open AI Chat GPT, Microsoft Co-Pilot or Google Gemini. They work by using powerful analytic tools on very large amounts of training data to effectively work out the probability of words usually belonging together. Mathematically, each word is mapped in a multidimensional vector space with each dimension representing some type of context. In simple terms, we could think of the word 'Dog' and assume it is stored at a certain position, which can be represented by a series of numbers a bit like a simple (x,y) graph. The complex computers behind LLMs can plot a word using several hundred dimensions (x,y,z,a,b,c,d,e,f…). We can regard each of these letters as a likelihood of something, for example 'f' could be the likelihood of the word being a pet. If we looked at the dimensions for dog and cat, we may find that f,g,h are all the same. If we ask an LLM, 'give me examples of pets', it will look at the words we typed (or 'prompt') and then look for words in a similar space. It could draw on words where f,g,h are the same as 'pet', but the other several hundred dimensions vary slightly. This will all be based on the association of words in the training text. It does not represent any understanding of what a Dog, Cat or Pet is. This is obvious to us, but what if we were looking for peoples 'pet hate'. True it could be coded in the other dimensions, but it could only draw on word associations that were in the training text, not new ideas. Our understanding may be context-based. Take the example of someone asking 'Where is the bank?' The meaning is different if the person is holding a bank card, compared to sitting in a boat.
The point is that using vast amounts of data, LLMs can provide very plausible responses, which has 2 risks, especially in medical applications. First, the plausible response can give the impression of being like us, a person. We anthropomorphise the response and treat it and trust it like we would a human. Second, the plausible response can mask the fact that the LLM has essentially made up a coherent sounding answer based on word association without really understanding what is being asked. This is the phenomenon of 'hallucination'. The AI produces a plausible sounding response, based on word association in training data, which is factually incorrect. Imagine asking a junior doctor for some advice on a medication. They have heard of the medication before and know a bit about it, but are not sure about the doses, side effects or contraindications. Instead of saying they don't know, they just answer with something that sounds plausible. You would probably not trust that junior doctor again, but when it comes to generative AI, would you notice? The system is designed to give plausible responses, so the response will seem reasonable. Unless you have a degree of expertise in the field, it may be hard to spot the hallucination. This is artificial ignorance.
Whilst it is important to recognise the limitations of current technology, especially in a world where there is so much hype, a bigger question may be how can we make sure non-technical people have enough knowledge to be able to appraise a technology well enough to assess its true capability and safety in potential applications such as clinical settings? An interesting consideration may be a technology that is only 60% effective maybe very useful if our current effectiveness is only 40%. Human systems make lots of mistakes or errors in judgement, so an imperfect technology that is less imperfect than us could be a game changer as long as we are comfortable with the risks.
Finally, the hype is demonstrating a desire to have technology that does have a degree of trustworthy intelligence. We recognise the difference between current technology that may mimic humans in a plausible way, but how would a truly intelligent machine differ. This is an area we can explore further in another blog, but I would suggest the starting point would be to start with a theory of intelligence or at least be clear what key features such as demonstrating an understanding of meaning would be.