Image: Shutterstock (author modified)
It seems everyday new applications and new threats emerge from the AI world. This applies in particular to creators who see growing AI challenges to their livelihoods; graphic art and album covers spat out by AI generators; voice actors replaced by AI clones; authors struggling to make their works known in a sea of AI-generated slop; now AI artists are even making the Billboard charts. At the same time, AI has many other functions and produces a host of products that have little to do with artistic creation. In particular, it can be used as a crutch to assist and enable research in a wide range of fields. Today it is routinely used by everyone from school kids to law firms to health care researchers. And that is where the risks of mainlining AI are the most evident, because of the propensity of AI platforms to fabricate plausible sounding misinformation.
In a blog post earlier this year (AI’s Habit of Information Fabrication (“Hallucination”): Where’s the Human Factor?) I discussed some examples of law firms caught submitting non-existent case precedents in court as a result of sloppy legal research using AI. Judges have very limited tolerance for this practice, which wastes valuable court time, and they are increasingly imposing significant penalties—that is, if the fabricated information is actually spotted. The problem is not going away. This website maintained by Paris-based legal scholar Damien Charlotin has compiled a database of more than 550 legal cases in 25 countries where generative AI has produced hallucinated content. These are typically fake citations, but also include other types of AI-generated arguments. The US wins the lottery at 373 cases, but Canada is second with 39. Even Papua-New Guinea has one case.
As you can well imagine, AI hallucinated results in health care could be fatal. As Dr. Peter Bonis, Chief Medical Officer at Wolters Kluwer Health points out, hallucination in the health care field has led to various consequences such as recommending surgery when it was not needed, advising that a specific drug could be safely stopped abruptly when this was known to be dangerous, avoiding recommending vaccinations based on known allergies even though it was safe to do so, proposing wrong starting treatments for patients with rheumatoid arthritis and so on. You get the picture. You don’t want your family doc using AI search for the remedy for whatever ails you. The fact that the models present incorrect information with such confidence, and that potentially dangerous incorrect information is embedded with a lot of correct information, makes proper use of AI outputs particularly challenging.
How is it that AI platforms consistently produce unreliable results? This MIT Sloan article identifies three elements;
- Training data sources (the uneven quality of inputs, including pirated, biased and otherwise unreliable content)
- Limitations of generative models (generative AI models are designed to predict the next word or sequence based on observed patterns and to generate plausible content, not to verify its accuracy)
- Inherent Challenges in AI Design (The technology isn’t designed to differentiate between what’s true and what’s not true)
This is all pretty concerning if people are going to surrender personal judgement to AI and use it to cut corners without verification. One way to address part of the problem is to ensure the training data used is reliable and of high quality. That is where licensing of accurate, curated data and content as training inputs is important and that is why a licensing market is developing as AI companies seek out better quality data to distinguish their product from that of their competitors. This can be very helpful where the AI platform is limited to discrete areas of knowledge, such as in the medical field for example, where usage can be limited to professionals who are prepared to pay for a bespoke AI product and who are qualified to interpret the results properly. AI for the general public is another matter, and this is where most of the problems arise. Unfortunately, while improving the quality of training data helps reduce hallucinations, it does not completely eliminate them. As the New York Times has reported,
“Because the internet is filled with untruthful information, the technology learns to repeat the same untruths. And sometimes the chatbots make things up. They produce new text, combining billions of patterns in unexpected ways. This means even if they learned solely from text that is accurate, they may still generate something that is not.”
User beware. Nonetheless, better inputs lead to better outputs. As AI developers work to take their products to the next level by refining their training processes and making outputs more predictable and trustworthy, they will need access to curated, proprietorial content and closer collaboration with content owners. Dr. Bonis noted that for specialized areas like health care, AI companies will get better quality feedstock while creators of the content will receive funding allowing them to continue research. A virtuous circle.
Users bear a big responsibility to ensure AI is employed effectively. The mindless, unjudgemental use of AI to reach conclusions in areas where the user has little knowledge can be dangerous. By all means use AI as a tool to sort and categorize, but don’t rely on it to produce the answers on which substantive decisions will be based. Any sensible user of AI has a pretty good idea of the answer to the question before it is even asked. It is also a good idea to refine the question, so you narrow the range of possibilities.
Some proprietary AI models offer RAG (Retrieval Augmented Generation) where the AI will retrieve relevant information from trusted sources to supplement its preliminary analysis. This can increase reliability. However RAG, where the AI goes after specific inputs to bolster its results, can also expose AI developers to charges of copyright infringement, as is currently the case with Canadian AI company, Cohere, which is being sued by a number of newspaper publishers, including the Toronto Star, for copyright infringement. As Canadian lawyer Barry Sookman has pointed out in a recent blog, use of RAG can create risk for the AI platform. In the case of Cohere, when its RAG feature was switched on, it reproduced large amounts of almost verbatim text pulled directly from the litigating news sources. But if the RAG function was switched off, it produced fabricated information (hallucinations) yet still identified this false information as coming from an identified reliable news source, leading to charges of trademark dilution. The value of the brand was diminished by the attribution of false information to it. This trademark dilution issue is also part of the New York Times case against OpenAI.
At the end of the day, it is a case of user beware as a recent case in Newfoundland demonstrates well. The Government of Newfoundland commissioned an in-depth study on the future of education in the province. The 410 page report containing over 110 recommendations, authored by two university professors, was released with great fanfare at the end of August. No doubt a great deal of careful research had gone into producing the study over the 18-month production period. But then cracks started appearing in the edifice. It was chock full of made-up citations. The more people started checking, the more they found. The Department of Education and Early Childhood Development tried to whitewash the issue by saying it was aware of a “small number of potential errors in citations” in the report. But even one fabricated citation is one too many! If you search for the report online now you get the classic “404 Not Found” message. A lot of work has potentially gone down the drain, and possibly the credibility of two academics has been destroyed by careless use of AI. This is a cautionary tale that I have no doubt will be repeated.
In fact, it was repeated just a few days later. It seems Newfoundland is particularly prone to victimization by hallucinating AI platforms. After the education report debacle, new reports have surfaced that a $1.5 million study on the health care system conducted by none other than Deloitte also contains fabricated information included made up references. The opposition party is demanding the government insist on a refund.
In our rush to embrace AI, many seem to have forgotten the value of human creativity and judgement. Coming back to the creative industries and AI, some of those whose livelihoods may be threatened by this new phenomenon are bravely trying to find a silver lining. Some voice actors are generating an additional revenue stream by licensing their voice clips for AI training, and many graphic artists use AI as an assist. Are they putting themselves out of work in the long run or are they simply adapting? The jury is still out, but the generally low quality of AI produced art, music and literature, as well as the ongoing problem of hallucination, suggests that there will always be a need for real human input. Anyone planning on substituting AI for “real work” had better think again.
© Hugh Stephens, 2025. All Rights Reserved.
