Navigating AI in the Workplace: Understanding AI Risk for Employers
As artificial intelligence continues to permeate various sectors, understanding and managing its associated risks becomes paramount. Whether you have an AI Use Policy in place or not, there is a good chance your employees are using generative AI in their day-to-day work. Here, we provide an overview of the most pressing risks associated with generative AI, supplemented by real-world examples and additional insights.
1. Confabulation (Hallucination)
Generative AI models can produce outputs that are factually incorrect or nonsensical, a phenomenon known as "hallucination." This issue arises because these models predict text based on patterns in data rather than verifying facts. Depending on the subject matter and the AI model being used, a "hallucination" can manifest as a factual error or inaccurate analysis of summarized text. In more extreme cases, generative AI models can create sources, complete with citations, to support their analysis.
Risk: Dissemination of false information. You may be held liable when customers make purchasing decisions based on inaccurate, misleading, or non-existent data.
Real-World Example: Air Canada was held liable for a misleading response provided by its AI-powered customer service chatbot. A passenger inquired about bereavement fares and was informed by the chatbot that they could apply for a reduced fare retroactively within 90 days of ticket issuance. Relying on this information, the passenger booked a flight and later sought a refund, which the airline denied, stating that their policy did not allow retroactive applications. Air Canada was held responsible for the chatbot's misinformation and awarded the passenger $812.02 in damages and court fees.
2. Data Privacy Violations
Generative AI models commonly infer and retain information from your past conversations to improve future interactions, even if you haven't explicitly asked it to remember those details. Personal data entered into a prompt is stored in memory maintained by the company running the model.
Risk: Unauthorized use of personal data, potential breaches of privacy laws, and erosion of public trust.
Real-World Example: In May 2023, Samsung employees inadvertently leaked confidential information by inputting sensitive internal data into ChatGPT for assistance with code and document reviews. This data was not part of ChatGPT's original training set but became accessible through user interactions. As a result, Samsung banned the use of generative AI tools across the company to prevent future breaches.
Real-World Example: Security researchers discovered a novel attack method named "Imprompter," which enables malicious actors to extract personal data from AI chatbots. By embedding transformed prompts that appear as random characters, attackers can instruct the AI to gather and transmit personal information from user interactions to unauthorized domains. This technique was tested on models like LeChat by Mistral AI and ChatGLM, achieving an almost 80% success rate.
3. Cybersecurity Threats
The capabilities of generative AI can be exploited to enhance cyberattacks, making them more sophisticated and harder to detect.
Risk: Increased vulnerability to phishing, malware, and other cyber threats.
Real-World Example: Prompt Injection Attacks: An attacker embeds malicious instructions into user input or third-party content that a generative AI system interprets as commands, hijacking AI behavior, leaking sensitive data, or executing unintended actions.
Hackers used indirect prompt injection to tamper with Gemini AI’s long term memory, allowing the attackers to plant false information.
Real-World Example: Model Leaking / Model Inversion: Attackers extract sensitive data by probing a generative model (e.g., using carefully crafted queries), which can lead to the exposure of proprietary data, user info, or trade secrets.
Researchers extracted private data from a large language model trained without privacy-preserving algorithms.
Real-World Example: Data Poisoning: Attackers insert malicious or misleading data into the model's training or fine-tuning pipeline to influence behavior, causing models to produce biased, unsafe, or attacker-controlled outputs.
Real-World Example: AI-Powered Vulnerability Discovery: Generative AI can analyze code, configurations, and logs to find exploitable bugs at scale. This could empower attackers to scan for vulnerabilities with greater efficiency.
DARPA's AI Cyber Challenge launched to proactively develop AI tools for securing infrastructure. They acknowledge the dual-use risk of hackers using this technology to exploit bugs.
4. Bias and Discrimination
AI models can perpetuate and even amplify existing societal biases present in their training data, leading to discriminatory outcomes.
Risk: Reinforcement of stereotypes, unfair treatment of individuals or groups, and societal inequities.
Real-World Example: SafeRent Solutions faced a lawsuit after its AI-powered tenant screening tool allegedly assigned disproportionately low scores to Black and Hispanic applicants, leading to housing denials. One plaintiff, despite having a good rental history and a housing voucher, was denied an apartment due to a low AI-generated score. SafeRent settled the case for $2.3 million and agreed to halt the use of such scoring systems for five years.
Real-World Example: Uber has been sued in the UK over allegations that its facial recognition system failed to accurately identify drivers with darker skin tones, leading to wrongful terminations. The lawsuit claims that the AI system's bias resulted in discriminatory practices against people of color.
Real-World Example: Amazon discontinued an AI recruitment tool after discovering it favored male candidates over female ones. The system was trained on resumes submitted over a 10-year period, predominantly from men, leading the AI to downgrade resumes that included the word "women's," as in "women's chess club captain."
5. Supply Chain Vulnerabilities
The integration of third-party AI models and components and code “packages” into software products introduces supply chain risks.
Risk: Compromised data integrity, increased exposure to cyber threats, and operational disruptions.
Real-World Example: The PyTorch machine learning framework experienced a supply chain attack when a malicious package named torchtriton was uploaded to the Python Package Index (PyPI). This package mimicked a legitimate PyTorch dependency and was inadvertently installed by users, allowing attackers to execute arbitrary code on affected systems.
Real-World Example: Over 100 malicious AI models were discovered on Hugging Face, a popular platform for sharing machine learning models. These models were designed to appear legitimate but contained hidden malicious code. Organizations that incorporated these models into their systems faced potential data breaches and unauthorized access, emphasizing the need for rigorous vetting of third-party AI models.
6. Intellectual Property Infringement
AI models can generate content that incorporates copyrighted or trademarked materials without proper authorization, raising legal issues.
Risk: Violation of intellectual property rights and potential legal consequences.
Real-World Example: The New York Times sued OpenAI and Microsoft, claiming that their AI models, including ChatGPT, reproduced substantial portions of its articles without authorization. The lawsuit alleges that users could prompt ChatGPT to generate content nearly identical to Times articles, raising concerns about user liability for distributing such outputs.
Real-World Example: A trend has emerged where users employed AI tools to create images in the distinctive style of Studio Ghibli, a renowned animation studio. While the AI-generated images were not direct copies, they closely resembled the studio's unique aesthetic. This raised legal and ethical concerns about potential infringement on the studio's intellectual property rights, especially if such images were used commercially
Final Thoughts:
The National Institute of Standards and Technology (NIST) AI Risk Management Framework provides a structured approach to identifying and mitigating the risks associated with AI systems. By understanding these risks and implementing appropriate safeguards, organizations can harness the benefits of AI while minimizing potential harms.
