What is Shadow AI
Shadow AI is the use of AI within an organization without the
knowledge of or oversight from the IT or Compliance
Department. Within Shadow AI, there are two categories of
concern that can be particularly relevant:
Hidden Internally Built Models: Updating existing models for use cases such as
automation internally or external interactions with customers in undetected ways
(updating a credit risk model using an OpenAI model).
AI in 3rd party applications in models: Having AI within 3rd party applications or
models upon installation or update without the knowledge of the firm using the
application.
Unauthorized Internal Use of 3rd Party AI Tools: Using Generative AI tools such as Chat
GPT to write code, get answers to questions, etc. without being careful about data &
privacy leaks, sharing proprietary code, or lack of proper governance.
Why is Shadow AI so Dangerous?
What makes these models especially dangerous is that firm’s don’t know what they don’t know,
making addressing this risk particularly difficult without consistent and robust monitoring systems.
Here are some specific examples of when Shadow AI posed a significant risk to organizations:
Data Breaches: A notable e-commerce company suffered a significant data breach when an
employee made use of an unauthorized AI product in order to optimize customer data
analysis. This tool lacked the proper security measures and this resulted in the leak of
customer information.
Biased Decision Making: An unapproved AI algorithm was found to be biased against
certain demographic groups, which led to unfair lending practices. This resulted in a
significant regulatory penalty to the organization.
Operational Failure: Major production delays and losses were incurred to a manufacturing
company when an AI system without proper monitoring started making error-prone
predictions for predictive equipment maintenance.
Regulatory Landscape
In addition, these models specifically fall under the supervision of several current & likely upcoming
regulations and therefore pose additional risk of regulatory penalties.
SS 1/23: This Supervisory Statement from the PRA goes into effect May 17th and sets the expectations for
banks and financial firms that operate within the UK. SS1/23 Principle 2.6 Use of externally developed models,
third-party vendor products. Firms should:(i) satisfy themselves that the vendor models have been validated to
the same standards as their own internal MRM expectations.
The AI Risk Management Framework (U.S.): Released by NIST from the U.S. Department for Commerce on
January 26, 2023, this framework guides organizations on how to govern, map, and measure risk to the
organization, including 3rd party shadow AI risk. NIST GOVERN 6.1: Policies and procedures are in place that
address AI risks associated with third-party entities, including risks of in- fringement of a third-party’s
intellectual property or other rights.
The E.U. AI Act: This legislation passed by the E.U. more broadly regulates the use of AI within firms that may
directly impact the safety and well being of the public and holds firms accountable for errors or poor practices
that lead to public harm.
The Artificial Intelligence and Data Act (Canada): Sets the expectations for the use of AI within Canada in order
to protect the interests of the public and require that appropriate measures be put in place to identify, assess,
and mitigate risks of harm or biased output. 3rd party vendors that pose a risk to creating bias or harm within
models are likely included within the risk mentioned within the regulation.
Hidden Use of AI or GenAI within an Organization
Understanding the use of AI internally without the proper knowledge or oversight of the appropriate teams is a
significant step in addressing the risk from Shadow AI. The following are strategies to mitigate this risk:
Consistent Monitoring for Undetected AI Models: Periodically scheduled scans that detect the
probability of the use of AI within Models & EUCs can uncover risks before they result in errors and help
meet regulatory requirements.
Comprehensive AI Testing Suite: Implementing a comprehensive AI testing suite is crucial for detecting
and controlling Shadow AI. This suite should include tests for data drift, validity, reliability, fairness,
interpretability, and code quality. Consistent documentation of test results in a standardized format helps
maintain transparency and accountability for AI models that are detected.
Large Language Models (LLMs) Vulnerability Testing: Testing LLMs for vulnerabilities such as bias, fairness, harmful
content generation, and revealing sensitive information helps stress test a model before it’s used by customers.
Explainable Large Language Models (LLMs): Content Attribution can help explain where within internal data
sources the responses for prompts are coming from, helping to identify and mitigate causes of errors or the
dissemination of incorrect information.
LLM Hallucination Testing: New research suggests that hallucination rates for LLMs may be higher than initially
expected. As competitors race to adopt this technology and leverage it to enhance the customer experience, it
can be critical to leverage the latest developments in RAG models and Challenger LLMs to monitor rates of LLMs
giving customers incorrect information, or Hallucination Rates.
Implementing Controls and Accountability Measures: Controlling the use of Shadow AI involves managing
access to End User Computing (EUC) models and tools. Implementing an Audit Trail to track model changes and
Approval Workflows to ensure accountability can help mitigate risks associated with Shadow AI.
Identifying AI in 3rd Party Applications
AI adoption in financial services has surged 2.5x from 2017 to 2022, per McKinsey, and continues to rise. As thirdparty vendors increasingly embrace AI, the risk of its use without firms' knowledge also grows. Key mitigation
strategies include:
Identifying AI Models within 3rd Party Applications: Monitoring the behavior of 3rd party tools and
executables and looking for patterns that may be indicative of the use of AI can be a necessary way to
identify hidden risk of shadow AI. Consistent scheduled scans to identify and look for this risk can be a
great way to mitigate this risk.
Interdependency Map: A model’s level of risk is highly dependent on the models and data sources that serve as
inputs to that model. With an interdependency map, you can easily visualize these relationships and
interdependencies. Paying special attention to 3rd Party Models that feed into high impact models can help
prioritize where to look for shadow AI.
Security Vulnerabilities: Even if firms are aware of the use of AI within a 3rd party, it can be important to
automate checks for security vulnerabilities within AI 3rd party libraries.
Monitor 3rd Party Model Performance: Many of these 3rd party models are black boxes and here the risk of
shadow AI is highest as firms do not know what techniques a 3rd party vendor is using. Monitoring 3rd party
models for sudden changes in performance can be an indicator for the use of shadow AI.
Monitoring Improper Use of 3rd Party Generative AI
Unregulated reliance on tools such as Chat GPT or Microsoft Co-Pilot can lead to accidents such as
how, at a major technology company, code was leaked to Open AI through the use of the company’s
LLMS. Effective Risk Management of this unauthorized use could involve the following:
GenAI Detection Reporting: Scanning your landscape of EUCs and Models
through cutting-edge AI detection algorithms can help get a better sense of
the overall risk profile of your ecosystem in terms of inadvertent uploads into
AI generators.
Securing Proprietary Code: Within code repositories, flagging the use of Generative AI
can help uncover risks of leaking proprietary code to 3rd parties.
Flagging Hallucination: Running reports through AI detection can help identify
documents that might suffer from errors due to hallucination from LLMs.
Demonstrating Governance and Compliance: Regulations such as The EU AI Act and SS1/23 are
just the start among regulations requiring the documentation and enforcement of policy regarding
the internal use of Generative AI within an organization.
Effective Shadow AI Risk Management
According to The Economist, 77% of bankers report that AI will be the key differentiator between
winning and losing banks so avoiding the use of AI is not impossible.
Shadow AI can be a tough challenge for organizations to face, but with the right level of proactive
monitoring, firms can unleash the massive benefits of AI, and especially GenAI, while limiting the risk.
This involves effectively monitoring the risk from hidden AI models being used within
the organization, the AI within 3rd party applications, and the submission of
information into 3rd party AI Generators.
About Us
Established in 1988, CIMCON Software, LLC is a pioneer in end-user computing and model risk
management, serving over 800 companies across industries.
Recognized by Gartner, Insurance ERM, and others as a top risk management vendor, CIMCON brings 25+
years of experience and industry best practices to support AI & GenAI readiness and governance.
With the largest global installed base, our feature-rich, extensively tested solutions offer unmatched
depth, support, and reliability.
Contact Us
Boston (Corporate Office)
+1 (978) 692-9868
234 Littleton Road
Westford, MA 01886, USA
New York
+1 (978) 496 7230
394 Broadway
New York, NY 10013