To defend your castle, you must first understand what makes it: the walls, gates, towers, and personnel. You must know what passes through each of the gates, and which weapons can break the walls. Once you see and understand each component, and how it contributes to the whole, you can establish the risks and vulnerabilities attached to each part.
Playing cyber defense for AI is no different: the best defense is built on a thorough understanding of every component, vulnerability and potential entry point within your AI system – a Bill of Materials (AIBoM).
What’s AIBoM?
AIBoM is a comprehensive inventory of the components that comprise an AI system, including hardware, software, data and pipeline. AIBoM is to AI what SBoM is to software applications. It’s essential for the supply chain because it enables organizations to identify and mitigate potential risks and vulnerabilities in AI systems.
Rapid evolution, lack of transparency and accountability, and the potential for malicious use have raised concerns about the security of AI systems. The AI Bill of Materials (AIBoM) is a vital tool in addressing these concerns and driving better understanding and mitigation of AI attack surfaces.
The Sum of Many Parts: The Scope of AIBoM
In Formula 1, the pinnacle of motorsports racing, several teams that are engine manufacturers let competing teams use their engines. For example, Ferrari’s engines are used by teams such as Haas & Sauber. The Haas F1 car may have the same engine, fine tuned to their liking, placed in a chassis and drivetrain built by Haas themselves.
Both the engine and final F1 car are the sum of their components. The engine has its Turbo and MGUs while the F1 car has its brakes, crankshaft AND the engine.
With modern AI, we can think of Machine Learning models such as LLMs In the same way: engines that form the core of AI Applications. There are manufactures such as OpenAI, Meta or Mistral who provide the LLMs for organizations to integrate into their AI Application. Both LLMs and the final AI Application are a sum of parts, where the LLM also forms a part of the end-to-end AI application.
The scope of AIBoM is therefore both the Machine Learning models comprising libraries, datasets, arxivs and classifications, as well as the end-to-end AI System made up of ML/LLMs, private datasets, APIs and AI agents.
Given the public visibility of machine learning models, LLMs or SLMs – it’s crucial to define their AIBoM first – because that same visibility is also gifted to attackers.
Defining and Generating AIBoM for LLMs
Hugging Face
Hugging Face is a leading platform for Machine Learning models and tools, offering an extensive library of pre-trained models and datasets. Hugging Face Model Cards provide crucial documentation for each model, detailing its intended use, performance metrics, training data, limitations, and ethical considerations, ensuring transparency and informed usage.
Model cards seem like a great place to start constructing AIBoM across the largest machine learning hub. However, while model cards do well to identify datasets and chosen few libraries, a majority of the model “tags” are unclassified.
Our experts fired up machine learning weaponry to create meaningful groups of unclassified tags by collecting and correlating tags from ~500k Models. We were able to further categorize information such as base models:
- Libraries: Collections of pre-built functions and tools that facilitate the development, training, and deployment of machine learning models.
- Datasets: Structured collections of data used to train, validate and test machine learning models.
- Arxiv: An open-access repository where researchers share preprint versions of their academic papers, including cutting-edge research on machine learning.
- Licenses: Legal frameworks that define the terms under which machine learning models, datasets, and software can be used, modified, and distributed.
- Task Categories: Groupings of machine learning problems, such as classification, regression, or clustering, that define the type of output a model is expected to produce.
- Base Model(s): Pre-trained models that serve as starting points for further fine-tuning or transfer learning on specific tasks, often providing a foundation for building more complex models.
- Known LLM Vulnerabilities & Exposures (LVEs)>
The LVE Project
As LLMs have boomed, so has the attack surface vulnerabilities and exposures specific to LLMs, such as prompt injection attacks, prompt leakage or PII leakage. Unlike the CVE project which serves as a repository for known vulnerabilities in software applications, hardware and operating systems, there is no known standard repository for LLMs with similar purpose.
The LVE Project aims to bridge this gap by inviting open source contributions that aid in documenting known LLM vulnerabilities & exposures under five categories:
- Privacy
- Reliability
- Responsibility
- Security
- Trust
The significance of this project cannot be overstated. Our researchers incorporated known LVEs into the AIBoMs generated for LLMs, establishing a pipeline that processes the LVE repository, identifies subject LLM and maps it back to the Hugging Face model reference, providing invaluable visibility to known LLM vulnerabilities and exposures.
What’s Next?
Vulnerabilities like Log4j underlined the importance of visibility and transparency in software supply chains. As AI models evolve at a rapid pace, it’s crucial that defenders have the visibility and transparency they need to manage vulnerabilities and bring a risk perspective to the new reality. Transparency builds trust, and AIBoM plays a significant role in creating that transparency.
In the next two posts in this series, we’ll build on the concept, covering:
AI/ML Libraries & Vulnerabilities – With over 3000+ libraries identified across Huggingface within the AI/ML supply chain through our project, we take a closer look at the vulnerabilities in popular libraries and uncover key vulnerability intelligence insights!
Part 3:
The AI Attack Surface – MITRE ATLAS, OWASP Top 10 for LLMs & ML … how does it all fit over the AI Bill of Materials, our research provides a comprehensive AI Attack Surface overview neatly tying together the frameworks, vulnerabilities and AIBoM components.