Prompt Attacks: Crafting adversarial prompts that allow an adversary to influence the behavior of the model and, hence, the output, in ways that were not intended by the application.

Prompt injections that are invisible to victims and change the state of the victim’s account or any of their assets.

Prompt injections into any tools in which the response is used to make decisions that directly affect victim users.

Prompt or preamble extraction in which a user is able to extract the initial prompt used to prime the model only when sensitive information is present in the extracted preamble.

Using a product to generate violative, misleading, or factually incorrect content in your own session: e.g, “jailbreaks.” This includes “hallucinations” and factually inaccurate responses. Google’s generative AI products already have a dedicated reporting channel for these types of content issues.

Out of scope

Training Data Extraction: Attacks that are able to successfully reconstruct verbatim training examples that contain sensitive information. Also called membership inference.

Training data extraction that reconstructs items used in the training data set that leak sensitive, non-public information.

Extraction that reconstructs non-sensitive/public information.

Out of scope

Manipulating Models: An attacker able to covertly change the behavior of a model such that they can trigger pre-defined adversarial behaviors.

Adversarial output or behavior that an attacker can reliably trigger via specific input in a model owned and operated by Google (“backdoors”). Only in scope when a model’s output is used to change the state of a victim’s account or data. 

Attacks in which an attacker manipulates the training data of the model to influence the model’s output in a victim’s session according to the attacker’s preference. Only in scope when a model’s output is used to change the state of a victim’s account or data.

Adversarial Perturbation: Inputs that are provided to a model that results in a deterministic, but highly unexpected output from the model.

Contexts in which an adversary can reliably trigger a misclassification in a security control that can be abused for malicious use or adversarial gain.

Contexts in which a model’s incorrect output or classification does not pose a compelling attack scenario or feasible path to Google or user harm.

Out of scope

Model Theft/Exfiltration: AI models often include sensitive intellectual property, so we place a high priority on protecting these assets. Exfiltration attacks allow attackers to steal details about a model such as its architecture or weights.

Attacks in which the exact architecture or weights of a confidential/proprietary model are extracted.

Attacks in which the architecture and weights are not extracted precisely, or when they’re extracted from a non-confidential model.

Out of scope

If you find a flaw in an AI-powered tool other than what is listed above, you can still submit, provided that it meets the qualifications listed on our program page.

A bug or behavior that clearly meets our qualifications for a valid security or abuse issue.

Using an AI product to do something potentially harmful that is already possible with other tools. For example, finding a vulnerability in open source software (already possible using publicly available static analysis tools) and producing the answer to a harmful question when the answer is already available online.

Out of scope

As consistent with our program, issues that we already know about are not eligible for reward.

Out of scope

Potential copyright issues — findings in which products return content appearing to be copyright protected. Google’s generative AI products already have a dedicated reporting channel for these types of content issues.

Out of scope