Secrets never reach the reviewer

A key pasted into a pull request is one of the easiest security bugs to ship, and one of the easiest to find. It does not need judgement: an AWS access key looks like an AWS access key. So before ZeusQA's security review asks a model anything about a diff, it now checks the added lines against fixed patterns, and anything it finds never leaves that step in full.

This post covers what it checks, what a finding looks like, how it keeps the value out of everything it writes, and where it stops.

Patterns first, then the model

The security review reads each new commit's diff and sends it to a model, which looks for problems such as missing tenant scoping, injection or server-side request forgery. Secrets were on that list too, but a model is the wrong tool for them. It can miss a key, it can describe one in its own words, and it has to see the value to report it.

The secret scan runs before the model, with no model at all. It checks every added line of each file GitHub returns a diff for, up to the first 300 files of a pull request and lockfiles left out, for:

• AWS access key IDs and secret access keys
• private keys, in PEM files or inside a one-line JSON credential
• GitHub classic and fine-grained tokens, GitLab tokens, Slack tokens and webhook URLs
• Stripe live secret keys, OpenAI, Anthropic, Google, SendGrid, npm and PyPI keys
• database and queue connection strings that carry a password
• random-looking values assigned to names like secret, token, api_key or password

It also checks files the model skips: generated and vendored files, and files past the size cap of the diff the model reads. A key in a built bundle is still a key. Lockfiles and binary files are left out, and so are files GitHub sends without a diff, which it does for a very large file.

A finding on the line

Each match becomes a finding on the file and line the pull request adds, like every other finding of the review. It names what was found and says how to rotate it: deactivate the key in AWS IAM, revoke the token in GitHub, roll the key in the Stripe dashboard. It also says the part people forget: deleting the line in a later commit does not remove it from git history, so the key has to be rotated either way. Up to 30 matches are listed per commit, with a count of the rest. They do not depend on the model: if the model review fails, the check and the comment still carry them and say the model review failed, so only the secret patterns ran.

A live credential or a private key is critical. A connection string, a Google API key, a webhook URL or a secret-named value is high. The finding sits on the ZeusQA Security check as an annotation on that line, and in the one comment the review keeps up to date on the pull request.

Never the value

The review writes in three places: the check, its annotations and the pull request comment. None of them shows a secret. The quoted line keeps only the part of a key that is not secret, its provider prefix, so an AWS key reads AKIA…[redacted] and a GitHub token ghp_…[redacted]. Passwords and secret-named values show nothing at all. On a long line, such as a minified bundle, the quote is a short window around the match.

Then, before the diff goes to the model, every value the patterns recognise is masked in it, on added lines, removed lines and the unchanged lines around them, and a private key's whole body is replaced. The model is told those lines were masked and are already reported, so it does not report them twice. If it reports a secret on a line the scan already flagged, the scan's finding is the one kept.

Fewer false alarms

Repositories are full of things that look like secrets and are not. The scan lowers or drops these on purpose:

• A match in a test, mock, fixture or example file, such as .env.example or __mocks__/, is rated low, with a note to rotate it anyway if it is real.
• Documented example keys, like the one in AWS's own docs, and connection strings to localhost are rated low.
• Placeholders such as ${DB_PASSWORD}, <token> or your_api_key are not reported.
• A secret-named value is reported only when it looks random: letters and digits, high entropy, and not a UUID, a URL or route, a version string, a regular expression, a Stripe publishable or test key, or words like prod-db-password-v2 or X-Api-Key-V2-Header.

Where it stops

• It finds the formats it knows. A key in a format not on the list is left to the model, which sees it unmasked.
• It reads the diff of each new commit, not the repository's history. A key committed before the review was turned on is not found.
• It does not check whether a key is live. A match is reported on its format alone.
• It blocks a merge only through the repository's merge gate: a secret at the gate's severity (High by default) or above fails the check, and the check has to be required in a branch ruleset.
• It runs inside the security review: on Pro and Team, for repositories with Security review turned on in their GitHub settings, within the review's limits on pricing.

The controls behind this, and the gaps, are listed on the security page. The full release is in the changelog: Secret scanning on every pull request.