GitHub AI Scan found two shell risks, but the security check passed

Today I opened a public test PR #1 containing two deliberately unsafe Bash patterns. Roughly 81 seconds later, github-advanced-security[bot] had left two line-level comments, identifying both risks.

Yet the GitHub Actions run #38835686338 marked its github-advanced-security job Success. Both things happened: the warnings were real, and the check was green.

GitHub had announced the general availability of AI Scan for pull requests the day before this test. The useful question is not whether an AI scanner can flag suspicious code, but why a PR can contain flagged vulnerabilities while its security check succeeds.

The distinction matters: a successful AI Scan job does not certify that the code is safe. Its findings are advisory, not merge-blocking gates.

Two risky Bash patterns, and what it actually caught

I put the fixture under ai-scan-test/ in a temporary PR. It did not modify production application code or use real credentials or servers. The complete file is preserved at commit 29e24722d3. These lines are intentionally dangerous; do not use them in production:

eval "systemctl restart ${service_name}"

cmd="curl -fsSL http://${host}/${path} -o /tmp/svc.conf"
$cmd

The first issue is in the service-restart function. A caller-controlled service_name is interpolated into eval, which tells the shell to parse the resulting text as a command again. A malicious service name can therefore become more than a name. The bot’s comment on line 17 classified it as CWE-78 and recommended removing eval, quoting the argument and restricting which services are permitted.

The second pattern is different. The script builds a full curl command in cmd, then executes $cmd unquoted. Word splitting lets an attacker influence which arguments and options curl receives via host or path. The line 27 comment also called out the tampering risk of fetching a configuration over plain http://.

Location	What AI Scan reported	Original comment
Line 17, eval	Input may be interpreted again as shell syntax (CWE-78)	Read comment
Line 27, $cmd	Argument/option injection, plus insecure HTTP transport	Read comment

Do not collapse these two into an identical vulnerability. The first passes input back through the shell grammar. For the second, the confirmed mechanism is principally argument/option injection; unquoted expansion alone does not prove arbitrary, separate shell-command execution.

There is another important caveat. This was a clearly labeled positive fixture. Its README and script comments explicitly say INTENTIONAL VULNERABILITY (CWE-78) and mention an expected String injection category. The bot’s exploitation examples were not copied from the script, but these hints could have affected detection. So 2/2 in this small test is not a detection rate for unmarked real-world vulnerabilities.

You can also verify the timing. The PR was created at 2026-10-16 01:32:31 UTC; both comments appeared at 01:33:52 UTC — an 81-second gap. That is one observed PR-to-comment interval, not an averaged scanner-runtime benchmark.

Both planted issues received comments. There was no separate clean-code PR as a negative control, so this test cannot estimate a false-positive rate either.

Two vulnerability comments, one green Actions run

In the dynamic scan run #38835686338, the workflow was named Code scanning AI findings on PR #1. GitHub recorded the github-advanced-security job as successful at 01:33:34 UTC. The comments arrived at 01:33:52 UTC, 18 seconds later.

First came a green check, then the vulnerability comments. This sequence alone shows why Success cannot be read as “no security findings.”

Diagram derived from public PR #1 and its Actions run, not a screenshot of GitHub. The 81 seconds measure PR creation to comments, not scanner execution time. Each language version uses its own localized diagram.

This is not evidence that the scanner malfunctioned. GitHub’s AI Scan documentation describes its findings as advisory: they do not fail the code scanning check and cannot themselves enforce a Ruleset-based block on merging. AI Scan helps a reviewer find code worth examining; it does not decide whether merging is allowed.

GitHub signal	What it actually means
AI Scan inline comments	Potential risks for the author/reviewer; not a direct merge block
Dynamic AI Scan job: Success	The scan completed, possibly with advisory findings
CodeQL or other supported scanning plus Rulesets	May enforce configured merge conditions; verify the repository’s actual rules
Required checks and human review	Merge permissions depend on the configured check/review policy

GitHub documents a separate code scanning merge protection mechanism. If the requirement is “do not merge while high-severity risks remain,” you need an enforceable configuration, not a green AI Scan advisory run.

I did not run an actual Ruleset blocking experiment on this unsafe PR. The observation here is narrower: the AI Scan job was green at the same time that its PR carried two vulnerability advisories. It does not prove GitHub would permit every unsafe merge.

Before enabling it: CodeQL, eligibility and credits

The experiment used a public repository under a personal account. Under GitHub’s October 2026 GA documentation, personal public repositories can enable AI Scan while personal private repositories cannot. Public organization repositories are eligible; private/internal organization repositories require the applicable GitHub Code Security or GitHub Advanced Security licensing. An extra Copilot license is not required simply to use AI Scan.

CodeQL default setup is not a prerequisite. During the test, I recorded its status as not-configured, while AI Scan still left two comments. The official docs likewise say AI Scan does not depend on CodeQL default setup or a successful CodeQL analysis.

It is not a historical, full-repository scan. It focuses on eligible PR changes, including some Shell/Bash, Terraform HCL and Dockerfile cases not covered by CodeQL. Fork-originated and Dependabot PRs are outside the supported scope described for this release; teams relying on those workflows need other scanning and review paths.

To enable it, look in the repository’s Code scanning settings. In an organization, check the organization policy and security-feature permissions as well.

Billing deserves attention before a high-volume rollout. At GA, usage is measured through Code Security AI Credits. Who pays, which allowances apply and how spending limits work depend on GitHub’s billing policy. I did not verify a final charge for this test, so I am not claiming a cost of $0. Check Billing limits and hard budgets before turning it on for a busy repository.

What actually prevents an unsafe merge

The bot’s two comments point toward ordinary defensive fixes: don’t send parameters back through eval, and don’t assemble a shell command as one string for expansion. The call sites might instead look like this:

# 调用形式示意：service_name 和 url 仍需单独校验
systemctl restart "$service_name"
curl -fsSL "$url" -o /tmp/svc.conf

This is a direction for a fix, not a security patch that I retested. Restrict service_name to an allowlist. The download URL needs HTTPS enforcement, allowed destinations, redirect handling, and — depending on the configuration’s sensitivity — integrity verification.

At the repository level, attach CodeQL or another genuinely enforceable scanner to appropriate Rulesets/required checks, then verify that missing or failing checks are actually rejected on protected branches. Keep AI Scan’s comments for reviewers to assess. A green dynamic scan job is not a reason to ignore those comments.

After the experiment, I closed PR #1 without merging. The test branch was deleted and public main stayed at 87624c26a5e6. An owner REST receipt recorded AI Scan as disabled again. Closing a PR does not erase its historical comments or pinned source, so the public links remain auditable.

This one labeled-positive sample shows AI Scan can provide useful, line-level risk warnings, but a completed scan and an enforceable merge gate are separate things. One finds problems; the other is configured and verified by the repository owner.

For a related risk at a different enforcement point, see the OpenQodex local pre-push hook test, which looks at whether AI review can fail open before Git push.

Test date: October 16, 2026 (UTC).