Code scanning and rulesets: make a green check mean something

Problem

Many repositories run code scanning and still merge vulnerable code. The scan runs, the results land in the Security tab, and the pull request turns green anyway, because nothing connects the scan's findings to the decision to merge. Scanning that cannot block a merge is a report, and reports get read late.

GitHub offers three separate mechanisms that people often blur together. Required status checks say a named job must succeed. Code scanning merge protection, configured in a ruleset, says a pull request cannot merge if a required tool finds alerts above a chosen severity, or has not finished, or is not set up. Code owner review says the right people must approve changes to sensitive paths. The GitHub documentation is explicit that merge protection through rulesets is not the same thing as status checks. Using the right one for each job is the whole skill.

Options and trade-offs

• Report only: scanning runs and alerts appear in the Security tab. There is no friction and no enforcement.
• Required status check on the scan job: the job must pass. It is easy to set up, but it checks that the job ran, not what it found, unless the job itself fails on findings.
• Code scanning rule in a ruleset: blocks merges based on alert severity thresholds and also when analysis is incomplete or missing. This is the closest match to "do not merge known problems".
• Code owner review: guarantees that people who understand a directory look at changes to it. It adds latency, so reserve it for paths that carry real risk.
• Branch protection rules versus rulesets: both can require checks and reviews. Rulesets layer, can target tags as well as branches, and allow granular bypass lists. Rulesets are available on public repositories for every plan, and on private repositories with Pro, Team or Enterprise Cloud.

My recommendation

Use a ruleset on the default branch that combines four things: a pull request with at least one approval, a required build check, the CodeQL code scanning rule with a severity threshold you can live with, and code owner review for the directories that matter. Start the severity threshold at high or critical so the gate earns trust, then tighten it as the backlog shrinks. Keep the bypass list short and limited to a role you audit.

Minimal working example

First, a CodeQL workflow that runs on pull requests and on a schedule:

name: codeql
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  schedule:
    - cron: "17 3 * * 1"

permissions:
  contents: read
  security-events: write

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/init@v3
        with:
          languages: java-kotlin
          build-mode: autobuild
      - uses: github/codeql-action/analyze@v3

Next, code owners for sensitive paths (.github/CODEOWNERS):

# Default reviewers
*                      @my-org/maintainers

# Security-sensitive paths need the security group
/src/main/java/**/auth/   @my-org/security
/.github/workflows/       @my-org/security

Then a ruleset, saved as ruleset.json and created with the REST API. Verify the field names against the current API reference before relying on it:

{
  "name": "main-protection",
  "target": "branch",
  "enforcement": "active",
  "conditions": {
    "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] }
  },
  "bypass_actors": [
    { "actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "pull_request" }
  ],
  "rules": [
    {
      "type": "pull_request",
      "parameters": {
        "required_approving_review_count": 1,
        "dismiss_stale_reviews_on_push": true,
        "require_code_owner_review": true,
        "require_last_push_approval": false,
        "required_review_thread_resolution": true
      }
    },
    {
      "type": "required_status_checks",
      "parameters": {
        "strict_required_status_checks_policy": true,
        "required_status_checks": [ { "context": "build" } ]
      }
    },
    {
      "type": "code_scanning",
      "parameters": {
        "code_scanning_tools": [
          { "tool": "CodeQL", "security_alerts_threshold": "high_or_higher", "alerts_threshold": "errors" }
        ]
      }
    }
  ]
}

gh api -X POST repos/OWNER/REPO/rulesets --input ruleset.json

Test it in a throwaway branch. Open a pull request that introduces a known issue (for example SQL built by string concatenation) and confirm the merge is blocked. Then fix it and confirm the block lifts. Finally, open a pull request that touches a code-owned path and confirm it waits for the owning team.

Checklist

• Enable code scanning with the CodeQL workflow, and confirm the first run on the default branch finishes before turning on enforcement.
• Add the code scanning rule to a ruleset, with a severity threshold chosen from your current alert backlog.
• Keep a required build check and a pull request rule in the same ruleset.
• Add CODEOWNERS entries only for paths where review adds real value, and turn on code owner review.
• Use enforcement: evaluate first if your plan supports it, to see what would have been blocked before you enforce.
• Keep the bypass list to one audited role.
• Be careful with path filters on workflows that feed required checks. A skipped workflow never reports a status, and a required check that never reports blocks the merge forever.
• Organization-wide code scanning rules are limited to GitHub Enterprise plans. Smaller teams configure the rule per repository.

Where this doesn't apply

Repositories that use an external scanner and never upload SARIF results to GitHub cannot use code scanning merge protection, so the gate has to live in that tool's own CI check. Monorepos that gate through a separate policy engine may prefer to keep one source of truth.

One honest limitation

A severity gate catches what the scanner finds, and scanners miss things. The gate also creates pressure to dismiss alerts rather than fix them, so review dismissals as carefully as you review merges.

Sources

• Code scanning merge protection — https://docs.github.com/en/code-security/concepts/code-scanning/merge-protection
• Available rules for rulesets — https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/available-rules-for-rulesets
• About code owners — https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
• REST API for repository rules — https://docs.github.com/en/rest/repos/rules
• Configuring default setup for code scanning — https://docs.github.com/en/code-security/code-scanning/enabling-code-scanning/configuring-default-setup-for-code-scanning

---