One of the hardest unattended-runner failures to debug is not an explicit error. It is a command such as docker pull or docker build that simply stops making progress, with no obvious Registry response in the log.

The first things I normally check are network routing, DNS, proxies, and the Docker daemon. This time, following a real incident led to a less obvious path: before Docker actually pulls an image, the CLI may look up Registry credentials. If the background runner inherited the same credential helper as a desktop session, the automation may be coupled to an interactive Keychain without anyone realizing it.

The lead came from Fishhawk issue #4046. The project documented an unattended macOS runner where the Docker credential-helper chain stalled while the screen was locked and continued after unlock. I did not independently reproduce that real Keychain behavior, so I am not treating it as something every macOS machine will do.

What I did want to verify was the mechanism underneath it.

My initial assumption was simple: give the runner a separate DOCKER_CONFIG containing only:

{"auths":{}}

and it should no longer touch the user’s Keychain.

The current Docker CLI source and the macOS experiment below both disproved that assumption. An empty auths object is not the same as an explicit anonymous posture. With the Docker CLI version I tested, it can still trigger platform-default credential-helper discovery.

That is the edge case worth knowing.

Why Docker CLI can touch a credential helper even for a public image

Docker’s current documentation still describes ~/.docker/config.json as the client config and documents DOCKER_CONFIG or --config as ways to select another config directory. It also defines credsStore as the default credential store and credHelpers as per-Registry overrides. The relevant references are Docker CLI configuration and docker login / credential stores.

The important part is what the CLI does when the loaded config contains no authentication information.

In the Docker CLI commit I pinned for this investigation, 3d2650ce76053393872e8389c2b77033146cb356, LoadDefaultConfigFile() contains:

if !configFile.ContainsAuth() {
    configFile.CredentialsStore =
        credentials.DetectDefaultStore(configFile.CredentialsStore)
}

And ContainsAuth() is based on three things:

return c.CredentialsStore != "" ||
    len(c.CredentialHelpers) > 0 ||
    len(c.AuthConfigs) > 0

That is why {"auths":{}} is not enough. AuthConfigs still has length zero, so ContainsAuth() is false and the CLI enters DetectDefaultStore().

On Darwin, the platform default in this pinned source is osxkeychain. If docker-credential-osxkeychain is discoverable on PATH, the CLI can select it as the default store.

So an apparently empty runner config can turn into this:

no credsStore
no credHelpers
no auth entries
        ↓
ContainsAuth() == false
        ↓
discover the platform default helper
        ↓
macOS: docker-credential-osxkeychain

Source inspection explains the path, but I also wanted to know whether a real Darwin process would actually take it.

I ran four config shapes on the same macOS runner

I deliberately kept real accounts and the real Keychain out of the experiment. I did not execute Fishhawk code, did not run docker login, and did not modify the GitHub runner’s system Keychain.

The fixture used a hosted macOS ARM64 runner with Docker CLI source pinned to the commit above. I placed two self-authored fake helper executables at the front of PATH:

docker-credential-osxkeychain
docker-credential-probe

They never read real credentials. They only recorded which helper was invoked, which action was requested, and which Registry key was queried.

The exact command was:

GOFLAGS=-mod=vendor go test \
  -run '^TestAITokenCredentialExperiment$' \
  -count=1 -v ./cli/config

GitHub Actions run 37620686435 executed on macOS ARM64, macos26, with Go 1.26.8. The command exited 0. The raw terminal output was short:

=== RUN   TestAITokenCredentialExperiment
AITOKEN_RESULT .../data/experiments/docker-credentials/raw-go-test.json
--- PASS: TestAITokenCredentialExperiment (0.95s)
PASS
ok   github.com/docker/cli/cli/config   0.956s

The marker output from the four arms is the useful part:

Config	ContainsAuth()	Store after default load	Observed helper call
{"auths":{}}	false	osxkeychain	docker-credential-osxkeychain get https://index.docker.io/v1/
{"auths":{"fishhawk.invalid":{}}}	true	file store	no helper call
{"credsStore":"probe","auths":{}}	true	probe	docker-credential-probe get https://index.docker.io/v1/
{"credHelpers":{"example.invalid":"probe"}}	true	file store	probe called only for example.invalid

The first row is the surprising one. A bare {"auths":{}} did not create an explicitly helper-free state. Once the fake docker-credential-osxkeychain executable was on PATH, this Docker CLI selected it and actually invoked it while resolving Docker Hub authentication.

The second row explains an odd-looking detail in Fishhawk’s merged implementation. Instead of leaving auths empty, NewAnonymousDockerConfig() writes a no-secret placeholder:

{
  "auths": {
    "fishhawk.invalid": {}
  }
}

There is no username, password, or token in that object. Its purpose is control flow: make the auth map non-empty, which makes ContainsAuth() true and skips platform-default helper discovery.

At first glance, “add an auth entry so that the runner does not authenticate” looks backwards. The source plus the runtime result make the reason clear.

There is an important version boundary here. I verified this against a pinned Docker CLI commit. I would not treat the placeholder trick as a permanent protocol contract. If your automation depends on it, keep a regression test around the behavior when Docker CLI versions change.

Keep the runner’s Docker config separate from the desktop user’s config

After this experiment, I would not fix an unattended job by deleting credsStore from the user’s normal ~/.docker/config.json.

Credential stores exist for a good reason. Docker’s own documentation recommends external stores because keeping Registry credentials in a system Keychain or credential helper is generally safer than writing them directly into config.json. The problematic design is the background runner silently inheriting an interactive user’s credential state.

For a runner that only pulls public images, I prefer a runner-owned DOCKER_CONFIG:

runner_docker_config="$(mktemp -d)"

cat > "$runner_docker_config/config.json" <<'JSON'
{
  "auths": {
    "runner.invalid": {}
  }
}
JSON

DOCKER_CONFIG="$runner_docker_config" docker pull alpine:latest

The placeholder key contains no real credential. The point is also not that it must be named runner.invalid. The important boundary is that the runner no longer reads the desktop user’s ~/.docker/config.json, while this tested Docker CLI version also avoids auto-detecting the platform helper.

Private images are a different case. An anonymous config cannot authenticate to a private Registry, so the runner needs an explicit automation credential path.

When possible, I prefer a Registry-scoped credHelpers mapping over one broad default credsStore because the scope is visible in the config:

{
  "credHelpers": {
    "registry.example.com": "your-noninteractive-helper"
  }
}

In the fixture I queried both example.invalid and other.invalid. The fake helper was invoked exactly once, for example.invalid. That matches Docker’s documented per-Registry helper behavior.

What your-noninteractive-helper should actually be depends on the Registry provider and the lifetime of the credential. I would not turn one helper into a universal recommendation. For unattended automation, the useful properties are machine-oriented credentials, narrow Registry scope, preferably short-lived access, and predictable behavior without an interactive desktop session.

There is one operational wrinkle that is easy to miss: DOCKER_CONFIG is the whole Docker CLI config directory, not just an auth file. If your desktop setup also relies on Docker Desktop’s buildx, Compose, or other CLI plugins, pointing the runner at a brand-new directory can affect plugin discovery too.

Fishhawk’s merged implementation explicitly carries forward plugin directories into the runner-owned config. The idea is worth copying; the exact paths are not. If your runner only does docker pull, this is usually simple. If it also uses docker buildx, Compose, or custom plugins, verify plugin discovery when you isolate credentials.

This is similar to the boundary I ran into in using CLIProxyAPI as a Docker-hosted Codex proxy: reusing one person’s state is convenient at first, but once a service has to run unattended for months, the hidden coupling becomes harder to reason about than a little extra explicit configuration.

If a helper blocks, do not wait for the whole docker pull timeout

I ran one more small experiment. The fake docker-credential-probe get command was changed to deliberately never return, then the Docker CLI credential lookup was sent through it.

The outer test harness allowed only 900 ms.

The marker first recorded:

docker-credential-probe|get|https://index.docker.io/v1/

The child process then stayed blocked until the outer deadline terminated it at 933 ms.

This proves a narrow but useful point: once the credential lookup is inside an external helper, an unresponsive helper can hold that lookup path open. The current docker-credential-helpers client Get() runs an external helper and waits for its output; that layer does not choose your runner’s business-level timeout for you.

For unattended jobs, I want the timeout at a place that explains what failed.

A single 30-minute job timeout that eventually reports “docker pull failed” is much less useful than a bounded probe before a private-Registry pull, or at minimum a dedicated timeout around the Docker step with a failure reason that says the credential helper did not answer.

Fishhawk’s merged implementation follows that direction. When an operator explicitly supplies a Docker config, the runner identifies the helpers that the requested images would use and probes them under a bound before the pull. Fishhawk currently uses 20 seconds, but 20 seconds is Fishhawk’s engineering policy, not a Docker default and not a universal recommendation from this article.

A local machine-identity helper may deserve only a few seconds. A network-backed vault may need longer. The reusable idea is that credential lookup gets its own observable bound.

What the macOS lock-screen evidence actually proves

This is where the article can most easily overclaim.

Fishhawk #4046 describes a real macOS Docker Desktop host whose interactive config used credsStore: desktop. According to that project’s incident evidence, the helper chain reached docker-credential-osxkeychain, blocked in the Keychain path while the screen was locked, and resumed after unlock. The project later merged its hardening in PR #4059.

My first-party experiment did not lock a real macOS session and did not ask the real docker-credential-osxkeychain to read the system Keychain. I used a same-named fake helper to verify helper selection and a deliberately blocking fake helper to verify the wait behavior.

So the evidence supports these statements:

• DOCKER_CONFIG, credsStore, and credHelpers directly determine the credential path an unattended Docker CLI can take.
• With the pinned CLI version I tested, {"auths":{}} on macOS can still auto-select osxkeychain when that helper is discoverable.
• A no-secret placeholder auth entry prevented that default-helper discovery in the same fixture.
• An external credential helper can keep credential lookup waiting until the caller imposes an outer bound.
• Fishhawk’s real locked-screen Keychain incident is consistent with this mechanism, but the actual SecItemCopyMatching lock-screen behavior remains Fishhawk’s third-party field evidence.

If an unattended runner works while someone is using the Mac but hangs after lock, logout, or during an overnight task, this is the order I would inspect things:

1. Print the runner’s actual DOCKER_CONFIG, HOME, and config path. Do not assume it is reading the file you expect.
2. Inspect config.json for credsStore and Registry-specific credHelpers.
3. Check which docker-credential-* executables are actually visible on PATH.
4. Compare against a dedicated runner config before changing the user’s global ~/.docker/config.json.
5. If a helper is required, observe its lookup separately from the full image operation and give that step an explainable timeout.
6. After changing DOCKER_CONFIG, verify buildx, Compose, and any required CLI plugins still resolve.

These failures are easy to misdiagnose as Docker networking because the last visible command is often just docker pull. The real stall may happen earlier, while the CLI is deciding whether the Registry has credentials and where to retrieve them.

I now treat Docker credential configuration as part of the runner’s runtime state, not as a harmless user preference that background automation can inherit. The desktop user can keep using the Keychain. The unattended runner gets its own explicit, testable, bounded configuration. Once those two states are separated, the failure path becomes much easier to reason about.

If this runner is going to live on a server or development machine for the long term, the same principle also applies to recovery. My earlier guide on backing up Docker volumes to GitHub is about a different failure mode, but the operational lesson is the same: do not stop at “it works today.” Know where state lives, how to diagnose it, and how to rebuild it on another machine.