docker driver only supports a single instance. Creating another fails with
'additional instances of driver docker cannot be created'. Just use the
default builder that ships with the docker daemon.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker-container driver requires moby/buildkit:buildx-stable-1 from
Docker Hub, which is unreachable from the build network (EOF / timeout).
The docker driver runs buildkit inside the host daemon without needing
a separate container image.
Also explicitly rm + recreate the builder each run so stale
docker-container instances don't linger as the default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removing <RuntimeIdentifier> from the csproj changed the publish layout,
but the old registry build cache still contains the RID-poisoned layers.
Subsequent builds cached at the COPY/RUN layer boundary reuse those stale
layers, causing FileLoadException at runtime.
Add --no-cache to both API and Web buildx invocations so every build
produces fresh layers. Once the pipeline stabilizes, we can re-enable
cache-from with the updated cache tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
:latest manifests already exist from the first successful run (build 123),
so subsequent docker manifest create without --amend fails with
'refusing to amend an existing manifest list'. Add --amend so every
pipeline run can update the :latest tag.
Also remove DEBUG config.json prints now that the auth path is verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Harbor project is named 'live_recorder' (with underscore), matching
the robot account robot$live_recorder+live. The pipeline was pushing to
'liverecorder' (no underscore) — a different/nonexistent project path —
so the robot's push permission did not apply and every push got 401
despite 'Login Succeeded'.
This was THE root cause of the persistent 401s, not buildx auth
forwarding or token expiry (those were all red herrings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every variant of docker login (--password-stdin, -p) has shown
'Login Succeeded' but docker push consistently gets 401.
Write the base64-encoded user:pass directly into
/root/.docker/config.json, eliminating docker login as a
variable entirely. If push still 401s after this, the problem
is definitively on the Harbor side (robot permissions).
Also fixes broken heredoc escaping from previous commit where
<<'DOCKERCFG' prevented shell variable expansion of AUTH_B64.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Suspect that echo + pipe to --password-stdin may be mangling the Jenkins
masked password variable. Switch to -p (command-line password) for a
cleaner auth path and add debug output to inspect /root/.docker/config.json
after login.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker push is also getting 401 (not just buildx --push), suggesting
the Harbor login token expires during long builds or there is a
credential propagation gap between the standalone Login stage and the
build stages.
Move withCredentials into each build stage and re-login immediately
before every docker push / manifest push. This gives each push
operation a fresh token.
Pipeline is now: Checkout -> Prepare Buildx -> Build API (login + build
+ push + manifest) -> Build Web (same).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildx build --push has failed 401 on every attempt. The buildkit auth
forwarding (whether via docker-container or docker driver) does not work
reliably on this builder.
New strategy: build each platform separately with --load (into local
docker, which can read /root/.docker/config.json), then push with
native docker push, then assemble a multi-arch manifest with
docker manifest create/push.
Per-platform tags are pushed as :<BUILD_ID>-amd64 / :<BUILD_ID>-arm64
and the manifest combines them under the canonical :<BUILD_ID> and :latest.
This replaces a single buildx --push call with:
1. buildx build --platform linux/amd64 --load
2. docker push (amd64)
3. docker rmi (free disk)
4. buildx build --platform linux/arm64 --load
5. docker push (arm64)
6. docker rmi (free disk)
7. docker manifest create + push (multi-arch)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker-container driver runs buildkit in a separate container that
cannot reliably forward host Docker registry credentials via the buildx
session mechanism, causing every --push to fail with 401 Unauthorized.
The --auth flag doesn't exist in buildx 0.23.0 on this builder.
Fix: switch to 'docker' driver which runs buildkit inside the host
Docker daemon and naturally shares its registry auth state.
Changes:
- Login Registry stage restored (before Prepare Buildx)
- Prepare Buildx: driver docker (not docker-container), no driver-opts
- Build stages: stripped withCredentials wrappers and --auth flags
- Removed buildkitd.toml max-parallelism config (docker driver doesn't
support it; swap provides the safety net for OOM)
Pipeline flow: Checkout -> Login -> Prepare Buildx -> Build API -> Build Web
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker-container driver's buildkit session mechanism does not reliably
forward host Docker credentials to the buildkit container, causing every
docker buildx build --push to fail with 401 Unauthorized on the manifest
HEAD request.
Fix: remove the standalone Login Registry stage and embed withCredentials
directly into each build stage, passing credentials to buildkit via the
docker buildx build --auth flag:
--auth 'reg.nxsir.cn=:'
This sends auth directly to buildkit rather than relying on the implicit
docker login -> config.json -> session forwarding chain.
Each stage also does a docker login for the host CLI (needed for
buildx inspect --bootstrap to pull images from the registry, and for
cache-from/cache-to operations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Docker buildx create --driver-opt splits values on commas as list
separators, causing 'env.NO_PROXY=127.0.0.1,localhost,...' to be parsed
as separate key=value entries and failing with:
invalid value "localhost", expecting k=v
The NO_PROXY hosts for build containers are already passed via build-arg
in the build stage; the buildkit container itself doesn't need them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move Login Registry stage before Prepare Buildx so Docker credentials are
in /root/.docker/config.json BEFORE the buildx docker-container builder
is created and bootstrapped. Previously the builder started without auth,
causing buildx --push to fail with '401 Unauthorized' on the manifest
HEAD request (blob layers pushed but manifest rejected).
Also expand the NO_PROXY driver-opt from just 'reg.nxsir.cn' to the full
NO_PROXY_HOSTS list (Tsighua, MCR, daocloud) so the buildkit container
itself also bypasses the proxy for these direct-reachable registries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 7890 proxy intermittently returns '502 Bad Gateway' when tunneling to
mirrors.tuna.tsinghua.edu.cn (observed on libglapi-mesa during
apt-get install in the runtime layer). The Tsinghua mirror, MCR, and
daocloud are all directly reachable from this builder, so the proxy adds
no value and only introduces failure modes.
Add the following hosts to NO_PROXY_HOSTS so apt/curl inside the build
containers hit them directly:
- mirrors.tuna.tsinghua.edu.cn, .tsinghua.edu.cn
- mcr.microsoft.com
- docker.m.daocloud.io
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multi-platform buildx (linux/amd64 + linux/arm64) races both platforms in
parallel by default. On the 4GB build agent, QEMU-emulated 'dotnet restore'
for arm64 alone spikes to 2-3GB and racing amd64 apt-installs push us into
the OOM Killer ("cannot allocate memory" at build 4/5).
Fix: write a small buildkitd.toml with max-parallelism = 1 and pass it to
'docker buildx create --config'. buildkit now runs stages sequentially so
the two platforms don't step on each other's RSS.
Recreate the builder every run so the config always takes effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build agent cannot reach auth.docker.io from behind the local network,
causing 'docker run tonistiigi/binfmt --install arm64' to fail with a token
fetch timeout.
Fix: three-tier fallback for ARM64 binfmt registration:
1. Skip if /proc/sys/fs/binfmt_misc/qemu-aarch64 already exists.
2. Pull tonistiigi/binfmt via docker.m.daocloud.io mirror.
3. Fall back to qemu-user-static via apt.
Add a post-registration sanity check (arm64 alpine + uname -m) so
misconfigured builders fail fast rather than at buildx invocation time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nanxun
