Chapter 09Lesson 04170–225 min

PyPI, Conda, Python Package Indexes, Proxy Behavior, Uploads, and Metadata Management: Diagnostics, Failure Modes, Security, and Performance

Diagnose Python package repository failures from evidence: dependency confusion, 401/403, duplicate publication, stale proxy metadata, incompatible wheel tags, source-build execution, wrong Simple API paths, and stale assumptions about Conda/PyPI support.

401 / 403Metadata cacheWheel tagssdist buildDiagnostics

Learning objectives

  • Diagnose Python package failures using a fixed evidence sequence rather than trial-and-error cleanup.
  • Recognize dependency confusion, stale index metadata, authentication failure, duplicate publication, and wheel incompatibility as different classes of failure.
  • Distinguish pip cache, Nexus proxy cache, database/blob health, and upstream behavior.
  • Interpret source-distribution build execution as a supply-chain event, not merely an installer detail.
  • Repair a controlled PyPI failure without weakening TLS, authorization, deployment policy, or namespace controls.

Failure lab rule. Preserve the original error before fixing anything. All examples target disposable Chapter 09 repositories and fake/synthetic packages. Never troubleshoot a production PyPI/Conda repository by deleting files under the Nexus data directory or editing database rows.

1. The evidence-first diagnostic sequence

Use the same operating sequence introduced earlier in the course, now with Python-specific checkpoints:

  1. Preserve the exact pip/Twine/Conda error and command context without secrets.
  2. Confirm Nexus version, edition, Java/runtime, and repository format/type.
  3. Inspect the client index URL, credential source, and whether direct public fallback is configured.
  4. Inspect group members/order and hosted deployment policy.
  5. Test repository authorization with the same identity.
  6. Inspect Simple API project metadata plus Nexus component/assets.
  7. Inspect proxy cache age, negative cache, upstream status, and ETag/freshness evidence.
  8. Check database/blob/disk health and relevant logs/tasks.
  9. Change the smallest controlled state.
  10. Verify with a fresh venv/cache and one known request.

2. Failure: dependency confusion through extra-index-url

Suppose an internal requirement is acme-forecast-engine>=2. A developer configures the internal index as index-url and public PyPI as extra-index-url. pip discovers 2.4.0 internally and an attacker publishes 99.0.0 publicly. pip's documented behavior is to consider all configured locations; the public candidate can win if it is the best version match.

Do not reproduce this on public PyPI. The correct lab is conceptual or uses two local synthetic indexes. The repair is architectural: remove direct public extra-index routing for internal names, route public dependencies through Nexus, reserve internal namespaces, and apply routing/policy controls.

3. Intentionally broken example: preserve a Twine authentication failure

Use the disposable academy-ch09-hosted repository and one synthetic distribution file. The wrong password below is deliberately fake and cannot authenticate. Its purpose is to capture the original 401/403-class evidence without risking a real credential.

export TWINE_REPOSITORY_URL="http://127.0.0.1:8081/repository/academy-ch09-hosted/"
export TWINE_USERNAME="academy-ch09-publisher"
export TWINE_PASSWORD="FAKE-INVALID-CH09-CREDENTIAL"

python -m twine upload dist/learner_ch09_widget-0.1.0-py3-none-any.whl   2>&1 | tee "$LAB/evidence/intentional-auth-failure.txt" || true

unset TWINE_PASSWORD TWINE_USERNAME TWINE_REPOSITORY_URL

Interpret the result before repair. If it is 401, authentication failed. A 403 normally means the identity was authenticated but lacks the required repository privilege. A connection error or 404 points somewhere else entirely. Do not respond to 401 by granting administrator privileges.

4. Repair authentication without broadening privileges

Confirm the username exists, its role contains only the required hosted add/read/browse privileges, and the hosted URL is correct. Enter the real disposable password interactively into an environment variable only for the retry, then unset it. If a successful upload now fails as a duplicate, that is evidence that authentication is repaired and the deployment policy is correctly blocking redeploy.

5. Failure: duplicate version or filename publication

With Disable redeploy, a second upload of an already accepted distribution is rejected. That is not a storage outage. It is policy protecting release identity. The correct fix is to publish a new version after rebuilding intentionally, or—if the first upload itself was bad—follow the organization's controlled deletion/revocation process rather than switching the whole repository to Allow redeploy for convenience.

Preserve the original Twine HTTP response and compare Nexus Browse/Search: the first accepted asset remains present with its original hash.

6. Failure: a new upstream release does not appear

First test with a fresh pip cache. If the client still sees stale candidates, inspect the Nexus proxy repository's metadata/component ages and negative cache. A proxy can intentionally avoid re-querying the upstream until its configured freshness interval expires. Current 3.95.x also contains PyPI cache correctness fixes, so record the exact patch version before applying old workarounds.

Observation Likely layer Least-destructive next step
Fresh client sees old project file list Nexus proxy metadata/cache Inspect maximum metadata/component age and remote health; invalidate supported cache state only if justified.
Only one workstation sees old package pip/client cache Use a fresh cache/venv or --no-cache-dir diagnostic.
Nexus upstream request returns 304 Remote/ETag freshness Compare ETag and upstream metadata; 304 can be a valid unchanged response.
Project was previously absent and remains absent Negative cache Inspect Not Found Cache TTL before assuming upstream is broken.

7. Failure: “No matching distribution” despite the project existing

A Simple API page can list a wheel that is incompatible with the current interpreter, OS, ABI, or architecture. For example, a Windows CPython wheel is not a valid candidate on Linux. Preserve pip debug --verbose output and the wheel filename tags before changing Nexus.

python -m pip debug --verbose   | tee "$LAB/evidence/pip-debug-tags.txt"

If no compatible wheel exists, pip may fall back to an sdist if one is available. That changes the execution path and may require compilers or build dependencies. The fix may be “publish the correct wheel,” not “clear the repository cache.”

8. Failure: an sdist triggers build code or unexpected build dependencies

A source distribution installation can create an isolated build environment and install the packages listed under [build-system].requires. This is expected modern packaging behavior, but it is also executable supply-chain activity. If the build fails, capture the build-backend name, build requirements, exact index routes used to resolve those requirements, and compiler/system dependency evidence.

Do not bypass this by adding arbitrary public indexes. If a build requirement is missing, decide whether it should be proxied through the approved Nexus route, prebuilt into the environment, or removed from the package. Adding extra-index-url can turn a build failure into a dependency-confusion exposure.

9. Failure: wrong repository URL or missing /simple

Twine and pip intentionally use different URL shapes. Twine publishes to the hosted repository root. pip consumes the Simple API endpoint ending in /simple. If pip is pointed at a Twine upload URL, its project-discovery requests will not have the expected index semantics. If Twine is pointed at the group Simple API, publication is routed to the wrong interface.

10. Failure: old Conda assumptions applied to a current Nexus

A runbook written before Nexus 3.92 might state “Conda is proxy-only.” That statement is stale for current 3.95.2. Conversely, an old Nexus instance may genuinely lack hosted/group support. Diagnose from the actual version and available repository recipes, not from a current screenshot applied retroactively.

Conda failures also need channel-specific evidence: channel URL, subdir, repodata.json availability, package filename/build, authentication method, and whether the request is going through a proxy/hosted/group repository.

11. Performance: measure the layer that can cause the latency

Latency source Typical evidence Do not confuse with
pip client cache Verbose pip output; cache directory activity. Nexus proxy cache.
Nexus proxy freshness/upstream Request logs, cache ages, remote response timing. Blob-store throughput.
Database latency Repository/search/API timing plus DB metrics. Python wheel download time alone.
Blob IO Large asset read/write timing and storage metrics. Simple API metadata generation alone.
sdist build Build-backend/compiler logs. Repository download latency.
Network/TLS Connection timing, proxy/reverse-proxy logs. Package resolver candidate logic.

Do not tune JVM heap because a native extension takes three minutes to compile. Do not clear pip cache because the Nexus database is read-only due to disk pressure. Follow causal evidence.

12. Controlled diagnostic drill

Starting from the Lesson 2 lab, collect: current group HTML/JSON project index; current hosted asset hash; a fresh pip install result; proxy remote status; and current role/privilege matrix. Then choose exactly one fault: temporarily remove the publisher's hosted add privilege, block the proxy repository, or intentionally point a fresh pip config at a non-existent group name. Capture the failure, restore only the changed setting, and repeat the same request.

The drill succeeds only if your evidence states both the cause and the state that did not change. For an authorization failure, no new blob should appear. For a wrong URL, repository state should be unchanged. For a blocked proxy, already-cached content may still be served while uncached public content fails.

Knowledge check

A fresh pip client sees a package but rejects all files. What should you inspect before Nexus storage?

Why can a second Twine upload failing be a healthy outcome?

Why is a 403 not fixed by changing the password?

What is the correct reaction to a build dependency missing from the approved index?

How can an old Conda runbook be wrong even if its commands once worked?

13. Summary and next step

Python package diagnosis is strongest when every error is classified by client route, authorization, metadata/candidate selection, cache/upstream behavior, build execution, and Nexus storage health. Lesson 5 integrates that discipline into an internal-only checkpoint where the learner proves package bytes, client routing, persistence, and cleanup.

Official references and version notes

Version-sensitive statements were rechecked against Sonatype and Python Packaging primary documentation on 2026-08-26. The mandatory lab assumes self-hosted Nexus Repository Community Edition 3.95.2, the bundled/supported Java 21 runtime, a disposable single-node local instance, and a current Python 3 environment. Record python --version, python -m pip --version, python -m twine --version, and python -m build --version locally because Python packaging clients evolve independently of Nexus.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.