Accounts, Two-Factor Authentication, SSH Keys, Access Tokens, Service Accounts, and Credentials: Diagnostics, Failure Modes, Security, and Performance

1. Diagnose with evidence before changing credentials

Credential incidents become dangerous when teams respond by repeatedly creating broader tokens. Use the same sequence every time: preserve non-secret evidence → identify host/offering and principal → identify resource and operation → classify authentication versus authorization → inspect scope/role/policy → make the least destructive correction → verify → revoke any superseded credential.

2. Failure: valid token, wrong scope

A token can be syntactically valid and associated with the correct user yet fail a request. GitLab REST commonly distinguishes missing/invalid authentication (401) from authenticated-but-insufficient privilege/scope (403) for relevant endpoints. Always read the response body and endpoint documentation rather than treating every failure as “bad password.”

Observed: HTTP 403 / insufficient_scope
Do not: create a full api token immediately
Check: endpoint → supported token types → required permission/scope → resource reach → user/bot role
Repair: add only the required scope/permission, preferably on a fresh short-lived lab credential
Verify: same request succeeds; old credential is revoked

3. Failure: right scope, wrong project/group/user

A read_api project access token remains project-bound. A personal token can reach only resources its user can access. A service account needs membership. A CI_JOB_TOKEN reaching another project can depend on the target allowlist and the triggering user's permission. Diagnose resource identity explicitly: host + namespace + project ID/path + endpoint.

4. Failure: SSH authenticates as the wrong account

Multiple GitLab accounts and keys on one workstation can make SSH choose an unexpected identity. Preserve the host and selected key evidence before editing configuration:

ssh -G git@gitlab.com | grep -E '^(hostname|user|identityfile) '
ssh -vT git@gitlab.com 2>&1 | grep -E 'Offering public key|Authentications|Welcome'

Verbose SSH output can include paths and usernames; sanitize before sharing. It should not display private key material. A robust multi-account configuration uses explicit host aliases and IdentityFile/IdentitiesOnly, then makes each Git remote point to the intended alias.

Host gitlab-lab
  HostName gitlab.com
  User git
  IdentityFile ~/.ssh/id_ed25519_gitlab_ch02
  IdentitiesOnly yes

5. Failure: credential copied into a remote URL or shell history

If a real token appears in git remote -v, a script, issue, terminal transcript, CI log, or repository history, assume exposure. The response order matters:

1. Revoke or rotate the credential first. Editing Git history cannot invalidate a token already copied.
2. Preserve minimal incident evidence without duplicating the secret.
3. Identify where it was exposed: repository, logs, artifact, package, issue, screenshot, shell history, clipboard sync, or external system.
4. Remove/redact the exposed value from those surfaces as policy requires.
5. Issue a narrower replacement only after you understand why the secret was exposed.
6. Verify consumers use the replacement and the old credential cannot authenticate.

For an HTTPS remote, set a clean URL without credentials and use a credential helper instead:

git remote set-url origin https://gitlab.com/YOUR_NAMESPACE/credential-hygiene-lab.git
git remote -v

6. Failure: second factor unavailable

Do not disable 2FA preemptively. Follow the documented recovery order: use a stored recovery code when appropriate; regenerate codes only when still authenticated; use documented SSH-based recovery where supported and previously configured; then follow the offering-specific reset/support path. Current GitLab.com documentation warns that Support cannot reset 2FA for Free accounts when recovery methods are exhausted.

The production lesson is to test recovery readiness before losing the factor: confirm the backup method exists, ownership is clear, and storage is independent from the lost device.

7. Failure: authentication succeeds, SSO authorization still blocks access

For GitLab.com groups using SAML SSO, a user may be signed in to GitLab.com but still need a valid SSO session/identity to access an enforced group hierarchy or perform Git activity. Check the group SAML policy, linked identity/session state, membership role, and whether the credential is a regular-user credential or a non-human credential with different enforcement behavior.

8. Intentionally broken lab — under-scoped API request

Create a temporary PAT for the disposable account with only read_repository. Securely load it into an environment variable without echoing it, then request /api/v4/user. read_repository is intended for repository read operations, while the authenticated-user endpoint requires user/API read permission. Record only the HTTP status and sanitized body.

read -rsp "Under-scoped token: " GITLAB_TOKEN; echo
curl --silent --show-error \
  --output response.json --write-out '%{http_code}\n' \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/user"
unset GITLAB_TOKEN
python -m json.tool response.json
rm -f response.json

Expected diagnostic: an authorization/scope failure rather than a successful profile response. If the target GitLab version returns a different documented status/body, teach the observed response rather than forcing this exact text. Revoke the under-scoped token after the exercise.

Then create a replacement token with read_user only, repeat the request, verify the profile response, and revoke the replacement. The correction increases permission only to the operation required; it does not jump to full api.

9. Rate limits and performance are not authentication problems

HTTP 429, timeouts, TLS failures, proxy failures, or slow API responses should not trigger credential expansion. Preserve headers/status, identify retry guidance and rate-limit policy, and back off appropriately. A new token can hide the symptom while making abuse harder to diagnose.

10. Evidence without secret material

Knowledge check

Summary

Credential diagnostics preserve evidence and classify the failure before changing authority. Distinguish 401/403-style authentication/authorization evidence, resource reach, SSH identity selection, SSO session state, and rate limiting. For leaked credentials, revoke/rotate first; history cleanup follows containment. The best repair is the least privilege that makes the required operation valid.

Official references

• GitLab Docs — Token overview
• GitLab Docs — Access token scopes
• GitLab Docs — Two-factor authentication
• GitLab Docs — 2FA troubleshooting and recovery
• GitLab Docs — Passkeys
• GitLab Docs — SSH keys
• GitLab CLI — Authentication
• GitLab CLI — glab auth status
• GitLab Docs — Personal access tokens
• GitLab Docs — Project access tokens
• GitLab Docs — Group access tokens
• GitLab Docs — Deploy tokens
• GitLab Docs — CI/CD job token
• GitLab Docs — Service accounts
• GitLab Docs — REST API authentication
• GitLab Docs — SAML SSO for GitLab.com groups
• GitLab Docs — Fine-grained personal access tokens