Language Analysis, Sensors, SCM Data, and Source-Code Indexing: Guided Hands-On Workflow

1. Lab assumptions and preflight

Use the disposable local Community Build 26.9.0.129388 instance from earlier chapters and Scanner CLI 8.1.0.6389. The fixture contains Python, XML, YAML, Dockerfile/IaC-style text, tests, generated code, and ignored scratch data. No proprietary source, public endpoint, CI provider, commercial edition, external plugin, or real credential is required.

export SONAR_HOST_URL='http://127.0.0.1:9000'
curl -fsS "$SONAR_HOST_URL/api/system/status"
sonar-scanner --version
git --version
java -version || true

2. Create the mixed-source fixture

rm -rf sonar-p08 sonar-p08-shallow
mkdir sonar-p08 && cd sonar-p08
git init
git config user.email 'academy@example.invalid'
git config user.name 'DevOps Academy Lab'
mkdir -p src/generated tests infra vendor scratch
cat > src/app.py <<'PY'
def classify(value):
    if value < 0:
        return "negative"
    return "non-negative"
PY
cat > src/helper.academy <<'PY'
def helper(name):
    return "hello " + name
PY
cat > src/generated/client.py <<'PY'
# synthetic generated fixture
def generated(x):
    return x
PY
cat > tests/test_app.py <<'PY'
from src.app import classify

def test_zero():
    assert classify(0) == "non-negative"
PY
cat > infra/service.yaml <<'YAML'
apiVersion: v1
kind: ConfigMap
metadata:
  name: academy-p08
data:
  mode: lab
YAML
cat > infra/settings.xml <<'XML'
<settings><mode>lab</mode></settings>
XML
cat > vendor/vendor.py <<'PY'
def vendored():
    return "outside initial scope"
PY
cat > scratch/ignored.py <<'PY'
print("ignored scratch")
PY
cat > .gitignore <<'EOF'
scratch/
.scannerwork/
*.log
EOF
cat > sonar-project.properties <<'EOF'
sonar.projectKey=academy-sonarqube-p08
sonar.projectName=Academy SonarQube P08 Indexing Lab
sonar.sources=src,infra
sonar.tests=tests
sonar.sourceEncoding=UTF-8
EOF
git add . && git commit -m 'mixed-source indexing fixture'
git rev-parse HEAD | tee revision-baseline.txt
find . -maxdepth 3 -type f -not -path './.git/*' | sort

Predict before scanning: vendor/vendor.py is outside sonar.sources; scratch/ignored.py is Git-ignored and outside scope anyway; tests/test_app.py is test code; src/generated/client.py is initially eligible source; and src/helper.academy is inside the source directory but should not yet be recognized as Python.

3. Baseline run: preserve verbose indexing and sensor evidence

read -rsp 'Project-analysis token: ' SONAR_TOKEN; echo
export SONAR_TOKEN
sonar-scanner -X -Dsonar.host.url="$SONAR_HOST_URL" 2>&1 | tee baseline-debug.log
cp .scannerwork/report-task.txt baseline-report-task.txt
grep -Ei 'indexed|excluded|included|sensor|scm|blame|language|project configuration' baseline-debug.log   | tee baseline-scope-evidence.txt

Do not count only log lines that contain the word “indexed”; scanner versions can format debug output differently. Preserve the complete debug log, then use the Project configuration/indexing/sensor sections to reconstruct the effective set. Correlate ceTaskId from report-task.txt with the server background task before interpreting measures.

4. Change exactly one path exclusion and compare

Add one source exclusion for the synthetic generated directory:

printf '
sonar.exclusions=src/generated/**
' >> sonar-project.properties
git add sonar-project.properties && git commit -m 'exclude generated fixture'
git rev-parse HEAD | tee revision-exclusion.txt
sonar-scanner -X -Dsonar.host.url="$SONAR_HOST_URL" 2>&1 | tee exclusion-debug.log
cp .scannerwork/report-task.txt exclusion-report-task.txt
grep -Ei 'excluded|indexed|project configuration|sensor' exclusion-debug.log   | tee exclusion-scope-evidence.txt

Prediction: the source candidate set becomes smaller because src/generated/client.py is filtered out. The test root does not change. vendor/ still cannot appear because it remains outside the initial scope. Verify from logs and resulting measures rather than assuming the configuration was honored.

5. Change one language suffix rule

Now teach Sonar that .academy is also a Python suffix for this disposable project. This is intentionally unusual so the effect is obvious and reversible.

printf '
sonar.python.file.suffixes=.py,.academy
' >> sonar-project.properties
git add sonar-project.properties && git commit -m 'recognize academy suffix as python'
git rev-parse HEAD | tee revision-suffix.txt
sonar-scanner -X -Dsonar.host.url="$SONAR_HOST_URL" 2>&1 | tee suffix-debug.log
cp .scannerwork/report-task.txt suffix-report-task.txt
grep -Ei 'helper\.academy|python|language|indexed|sensor' suffix-debug.log   | tee suffix-recognition-evidence.txt

Prediction: src/helper.academy remains in the same source directory but changes from unrecognized/custom text to a Python-analyzer input. That demonstrates why language recognition is a distinct layer from directory scope.

6. Prove SCM-ignore behavior without bypassing it

git check-ignore -v scratch/ignored.py
git status --ignored --short | grep scratch || true
grep -F 'scratch/ignored.py' baseline-debug.log || true

Sonar respects SCM ignore directives by default. The property sonar.scm.exclusions.disabled=true can disable that behavior, but this lab does not use it: forcing ignored scratch/vendor artifacts into analysis merely to prove a switch would teach the wrong production habit.

7. Demonstrate shallow SCM metadata safely

Create a local shallow clone of the same synthetic repository. A file:// URL is used so Git actually honors --depth 1 for a local source.

cd ..
git clone --depth 1 "file://$PWD/sonar-p08" sonar-p08-shallow
cd sonar-p08-shallow
git rev-parse HEAD | tee shallow-revision.txt
git rev-parse --is-shallow-repository | tee shallow-state.txt
# Same project key/token: this is a controlled comparison of checkout state.
sonar-scanner -X -Dsonar.host.url="$SONAR_HOST_URL" 2>&1 | tee shallow-debug.log || true
grep -Ei 'shallow|blame|scm|missing blame|could not find ref' shallow-debug.log   | tee shallow-scm-evidence.txt

Expected diagnostic shape: the scanner detects shallow history, cannot rely on normal blame retrieval, and may fail depending on the exact repository/analyzer state. Preserve the actual result. Do not claim a successful analysis if the scanner stopped.

8. Repair the SCM defect, not the evidence

git fetch --unshallow 2>/dev/null || git fetch --depth=2147483647
git rev-parse --is-shallow-repository | tee repaired-shallow-state.txt
sonar-scanner -X -Dsonar.host.url="$SONAR_HOST_URL" 2>&1 | tee full-history-debug.log
grep -Ei 'shallow|blame|scm' full-history-debug.log | tee full-history-scm-evidence.txt

The desired change is checkout history, not Sonar policy. Compare the exact revision and configuration across the shallow and repaired runs so the causal variable is clear.

9. Small challenge: diagnose a missing file

A teammate expects vendor/vendor.py in the Python analyzer log and adds sonar.inclusions=vendor/**/*.py. It still does not appear. Explain why and propose the least-surprising fix. Correct reasoning starts with the initial source roots: inclusions cannot expand sonar.sources=src,infra. Decide whether vendor code should be analyzed at all before changing sonar.sources.

10. Guarded cleanup

1. Preserve sanitized debug logs, revisions, report-task.txt files and CE/gate evidence.
2. Revoke the disposable project-analysis token and unset SONAR_TOKEN.
3. Delete only sonar-p08 and sonar-p08-shallow after evidence is copied.
4. Delete the disposable project only if later lessons will not reuse it.
5. Do not delete shared scanner caches, server data/search volumes, global exclusions, or unrelated projects.

Knowledge check

Official references and version notes

• Community Build — Setting initial scope — source/test roots, simple-path rules, and project-base-directory semantics.
• Community Build — Path-based inclusions and exclusions — wildcard filtering after the initial scope.
• Community Build — Verifying analysis scope — debug-log indexing evidence and SonarScanner Context.
• Community Build — File suffixes — language recognition through sonar.<language>.file.suffixes.
• Community Build — Checked-out code and SCM integration — full-clone/blame requirements and shallow-clone behavior.
• Community Build — Other scope adjustments — SCM ignore behavior and sonar.scm.exclusions.disabled.
• Community Build — Supported languages — current analyzer/language matrix.
• Community Build — External issues and generic report format.
• Scanner environment requirements — current JRE auto-provisioning/runtime boundary.