Rolling Upgrades, Binary Compatibility, Feature Compatibility Version, and Upgrade Sequencing

Learning objectives

1. AtlasMart problem: “all nodes run 8.3” is not the end of an upgrade

MongoDB separates the binary version—the executable running on a node—from Feature Compatibility Version (FCV), a deployment-level gate controlling use of features whose persisted state may be incompatible with older releases. This separation creates a deliberate downgrade window: you can upgrade binaries, verify behavior, and delay new incompatible feature usage until the deployment has burned in.

That also means “8.3 binary + old FCV” and “8.3 binary + FCV 8.3” are different operational states. A driver is a third versioned component. Backup validity and rollback prerequisites form separate gates again.

2. Observe version and FCV without changing either

docker rm -f atlasmart-ch26-l4 2>/dev/null || truedocker volume rm atlasmart-ch26-l4-db 2>/dev/null || truedocker run -d --name atlasmart-ch26-l4 -p 127.0.0.1:27207:27017 -v atlasmart-ch26-l4-db:/data/db mongodb/mongodb-community-server:8.3.8-ubuntu2204-slim --replSet rs26l4 --bind_ip_alluntil mongosh "mongodb://127.0.0.1:27207/admin?directConnection=true" --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null | grep -q 1; do sleep 1; donemongosh "mongodb://127.0.0.1:27207/admin?directConnection=true" --quiet --eval 'rs.initiate({_id:"rs26l4",members:[{_id:0,host:"atlasmart-ch26-l4:27017"}]})'until mongosh "mongodb://127.0.0.1:27207/admin?directConnection=true" --quiet --eval 'db.hello().isWritablePrimary' 2>/dev/null | grep -q true; do sleep 1; donemongosh "mongodb://127.0.0.1:27207/admin?directConnection=true" --quiet --eval 'printjson({binary:db.version(),fcv:db.adminCommand({getParameter:1,featureCompatibilityVersion:1})})'

The mandatory lab stops here. FCV is observed and never changed in mandatory labs. The remaining commands are runbook examples for a separately prepared staging deployment that already satisfies the version-specific prerequisites.

3. Current 8.2→8.3 replica-set sequence

The current MongoDB 8.3 procedure requires all members to be on the documented source version and FCV before the rolling binary change. For an 8.2→8.3 replica set: verify all members on 8.2, FCV 8.2, and no member in ROLLBACK or RECOVERING. Upgrade secondaries one at a time and wait for each to return SECONDARY. Step down the primary, confirm a new primary, then upgrade the old primary.

const a=db.getSiblingDB("admin");printjson(a.runCommand({buildInfo:1}).version);printjson(a.runCommand({getParameter:1,featureCompatibilityVersion:1}));const s=a.runCommand({replSetGetStatus:1});printjson(s.members.map(m=>({name:m.name,state:m.stateStr,health:m.health,optimeDate:m.optimeDate})));

# 1. Verify driver compatibility, release notes, compatibility changes, and tested restore.# 2. Upgrade ONE secondary's binary from the approved 8.2 patch to approved 8.3 patch.# 3. Wait for SECONDARY and normal replication/metrics before touching the next secondary.# 4. Repeat for each secondary.# 5. Step down the primary using rs.stepDown(); confirm another member is PRIMARY.# 6. Upgrade the old primary binary; wait for it to rejoin healthy.# 7. Burn in with representative workload and existing FCV before enabling incompatible 8.3 features.

4. FCV advancement is a separate decision

After all binaries are on 8.3, MongoDB recommends a burn-in period before enabling backward-incompatible 8.3 features. When the organization accepts that downgrade may require removing newly persisted incompatible features, the current command requires confirm:true.

// Staging/production runbook step only after current docs, backup, burn-in, and rollback review:db.getSiblingDB("admin").runCommand({setFeatureCompatibilityVersion:"8.3",confirm:true});

5. Rehearse the upgrade as a state machine before touching binaries

state={ "driverCompatible":True,"backupRestoreTested":True,"compatibilityReviewed":True, "members":[{"name":"n1","role":"PRIMARY","binary":"8.2","health":"UP"},{"name":"n2","role":"SECONDARY","binary":"8.2","health":"UP"},{"name":"n3","role":"SECONDARY","binary":"8.2","health":"UP"}], "fcv":"8.2","burnInPassed":False}def gate(label,condition):    print(label,"PASS" if condition else "BLOCK")    assert conditiongate("preflight",state["driverCompatible"] and state["backupRestoreTested"] and state["compatibilityReviewed"])gate("all healthy 8.2",all(m["binary"]=="8.2" and m["health"]=="UP" for m in state["members"]) and state["fcv"]=="8.2")for name in ["n2","n3"]:    next(m for m in state["members"] if m["name"]==name)["binary"]="8.3"    gate(f"{name} rejoined",next(m for m in state["members"] if m["name"]==name)["health"]=="UP")# election/stepdown occurs before upgrading the old primarystate["members"][0]["role"]="SECONDARY"; state["members"][1]["role"]="PRIMARY"state["members"][0]["binary"]="8.3"gate("all binaries 8.3",all(m["binary"]=="8.3" for m in state["members"]))state["burnInPassed"]=Truegate("fcv decision allowed",state["burnInPassed"] and state["backupRestoreTested"])print(state)

A simulator cannot prove package scripts, storage compatibility, election behavior, or application correctness. It does force the runbook to encode gates and prevents “upgrade the next member regardless of what happened” thinking.

6. Sharded-cluster sequencing and current-version discipline

A sharded upgrade has more planes. Follow the current 8.3 sharded-cluster procedure for the exact source release; upgrade the config server replica set, then shard replica sets, then mongos routers in the documented order, waiting for each member/component to return healthy. FCV is changed through mongos only after the binary rollout and burn-in. Do not extrapolate a 7.0→8.0 sequence to 8.2→8.3 without re-reading the current procedure.

7. Production judgment

An upgrade is a controlled compatibility migration, not package maintenance. Freeze unrelated changes, prove restore, verify drivers and release compatibility, establish SLO/alert baselines, roll binaries one member at a time, burn in, and treat FCV as a deliberate second commitment. Maintain a downgrade plan based on the current target/source pair—not a generic “replace binaries” script. The final lesson turns these controls into incident runbooks that tell operators what evidence to collect before acting.

Authoritative references

Operational fields, thresholds, upgrade paths, FCV behavior, Atlas metrics, and driver compatibility evolve. Re-check the current documentation for the exact server patch, deployment topology, driver, Atlas tier, and target upgrade/downgrade path before changing production systems.

• serverStatus command
• db.stats() / dbStats
• $collStats aggregation stage
• Database Profiler
• db.setProfilingLevel()
• $currentOp aggregation stage
• MongoDB Log Messages
• Explain Results
• Replication
• Replica Set Oplog
• Check Replica Set Replication Lag
• WiredTiger Storage Engine
• Atlas Monitoring and Alerts
• Atlas Monitoring and Alert Guidance
• Atlas Alert Basics
• Atlas Metrics
• PyMongo Release Notes
• PyMongo Upgrade Guidance
• MongoDB 8.3 Release Notes
• Upgrade 8.2 to 8.3
• Upgrade 8.2 Replica Set to 8.3
• Upgrade 8.2 Sharded Cluster to 8.3
• MongoDB 8.3 Compatibility Changes
• Downgrade 8.3 to 8.2
• MongoDB Versioning
• Backup Methods
• mongosh Release Notes