Embeddings, Dimensions, Cosine Similarity, HNSW Concepts, Recall, and Latency

Learning outcomes

AtlasMart wants semantic product retrieval: a query such as “light backpack for trail travel” should find related products even when the exact words differ. That requires a vector representation, a similarity rule, and a measurable retrieval system—not a claim that Redis somehow understands language by itself.

1. Embeddings are numbers produced outside Redis

An embedding is a numeric vector produced by a model or deterministic feature function so related inputs tend to occupy nearby regions of a vector space. The dimension is the number of numeric components. Redis can store and search these vectors; it does not turn product text into embeddings unless your application or another service supplies the vector.

2. Dimension and type mismatch are schema errors, not search tuning

A 384-dimensional query is not comparable to a 768-dimensional stored vector under the same mathematical model. Search vector fields require an exact configured dimension; Vector Sets establish a set dimension and reject incompatible additions. Validate the embedding model/version and dimension before the Redis call.

docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch10:lesson1:vectorsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VADD atlasmart:ch10:lesson1:vectors VALUES 3 1 0 0 p1001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VDIM atlasmart:ch10:lesson1:vectorsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VADD atlasmart:ch10:lesson1:vectors VALUES 2 1 0 bad-dimensiondocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VCARD atlasmart:ch10:lesson1:vectors

Expected evidence: VDIM reports 3; the two-dimensional addition fails; cardinality remains unchanged by the failed write. Exact error text can vary with release/client protocol, so the invariant is rejection of incompatible dimensionality.

3. Cosine similarity is not dot product and not distance

Cosine similarity divides the dot product by the vector magnitudes. Two vectors with the same direction can have different lengths but cosine similarity near 1. A raw dot product changes with magnitude. A cosine distance is usually derived from similarity (for example 1 - similarity), so “higher is better” and “lower is better” depend on the API.

from math import sqrtdef norm(v): return sqrt(sum(x*x for x in v))def cosine(a,b):    if len(a) != len(b): raise ValueError("dimension mismatch")    na, nb = norm(a), norm(b)    if na == 0 or nb == 0: raise ValueError("zero vector has undefined cosine")    return sum(x*y for x,y in zip(a,b)) / (na*nb)q=[0.98,0.10,0.02]items={"p1001":[1.0,0.0,0.0],"p1002":[0.90,0.25,0.05],"p1003":[0.0,1.0,0.0],"p2001":[0.75,0.20,0.60]}ranked=sorted(items, key=lambda k: cosine(q,items[k]), reverse=True)for k in ranked: print(k, round(cosine(q,items[k]),6))

This local calculation gives a small exact baseline independent of Redis approximation. It also refuses zero vectors and mismatched dimensions instead of silently producing nonsense.

4. Normalization is part of the model/metric contract

Some embedding pipelines emit unit-length vectors; others do not. Cosine similarity mathematically normalizes magnitudes during comparison, while inner-product workflows often depend on explicit normalization if you want cosine-equivalent ranking. Never add a normalization step merely because “vectors usually need it.” Record what the embedding model and chosen metric expect, then test with fixtures.

5. HNSW trades exhaustive work for graph navigation

HNSW means Hierarchical Navigable Small World. It builds a multi-layer graph where each vector connects to a bounded set of neighbors. Search starts in sparse upper layers, follows promising links, then refines near the bottom. It examines fewer candidates than a linear scan, which can reduce latency, but it may miss a true nearest neighbor.

6. Recall@k measures ANN fidelity, not business relevance

For a query, let the exact top-k set be ground truth and the approximate top-k be the production result. recall@k is the fraction of exact top-k neighbors recovered by the approximate method. If exact top-5 is A,B,C,D,E and approximate top-5 is A,B,C,F,G, recall@5 is 3/5 = 0.6. That says nothing about whether A through E are actually useful recommendations.

exact = ["A","B","C","D","E"]approx = ["A","B","C","F","G"]k=5recall=len(set(exact[:k]) & set(approx[:k])) / kprint(recall)  # 0.6

7. Vector Sets and Search vector fields solve overlapping but different problems

Redis 8 Vector Sets are a native vectorset data type with built-in HNSW-style similarity search and lightweight JSON attributes. Redis Search vector fields are secondary indexes over HASH/JSON documents and can use FLAT, HNSW, or SVS-VAMANA together with richer text/tag/numeric/geo predicates.

8. Make the server feature surface observable

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin INFO serverdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin COMMAND INFO VADDdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin COMMAND INFO VSIMdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin COMMAND INFO FT.CREATEdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin COMMAND INFO FT.SEARCHdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user academy-admin ACL WHOAMI

Expected evidence includes redis_version:8.10.1 plus command metadata. That proves this endpoint exposes the commands; it does not prove your managed product tier, client library, or production topology supports identical behavior.

9. Tiny Vector Set: deterministic neighbors before benchmarking

docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch10:lesson1:vectorsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VADD atlasmart:ch10:lesson1:vectors VALUES 3 1 0 0 p1001 SETATTR '{"tenant":"tenant-a","category":"outdoor"}'docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VADD atlasmart:ch10:lesson1:vectors VALUES 3 0.90 0.25 0.05 p1002 SETATTR '{"tenant":"tenant-a","category":"travel"}'docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VADD atlasmart:ch10:lesson1:vectors VALUES 3 0 1 0 p1003 SETATTR '{"tenant":"tenant-b","category":"urban"}'docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VSIM atlasmart:ch10:lesson1:vectors VALUES 3 0.98 0.10 0.02 WITHSCORES COUNT 3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VSIM atlasmart:ch10:lesson1:vectors VALUES 3 0.98 0.10 0.02 WITHSCORES COUNT 3 TRUTH

With such a tiny set, approximate and exact results may be identical. That is useful for syntax and ranking evidence but not a performance claim. Build a larger representative fixture before interpreting recall/latency.

10. Security filter belongs before results leave retrieval

If AtlasMart tenants share one vector structure, retrieving globally and filtering tenant IDs in application code can expose unauthorized identifiers, scores, or attributes before the application discards them. Apply enforceable tenant/category constraints in the retrieval query—or use stronger physical/logical isolation—then verify with adversarial fixtures.

docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VSIM atlasmart:ch10:lesson1:vectors VALUES 3 0 1 0 WITHSCORES WITHATTRIBS COUNT 3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app VSIM atlasmart:ch10:lesson1:vectors VALUES 3 0 1 0 WITHSCORES WITHATTRIBS COUNT 3 FILTER '.tenant == "tenant-a"'

The first query can return tenant-b. The second should not. This is retrieval filtering evidence, not a substitute for ACL/network/application authorization.

11. Latency measurement needs percentiles and recall beside them

Record p50, p95, and p99 latency after warmup at a disclosed cardinality, dimension, quantization mode, concurrency, topology, persistence mode, and query mix. Then report recall@k against exact ground truth for the same query set. A faster HNSW configuration that loses required neighbors may be worse.

samples_ms=[0.82,0.85,0.88,0.91,0.95,1.03,1.10,1.24,1.51,2.20]def pct(xs,p):    xs=sorted(xs); i=(len(xs)-1)*p; lo=int(i); hi=min(lo+1,len(xs)-1); f=i-lo    return xs[lo]*(1-f)+xs[hi]*ffor p in (0.50,0.95,0.99): print(p, round(pct(samples_ms,p),3))# Demonstration numbers only; replace with your own measured query timings.

12. Reproducible lesson lab and cleanup

Run the dimension, exact-cosine, tiny Vector Set, and tenant-filter demonstrations. Verify dimension=3, cardinality=3, exact/approximate ranking on the tiny fixture, and absence of tenant-b in the filtered result.

docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch10:lesson1:vectors

13. Production judgment

Use vector retrieval only after the embedding model/version, dimension, numeric format, metric, and authorization policy are explicit. HNSW is approximate: monitor recall as well as latency. Vector memory includes encoded coordinates, graph links, element labels, and attributes; persistence/replication/failover move that state too. In Cluster or managed services, verify cross-shard/query-engine behavior and feature compatibility. Retries can duplicate ingestion updates unless element IDs are intentionally stable. Never store embedding API secrets in Redis vectors or lesson fixtures.

14. Summary and next step

You now have the retrieval mental model: embeddings come from outside Redis; dimensions and metrics are contracts; HNSW trades exhaustive work for speed; recall@k is not task relevance; and authorization must constrain retrieval. Lesson 2 turns that model into the native Redis 8 Vector Set command surface.

Authoritative references

• Redis Vector Sets — native Vector Set data type, commands, filtering, and examples
• VADD — Vector Set insertion, quantization, REDUCE, EF, M, and attributes
• VSIM — similarity queries, scores, filters, EF, TRUTH, and NOTHREAD
• VINFO — Vector Set implementation and configuration evidence
• VEMB — stored/reconstructed vector evidence
• VSETATTR — JSON attributes attached to Vector Set elements
• Vector Set memory optimization — Q8/BIN/NOQUANT, dimensions, graph links, and memory tradeoffs
• Vector Set performance — quantization and vector-set performance considerations
• Redis vector search concepts — Search FLAT/HNSW/SVS-VAMANA vector indexes and runtime parameters
• Vector field options — Search vector field types, metrics, and index algorithms
• FT.CREATE — Search schema and vector field creation
• FT.SEARCH — KNN/hybrid query syntax and parameters
• Redis 8.10 commands — target-version command surface
• Redis 8.10 release notes — 8.10.1 security baseline including Vector Set fixes
• Redis 8 GA announcement — historical 8.0 Vector Set beta status and integrated Redis 8 capabilities