Vector Evaluation Metrics

A vector search system that feels fast and returns plausible results can still perform poorly. Measurement replaces guesswork. Evaluation metrics tell you whether the right documents appear, in the right order, at an acceptable speed.

Build a Test Set First

Every metric needs ground truth. Create a list of real questions, and for each question note which documents truly answer it.

TEST SET EXAMPLE

| Question                         | Correct documents |
|----------------------------------|-------------------|
| How do I reset my password?      | doc_12            |
| Can I get a refund after 30 days?| doc_45, doc_46    |
| Where is my invoice?             | doc_08            |

Aim for 50 to 200 questions that match real user behavior.

Quality Metrics

Recall@K

Recall@K asks how many of the correct documents appear in the top K results.

Question has 2 correct documents.
Top 5 results contain 1 of them.

Recall@5 = 1 / 2 = 50%

Precision@K

Precision@K asks how many of the top K results are correct.

Top 5 results contain 2 correct documents.

Precision@5 = 2 / 5 = 40%

Hit Rate

Hit rate records whether at least one correct document appears in the top K. Many question-answering systems care most about this score because one good document often suffices.

Mean Reciprocal Rank

Mean reciprocal rank, or MRR, rewards systems that place the first correct answer near the top.

Correct answer at rank 1  -> score 1/1 = 1.00
Correct answer at rank 2  -> score 1/2 = 0.50
Correct answer at rank 5  -> score 1/5 = 0.20

MRR = average of these scores across all questions

nDCG

Normalized discounted cumulative gain, or nDCG, handles results with different relevance levels, such as "perfect", "useful", and "unrelated". It gives more credit to highly relevant documents placed higher.

Which Metric Fits Which Goal

Speed and Capacity Metrics

• Latency p50: The time within which half of all queries finish.
• Latency p95 and p99: The time within which 95 or 99 percent of queries finish. These numbers reveal the slow experiences users remember.
• Queries per second: The number of searches the system handles at once.
• Index build time: The time needed to load and index new data.
• Memory use: The RAM that vectors and indexes occupy.

WHY AVERAGES MISLEAD

  100 queries: 95 finish in 40 ms, 5 finish in 2,000 ms
  Average = about 138 ms   (looks fine)
  p99     = 2,000 ms       (shows the real pain)

Running an Evaluation

1. Freeze the test set so every experiment uses identical questions.
2. Run each question through the system and record the top K results.
3. Calculate quality metrics and latency percentiles.
4. Change one setting, such as chunk size or index width.
5. Rerun the test and compare numbers side by side.

Avoid These Mistakes

• Testing with questions written by the same person who built the system.
• Changing several settings at once and losing track of what helped.
• Reporting only averages for speed.
• Skipping re-evaluation after new data or a model change.

Metrics turn opinions into evidence. Build a solid test set, track a small group of numbers, and rerun the evaluation after every meaningful change.