Key Components of Elasticsearch Scoring

Elasticsearch, built on top of Apache Lucene, utilizes Lucene’s robust scoring algorithms to determine the relevance of documents in response to a query. In this article, we will delve into the components of the scoring formula used by Elasticsearch, particularly the TF-IDF model, and illustrate it with a detailed example.

Key Components of Elasticsearch Scoring

1. Term Frequency (TF):

• This measures the number of times a term appears in a document. Higher term frequency generally increases the document's score because it indicates greater relevance.

2. Inverse Document Frequency (IDF):

• IDF measures how common or rare a term is across all documents in the index. Rare terms have higher IDF scores, which boosts the document's score because such terms are considered more informative.

3. Field Length Norm (Norm):

• Norm normalizes the score based on the length of the field. Shorter fields typically score higher because the term's presence is more significant in a shorter context.

4. Coordination Factor:

• This factor accounts for the number of query terms matched in the document. More matched terms result in a higher score.

5. Boost:

• Boost is a multiplier applied to certain fields to increase their importance in scoring.

Scoring Formula

Elasticsearch employs a variant of the TF-IDF (Term Frequency-Inverse Document Frequency) scoring model, specifically a modified version of the Okapi BM25 algorithm. The basic idea can be summarized as follows:

$$score(q,d)=∑t∈qIDF(t)×TF(t in d)×Norm(d)$$

Detailed Breakdown

Term Frequency (TF):

• If a term appears frequently in a document, it is considered more relevant. For example, if the term "poetry" appears many times in a document, that document will have a higher TF score for that term.

Inverse Document Frequency (IDF):

• If a term is rare across all documents, its IDF score is higher. For example, if "epic" is a rare term, documents containing this term will receive a higher score boost.

Field Length Norm (Norm):

• This factor normalizes the score to account for the length of the field. If the matching term is in a shorter field, it is considered more significant.

Coordination Factor:

• If a document matches more terms from the query, it is considered more relevant and scores higher.

Boost:

• Certain fields in your query might have a boost factor, making matches in those fields more significant. For instance, title^5 applies a high boost to matches in the title field.

Example Calculation

Let's say your query is "poetry epic", and you have the following document fields and terms:

• First Document:

  • Term: "poetry" (TF = 10, IDF = 5)

  • Term: "epic" (TF = 8, IDF = 7)

  • Field length normalization factor: 0.75

  • Coordination factor: Matches 2 query terms

$$score(q,d1)=(10×5)×0.75+(8×7)×0.75=37.5+42=79.5$$

• Second Document:

  • Term: "poetry" (TF = 8, IDF = 5)

  • Term: "epic" (TF = 7, IDF = 7)

  • Field length normalization factor: 0.80

  • Coordination factor: Matches 2 query terms

$$score(q,d2)=(8×5)×0.80+(7×7)×0.80=32+39.2=71.2$$

• Third Document:

  • Term: "poetry" (TF = 7, IDF = 5)

  • Term: "epic" (TF = 6, IDF = 7)

  • Field length normalization factor: 0.85

  • Coordination factor: Matches 2 query terms

$$score(q,d3)=(7×5)×0.85+(6×7)×0.85=29.75+35.7=65.45$$

In this simplified example, you can see how the scores for each document are calculated based on term frequency, inverse document frequency, field length normalization, and coordination factor. The actual scores might be more complex due to additional factors like boost values and more sophisticated normalization techniques used by Elasticsearch.

Practical Verification

To see the actual calculation of scores, you can use the explain API in Elasticsearch, which provides a detailed explanation of how each document’s score was computed:

GET /your_index/_explain/<document_id>
{
  "query": {
    "match": {
      "text": "poetry epic"
    }
  }
}

Replace <document_id> with the actual document ID to get a detailed breakdown of the scoring process for that document. This can help you understand why certain documents receive higher scores than others.

Example Output from Explain API

• First Document:

{
  "_index" : "your_index",
  "_type" : "_doc",
  "_id" : "1",
  "matched" : true,
  "explanation" : {
    "value" : 79.5,
    "description" : "sum of:",
    "details" : [
      {
        "value" : 37.5,
        "description" : "weight(poetry in 1) [PerFieldSimilarity], result of:",
        "details" : [
          {
            "value" : 37.5,
            "description" : "score(freq=10.0), computed as boost * idf * tf from:",
            "details" : [
              {
                "value" : 5.0,
                "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                "details" : [
                  {
                    "value" : 4,
                    "description" : "n, number of documents containing term",
                    "details" : [ ]
                  },
                  {
                    "value" : 100,
                    "description" : "N, total number of documents with field",
                    "details" : [ ]
                  }
                ]
              },
              {
                "value" : 10.0,
                "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                "details" : [
                  {
                    "value" : 10.0,
                    "description" : "freq, occurrences of term within document",
                    "details" : [ ]
                  }
                ]
              }
            ]
          }
        ]
      },
      {
        "value" : 42.0,
        "description" : "weight(epic in 1) [PerFieldSimilarity], result of:",
        "details" : [
          {
            "value" : 42.0,
            "description" : "score(freq=8.0), computed as boost * idf * tf from:",
            "details" : [
              {
                "value" : 7.0,
                "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                "details" : [
                  {
                    "value" : 2,
                    "description" : "n, number of documents containing term",
                    "details" : [ ]
                  },
                  {
                    "value" : 100,
                    "description" : "N, total number of documents with field",
                    "details" : [ ]
                  }
                ]
              },
              {
                "value" : 8.0,
                "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                "details" : [
                  {
                    "value" : 8.0,
                    "description" : "freq, occurrences of term within document",
                    "details" : [ ]
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

In this explanation, the scores for each term are detailed, showing how term frequency, inverse document frequency, and field length normalization contribute to the final score.

Conclusion

By understanding these principles and using the explain API, you can gain deeper insights into the scoring process in Elasticsearch. This knowledge allows you to fine-tune your queries to achieve the desired ranking for your documents, ensuring that the most relevant results are returned for your searches. With this approach, you can enhance the search experience in your applications and deliver more precise and meaningful results to your users.