Elasticsearch Custom Analyzers

Built-in analyzers cover common cases, yet real data often demands special treatment. Product names contain HTML tags. Customer names carry accents. Part numbers use dashes that should not split the code. A custom analyzer lets you choose each processing step and tailor the result to your data.

The Three Building Blocks

Every analyzer combines three kinds of components in a fixed order. Character filters clean the raw text. One tokenizer cuts the text into tokens. Token filters polish each token.

Raw text
   │
   ▼
[ Character filters ]   0 or more   clean characters (remove HTML, swap symbols)
   │
   ▼
[ Tokenizer ]           exactly 1   cut text into tokens
   │
   ▼
[ Token filters ]       0 or more   lowercase, remove accents, drop words
   │
   ▼
Tokens stored in the index

A Worked Example

Imagine a shop that stores descriptions with markup and accented letters. Visitors type plain letters, so "cafe" must find "Café."

Input:   <b>Café Déjà Vu</b>

html_strip        →  Café Déjà Vu
standard tokenizer →  [Café] [Déjà] [Vu]
lowercase          →  [café] [déjà] [vu]
asciifolding       →  [cafe] [deja] [vu]

Defining the Analyzer

Analysis settings live inside the index settings. The request below defines the analyzer and the mapping that uses it.

PUT /shop
{
  "settings": {
    "analysis": {
      "analyzer": {
        "clean_text": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "description": {
        "type": "text",
        "analyzer": "clean_text"
      }
    }
  }
}

The order inside the "filter" list matters. Lowercase runs before asciifolding in this example, and each filter sees the output of the one before it.

Testing with the Analyze API

Never guess what an analyzer produces. The _analyze endpoint shows the exact tokens.

GET /shop/_analyze
{
  "analyzer": "clean_text",
  "text": "<b>Café Déjà Vu</b>"
}

Result tokens: cafe, deja, vu

Run the test with several sample texts, including odd ones with symbols, numbers, and mixed languages.

Custom Character Filters

The mapping character filter swaps characters, and the pattern_replace filter uses regular expressions. Both prepare text before tokenization. A part-number field can turn dashes into nothing so "AB-1024" and "AB1024" match each other.

"char_filter": {
  "remove_dash": {
    "type": "pattern_replace",
    "pattern": "-",
    "replacement": ""
  }
}

"AB-1024"  →  "AB1024"

Choosing a Tokenizer

Tokenizer	Behavior	Good For
standard	Splits on word boundaries and punctuation	General text
whitespace	Splits only on spaces	Codes with symbols
keyword	Keeps the whole text as one token	Exact values with cleanup
pattern	Splits where a regular expression matches	Custom separators
path_hierarchy	Builds tokens for each folder level	File paths, categories

path_hierarchy on "/books/science/physics":

  /books
  /books/science
  /books/science/physics

Useful Token Filters

Filter	Job
lowercase	Converts letters to lower case
asciifolding	Replaces accented letters with plain ones
stop	Removes common words
stemmer	Reduces words to a root form
trim	Removes extra spaces around tokens
unique	Drops repeated tokens

Separate Analyzers for Indexing and Searching

A field can use one analyzer when storing text and a different one when reading queries. The "search_analyzer" setting enables this split. Autocomplete features depend on it, as a later topic shows.

"title": {
  "type": "text",
  "analyzer": "clean_text",
  "search_analyzer": "standard"
}

Normalizers for Keyword Fields

Keyword fields skip tokenizing, so analyzers do not apply. A normalizer offers a lighter option. It runs character filters and a limited set of token filters, such as lowercase, and keeps the value as a single token. Use it when "Delhi," "delhi," and "DELHI" should count as one value in filters and aggregations.

"analysis": {
  "normalizer": {
    "lower_only": { "type": "custom", "filter": ["lowercase"] }
  }
}

"city": { "type": "keyword", "normalizer": "lower_only" }

Changing an Existing Analyzer

Existing documents keep the tokens they received at indexing time. Changing an analyzer on a live field does not rewrite them. Create a new index with the updated settings, copy the data across with the Reindex API, and switch your application to the new index.

Design Checklist

Start from the questions users ask. Decide which differences should not matter, such as capital letters, accents, and dashes. Pick filters that remove those differences. Test the analyzer with real samples from your data, and keep the chain short, since every extra step adds indexing work.

Reuse analyzers across fields when the rules match. A single "clean_text" definition can serve titles, descriptions, and notes, which keeps behavior consistent and the settings compact.