Built-in analysis components #

The server ships 83 analyzers, 50 tokenizers, 15 normalizers and 327 token filters, registered at startup. The authoritative, always-current list on YOUR node is:

GET /_analysis/components

Components are referenced by name — in a field’s schema ("analyzer": "english"), in a custom analyzer definition, or in an ad-hoc _analysis/analyze pipeline.

How it works #

The catalog is assembled in two layers. The engine core registers the minimal set that is always present: the standard and whitespace tokenizers, the lowercase normalizer, the lowercase/uppercase/ stopwords token filters, and the two named analyzers composed from them (standard, whitespace). Every other name below comes from the ~40 analysis plugin crates under contrib/analysis-*, registered at startup in one pass — foundation plugins first (core, stemmers, icu), then the CJK and per-language plugins, which intentionally override the basic analyzers from core with richer pipelines (language stop words, normalization, stemming).

Dictionary-backed components (kuromoji, nori, jieba, ik, smartcn) are registered lazily: they are built on first use, so a node whose schemas never reference them never pays the dictionary-load cost. Components are plain names in a registry — a schema, a custom analyzer definition and an _analysis/analyze request all resolve through the same one, which is what the workbench page relies on.

Examples #

The three places a component name can appear:

"history": { "type": "text", "analyzer": "english" }

"analysis": {
  "analyzer": {
    "custom": {
      "my_analyzer": {
        "normalizers": ["lowercase"],
        "tokenizer": "standard",
        "token_filters": ["asciifolding"]
      }
    }
  }
}

POST /_analysis/analyze
{ "text": "The Quick Foxes", "analyzer": "english" }

Analyzers #

General purpose:

fingerprint, keyword, pattern, simple, standard, stop, whitespace

CJK and transliteration:

cjk, ik_max_word, ik_smart, jieba, kuromoji, nori, pinyin, smartcn stconvert_s2t, stconvert_t2s

Language analyzers (stop words + stemming per language):

afrikaans, amharic, arabic, armenian, aymara, azerbaijani, basque, bengali brazilian, bulgarian, catalan, cherokee, croatian, czech, danish, dutch english, estonian, filipino, finnish, french, galician, georgian, german greek, guarani, hebrew, hindi, hungarian, indonesian, irish, italian khmer, latvian, lithuanian, malay, marathi, mongolian, nahuatl, navajo nepali, norwegian, persian, polish, portuguese, quechua, romanian, russian scottish_gaelic, serbian, slovak, slovenian, sorani, spanish, swahili, swedish tagalog, tamil, thai, tibetan, turkish, ukrainian, urdu, vietnamese welsh, yiddish

The CJK tokenizers (ik_max_word/ik_smart, jieba, smartcn, kuromoji, nori) load their dictionaries at runtime from the external <config>/analysis/ directory (make init-analysis-dicts); without the dictionary files the analyzers answer but degrade to simpler segmentation.

Tokenizers #

burmese, camel_case, char_group, chinese_char, classic, code, compound_word, edge_ngram elision, email, emoji, fingerprint, hyphenated, icu_tokenizer, ik_max_word, ik_smart jieba, json_field, keyword, kuromoji_tokenizer, letter, log, lowercase, markdown microblog, ngram, nori_tokenizer, path_hierarchy, pattern, phone_number, pinyin, punctuation reverse, script_boundary, sentence, simple_pattern, simple_pattern_split, sliding_window, smartcn_tokenizer, standard stconvert_s2t, stconvert_t2s, structured_id, tab_separated, thai, truncate, uax_url_email, url whitespace, wildcard

standard is Unicode-aware word segmentation: maximal alphanumeric runs, CJK ideographs as single-character tokens. whitespace splits on whitespace only.

Normalizers #

Applied to raw text BEFORE tokenization (keyword fields use them at index time via the field’s normalizer schema parameter):

collapse_whitespace, html_strip, lowercase, mapping, pattern_replace, pinyin, pinyin_first_letter, stconvert_s2t stconvert_t2s, trim, unicode_nfc, unicode_nfd, unicode_nfkc, unicode_nfkd, uppercase

Token filters #

327 filters covering stemming (33 snowball_* variants plus per-language light stemmers), stop words (per-language *_stop lists), case and folding (asciifolding, cjk_width, decimal_digit, icu_folding), n-grams and shingles (edge_ngram, ngram, fixed_shingle), word splitting (word_delimiter, camel_case_split, snake_case), synonyms (synonym, synonym_graph), security masking (credit_card_mask, email_mask, ip_mask, phone_mask, ssn_mask, redact), encoding (base64_encode, hex_encode, url_encode, rot13) and more. See GET /_analysis/components for the live list; the analyze API accepts any of them by name in the filter array.

Use cases #

Pick the general analyzer first. standard for prose, keyword for exact whole-value matching, whitespace when punctuation inside terms is meaningful (product codes, tags), pattern for custom delimiters.
Use a language analyzer, not a pile of filters. english, german, kuromoji (Japanese), … each bundle the right normalizer, stop list and stemmer for their language; recreate that by hand only when you genuinely need to deviate.
Search Chinese by pinyin or across scripts. The pinyin normalizers (pinyin, pinyin_first_letter) and the stconvert components (stconvert_s2t/stconvert_t2s) cover romanized and traditional/simplified matching without changing the source text.
Sanitize before you index. The masking and detection filters (credit_card_mask, email_mask, ssn_mask, redact, sql_injection_detect) keep sensitive or hostile content out of the searchable terms in the first place.

Analyze text — run any of these components over sample text and watch the token stream stage by stage.
match — how the tokens these components produce become term clauses, and/or semantics and BM25 scores.
Analysis parameters — binding analyzer, search_analyzer and normalizer to schema fields.
Create collection — composing any of these into per-collection custom analyzers. The plugins themselves live under contrib/analysis-* in the source tree.