Analyze text #

The analysis workbench: run the server’s tokenizers, normalizers, token filters and analyzers over sample text and see the exact token stream — the same components the indexing path uses, so what you debug here is exactly what collections run. The analysis chain decides what is searchable: query and document must reduce to the same tokens.

How it works #

Every analyzer is assembled from three stages, run in this order:

1. Normalizers rewrite the raw text as a whole, before tokenization (lowercase, html_strip, stconvert_t2s, …).
2. Tokenizer splits the rewritten text into tokens.
3. Token filters transform the stream one token at a time, in order (stopwords, asciifolding, …). A filter may delete a token or emit extras (synonyms, n-grams); extras continue with the next filter, so self-expanding filters cannot loop.

There is no separate char-filter stage — pre-tokenization transforms are normalizers. They also cover the keyword field path: a keyword field is never tokenized, it just runs one normalizer (e.g. lowercase) over the whole value, so the field stays one exact term that term queries hit case-insensitively.

The engine core provides the standard/whitespace tokenizers, the lowercase normalizer, the lowercase/uppercase/stopwords filters, and the standard and whitespace analyzers. The rest of the catalog comes from the ~40 plugin crates under contrib/analysis-*, registered at startup into the same registry this API reads — so workbench results equal indexing results.

Collections define named analyzers at creation time under settings.analysis.analyzer.custom; fields reference them by name like a built-in (see custom analyzers). The bundled articles dataset ships one, folding accents on bodies:

"content_an": {
  "tokenizer": "standard",
  "token_filters": ["asciifolding"]
}

…used by "body": { "type": "text", "analyzer": "content_an" } — so cafe matches café in bodies, while titles index with standard.

List components #

GET /_analysis/components

Every registered component — the set a schema may reference. The server ships 83 analyzers, 50 tokenizers, 15 normalizers and 327 token filters; the full catalog.

Examples #

Run a named analyzer and inspect the token stream:

POST /_analysis/analyze
{
  "analyzer": "standard",
  "text": "The Quick Brown Fox"
}

Response — four tokens; offsets are UTF-16 code units (the JSON convention):

{
  "tokens": [
    { "token": "the",   "start_offset": 0,  "end_offset": 3,  "position": 0 },
    { "token": "quick", "start_offset": 4,  "end_offset": 9,  "position": 1 },
    { "token": "brown", "start_offset": 10, "end_offset": 15, "position": 2 },
    { "token": "fox",   "start_offset": 16, "end_offset": 19, "position": 3 }
  ]
}

Or an ad-hoc pipeline — normalizers first, then tokenizer, then filters:

POST /_analysis/analyze
{
  "text": "Café Olé",
  "normalizer": ["lowercase"],
  "tokenizer": "standard",
  "filter": ["asciifolding"]
}

The pipeline yields cafe, ole. Named analyzers compose the same components with language tuning — english strips possessives, drops stop words and applies Krovetz stemming ("The Quick Foxes running" → quick, fox, run).

Add "explain": true to also get the snapshot after every stage (a named analyzer runs as-is — no detail comes back):

"detail": [
  { "name": "normalizer: lowercase", "text": "café olé" },
  { "name": "tokenizer: standard", "tokens": [ ... ] },
  { "name": "filter: 1/1 asciifolding", "tokens": [ ... ] }
]

An unknown name in the chain fails with a 400 naming it — no silent skip.

Use cases #

• “Why doesn’t it match?” Run the field’s analyzer over the indexed text and over the query text; where the streams disagree is the relevance bug no amount of query tweaking fixes.
• Mixed CJK and Latin text. standard emits one token per CJK ideograph and whole Latin words (Pizza搜索引擎 → pizza, 搜, 索, 引, 擎); for word-level CJK use a dictionary-backed plugin analyzer (ik_smart, jieba, kuromoji).
• Folding and noise. lowercase + asciifolding make CAFE, café and cafe one term; stop filters keep high-frequency words out of the index — smaller posting lists, cleaner scoring.
• Pinyin and traditional/simplified. The pinyin and stconvert plugins index Chinese text so queries in either script or romanized form still match — see the catalog.

Parameters for _analysis/analyze #

• text
  (Required, string) The text to analyze.
• analyzer
  (Optional, string) A named analyzer; overrides the pipeline fields.
• tokenizer
  (Optional, string) Tokenizer for an ad-hoc pipeline; default standard.
• normalizer
  (Optional, array) Normalizers applied to the raw text, in order.
• filter
  (Optional, array) Token filters applied in order.
• explain
  (Optional, boolean) Per-stage snapshots; ad-hoc only. Default false.

Related #

• Built-in analysis components — the full catalog this API draws from.
• match — the query that consumes these tokens: analyzer output becomes term clauses and scores.
• Analysis parameters — analyzer, search_analyzer and normalizer on schema fields.
• Create collection — custom analyzers.