Text Analysis and Inverted Indexes
7534 ワード
_source
object is not directly used when searching for docs 1. Analyzer
Analyzer processes text before data store
3 components of analyzer
Character filters
Tokenizer
Token filters
1-1. Character filters
Adds, removes, or changes characters
1-2. Tokenizers
an analyzer contains one tokenizer. Tokenizing string into tokens
1-3. Token filters
Receive output of tokenizer as input. Token filters add, remove, or modify tokens
1-4. Default behavior of standard analyzer
![](https://s1.md5.ltd/image/82588f6fa346eb8a70d8636adf1a5d86.png)
2. Analyze API POST /_analyze
{
"text" : "2 guys wal into a bar, but the third... DUCKS! :-)",
"analyzer": "standard"
}
Output{
"tokens" : [
{
"token" : "2",
"start_offset" : 0,
"end_offset" : 1,
"type" : "<NUM>",
"position" : 0
},
{
"token" : "guys",
"start_offset" : 2,
"end_offset" : 6,
"type" : "<ALPHANUM>",
"position" : 1
},
{
"token" : "wal",
"start_offset" : 7,
"end_offset" : 10,
"type" : "<ALPHANUM>",
"position" : 2
},
{
"token" : "into",
"start_offset" : 11,
"end_offset" : 15,
"type" : "<ALPHANUM>",
"position" : 3
},
...
]
}
POST /_analyze
{
"text" : "2 guys wal into a bar, but the third... DUCKS! :-)",
"analyzer": "standard"
}
{
"tokens" : [
{
"token" : "2",
"start_offset" : 0,
"end_offset" : 1,
"type" : "<NUM>",
"position" : 0
},
{
"token" : "guys",
"start_offset" : 2,
"end_offset" : 6,
"type" : "<ALPHANUM>",
"position" : 1
},
{
"token" : "wal",
"start_offset" : 7,
"end_offset" : 10,
"type" : "<ALPHANUM>",
"position" : 2
},
{
"token" : "into",
"start_offset" : 11,
"end_offset" : 15,
"type" : "<ALPHANUM>",
"position" : 3
},
...
]
}
// same request as using standard analyzer
POST /_analyze
{
"text" : "2 guys wal into a bar, but the third... DUCKS! :-)",
"char_filter" : [],
"tokenizer" : "standard",
"filter" : ["lowercase"]
}
3. Inverted indexes
Field's values are stored in one of several data structures depending on it's data type, which ensures efficient data access
![](https://s1.md5.ltd/image/1b580db25f21b652ccd60768fcef1994.png)
inverted index enables efficient search of docs by term
inverted index contain many information including relevance scoring
(rank by how well doc match)
inverted index is created for each text field
![](https://s1.md5.ltd/image/7b9d4eb9ef04988518803487b1d63070.png)
fields with data type other than text uses different index data structures
Reference
この問題について(Text Analysis and Inverted Indexes), 我々は、より多くの情報をここで見つけました https://velog.io/@sangmin7648/Text-Analysis-and-Inverted-Indexesテキストは自由に共有またはコピーできます。ただし、このドキュメントのURLは参考URLとして残しておいてください。
Collection and Share based on the CC Protocol