Skip to content

E
elasticsearch-analysis-ik

myguguang / elasticsearch-analysis-ik
与 Fork 源项目一致

从无法访问的项目Fork

通知 5
Star 0
Fork 0
E
elasticsearch-analysis-ik

体验新版 GitCode，发现更多精彩内容 >>

提交 664e2b96 编写于 作者: S shikui
浏览文件
操作

Merge pull request #1 from medcl/master 

merged from head fork
上级 f5aef261 44fdf681
隐藏空白更改
内联 并排
Showing with 263 addition and 259 deletion
README.textile → README.md
浏览文件 @ 664e2b96
IK Analysis for ElasticSearch IK Analysis for ElasticSearch
================================== =============================
更新说明： The IK Analysis plugin integrates Lucene IK analyzer into elasticsearch, support customized dictionary.
对于使用es集群，用ik作为分词插件，经常会修改自定义词典，增加远程加载，每次更新都会重新加载词典，不必重启es服务。
Tokenizer: `ik`
更新：对于使用 ES 集群，用 IK 作为分词插件，经常会修改自定义词典的使用者，可以透过远程加载的方式，每次更新都会重新加载词典，不必重启 ES 服务。
The IK Analysis plugin integrates Lucene IK analyzer into elasticsearch, support customized dictionary.
Versions
Tokenizer: `ik` --------
Version IK version | ES version
------------- -----------|-----------
master | 1.5.0 -> master master | 1.5.0 -> master
1.4.0 | 1.6.0 1.4.0 | 1.6.0
1.3.0 | 1.5.0 1.3.0 | 1.5.0
1.2.9 | 1.4.0 1.2.9 | 1.4.0
1.2.8 | 1.3.2 1.2.8 | 1.3.2
1.2.7 | 1.2.1 1.2.7 | 1.2.1
1.2.6 | 1.0.0 1.2.6 | 1.0.0
1.2.5 | 0.90.2 1.2.5 | 0.90.2
1.2.3 | 0.90.2 1.2.3 | 0.90.2
1.2.0 | 0.90.0 1.2.0 | 0.90.0
1.1.3 | 0.20.2 1.1.3 | 0.20.2
1.1.2 | 0.19.x 1.1.2 | 0.19.x
1.0.0 | 0.16.2 -> 0.19.0 1.0.0 | 0.16.2 -> 0.19.0
Install
Thanks -------
-------------
YourKit supports IK Analysis for ElasticSearch project with its full-featured Java Profiler. you can download this plugin from RTF project(https://github.com/medcl/elasticsearch-rtf)
YourKit, LLC is the creator of innovative and intelligent tools for profiling https://github.com/medcl/elasticsearch-rtf/tree/master/plugins/analysis-ik
Java and .NET applications. Take a look at YourKit's leading software products: https://github.com/medcl/elasticsearch-rtf/tree/master/config/ik
<a href="http://www.yourkit.com/java/profiler/index.jsp">YourKit Java Profiler</a> and
<a href="http://www.yourkit.com/.net/profiler/index.jsp">YourKit .NET Profiler</a>. <del>also remember to download the dict files,unzip these dict file into your elasticsearch's config folder,such as: your-es-root/config/ik</del>
you need a service restart after that!
Install
------------- Configuration
you can download this plugin from RTF project(https://github.com/medcl/elasticsearch-rtf) -------------
https://github.com/medcl/elasticsearch-rtf/tree/master/plugins/analysis-ik
https://github.com/medcl/elasticsearch-rtf/tree/master/config/ik ### Analysis Configuration
<del>also remember to download the dict files,unzip these dict file into your elasticsearch's config folder,such as: your-es-root/config/ik</del> #### `elasticsearch.yml`
you need a service restart after that! ```yaml
index:
Dict Configuration (es-root/config/ik/IKAnalyzer.cfg.xml) analysis:
------------- analyzer:
ik:
https://github.com/medcl/elasticsearch-analysis-ik/blob/master/config/ik/IKAnalyzer.cfg.xml alias: [ik_analyzer]
type: org.elasticsearch.index.analysis.IkAnalyzerProvider
<pre> ik_max_word:
type: ik
<?xml version="1.0" encoding="UTF-8"?> use_smart: false
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> ik_smart:
<properties> type: ik
<comment>IK Analyzer 扩展配置</comment> use_smart: true
<!--用户可以在这里配置自己的扩展字典 --> ```
<entry key="ext_dict">custom/mydict.dic;custom/single_word_low_freq.dic</entry>
<!--用户可以在这里配置自己的扩展停止词字典--> Or
<entry key="ext_stopwords">custom/ext_stopword.dic</entry>
<!--用户可以在这里配置远程扩展字典 --> ```yaml
<entry key="remote_ext_dict">location</entry> index.analysis.analyzer.ik.type : "ik"
<!--用户可以在这里配置远程扩展停止词字典--> ```
<entry key="remote_ext_stopwords">location</entry>
</properties> you can set your prefer segment mode,default `use_smart` is false.
</pre> ### Mapping Configuration
Analysis Configuration (elasticsearch.yml) #### Quick Example
-------------
1. create a index
<Pre>
index: ```bash
analysis: curl -XPUT http://localhost:9200/index
analyzer: ```
ik:
alias: [ik_analyzer] 2. create a mapping
type: org.elasticsearch.index.analysis.IkAnalyzerProvider
ik_max_word: ```bash
type: ik curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'
use_smart: false {
ik_smart: "fulltext": {
type: ik "_all": {
use_smart: true "indexAnalyzer": "ik",
</pre> "searchAnalyzer": "ik",
Or "term_vector": "no",
<pre> "store": "false"
index.analysis.analyzer.ik.type : "ik" },
</pre> "properties": {
"content": {
you can set your prefer segment mode,default `use_smart` is false. "type": "string",
"store": "no",
Mapping Configuration "term_vector": "with_positions_offsets",
------------- "indexAnalyzer": "ik",
"searchAnalyzer": "ik",
Here is a quick example: "include_in_all": "true",
1.create a index "boost": 8
}
<pre> }
}
curl -XPUT http://localhost:9200/index }'
```
</pre>
3. index some docs
2.create a mapping
```bash
<pre> curl -XPOST http://localhost:9200/index/fulltext/1 -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
curl -XPOST http://localhost:9200/index/fulltext/_mapping -d' '
{ ```
"fulltext": {
"_all": { ```bash
"indexAnalyzer": "ik", curl -XPOST http://localhost:9200/index/fulltext/2 -d'
"searchAnalyzer": "ik", {"content":"公安部：各地校车将享最高路权"}
"term_vector": "no", '
"store": "false" ```
},
"properties": { ```bash
"content": { curl -XPOST http://localhost:9200/index/fulltext/3 -d'
"type": "string", {"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
"store": "no", '
"term_vector": "with_positions_offsets", ```
"indexAnalyzer": "ik",
"searchAnalyzer": "ik", ```bash
"include_in_all": "true", curl -XPOST http://localhost:9200/index/fulltext/4 -d'
"boost": 8 {"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
} '
} ```
}
}' 4. query with highlighting
</pre>
```bash
3.index some docs curl -XPOST http://localhost:9200/index/fulltext/_search -d'
{
<pre> "query" : { "term" : { "content" : "中国" }},
"highlight" : {
curl -XPOST http://localhost:9200/index/fulltext/1 -d' "pre_tags" : ["<tag1>", "<tag2>"],
{"content":"美国留给伊拉克的是个烂摊子吗"} "post_tags" : ["</tag1>", "</tag2>"],
' "fields" : {
"content" : {}
curl -XPOST http://localhost:9200/index/fulltext/2 -d' }
{"content":"公安部：各地校车将享最高路权"} }
' }
'
curl -XPOST http://localhost:9200/index/fulltext/3 -d' ```
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
' #### Result
curl -XPOST http://localhost:9200/index/fulltext/4 -d' ```json
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"} {
' "took": 14,
</pre> "timed_out": false,
"_shards": {
4.query with highlighting "total": 5,
"successful": 5,
<pre> "failed": 0
},
curl -XPOST http://localhost:9200/index/fulltext/_search -d' "hits": {
{ "total": 2,
"query" : { "term" : { "content" : "中国" }}, "max_score": 2,
"highlight" : { "hits": [
"pre_tags" : ["<tag1>", "<tag2>"], {
"post_tags" : ["</tag1>", "</tag2>"], "_index": "index",
"fields" : { "_type": "fulltext",
"content" : {} "_id": "4",
} "_score": 2,
} "_source": {
} "content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
' },
</pre> "highlight": {
"content": [
here is the query result "<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首 "
]
<pre> }
},
{ {
"took": 14, "_index": "index",
"timed_out": false, "_type": "fulltext",
"_shards": { "_id": "3",
"total": 5, "_score": 2,
"successful": 5, "_source": {
"failed": 0 "content": "中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"
}, },
"hits": { "highlight": {
"total": 2, "content": [
"max_score": 2, "均每天扣1艘<tag1>中国</tag1>渔船 "
"hits": [ ]
{ }
"_index": "index", }
"_type": "fulltext", ]
"_id": "4", }
"_score": 2, }
"_source": { ```
"content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
}, ### Dictionary Configuration
"highlight": {
"content": [ #### `config/ik/IKAnalyzer.cfg.xml`
"<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首 "
] ```xml
} <?xml version="1.0" encoding="UTF-8"?>
}, <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
{ <properties>
"_index": "index", <comment>IK Analyzer 扩展配置</comment>
"_type": "fulltext", <!--用户可以在这里配置自己的扩展字典 -->
"_id": "3", <entry key="ext_dict">custom/mydict.dic;custom/single_word_low_freq.dic</entry>
"_score": 2, <!--用户可以在这里配置自己的扩展停止词字典-->
"_source": { <entry key="ext_stopwords">custom/ext_stopword.dic</entry>
"content": "中韩渔警冲突调查：韩警平均每天扣1艘中国渔船" <!--用户可以在这里配置远程扩展字典 -->
}, <entry key="remote_ext_dict">location</entry>
"highlight": { <!--用户可以在这里配置远程扩展停止词字典-->
"content": [ <entry key="remote_ext_stopwords">location</entry>
"均每天扣1艘<tag1>中国</tag1>渔船 " </properties>
] ```
}
} ### 热更新 IK 分词使用方法
]
} 目前该插件支持热更新 IK 分词，通过上文在 IK 配置文件中提到的如下配置
}
```xml
</pre> <!--用户可以在这里配置远程扩展字典 -->
<entry key="remote_ext_dict">location</entry>
<!--用户可以在这里配置远程扩展停止词字典-->
have fun. <entry key="remote_ext_stopwords">location</entry>
```
热更新IK分词使用方法
---------- 其中 `location` 是指一个 url，比如 `http://yoursite.com/getCustomDict`，该请求只需满足以下两点即可完成分词热更新。
目前该插件支持热更新IK分词，通过上文在ik配置文件中提到的如下配置
1. 该 http 请求需要返回两个头部(header)，一个是 `Last-Modified`，一个是 `ETags`，这两者都是字符串类型，只要有一个发生变化，该插件就会去抓取新的分词进而更新词库。
<pre>
<!--用户可以在这里配置远程扩展字典 --> 2. 该 http 请求返回的内容格式是一行一个分词，换行符用 `\n` 即可。
<entry key="remote_ext_dict">location</entry>
<!--用户可以在这里配置远程扩展停止词字典--> 满足上面两点要求就可以实现热更新分词了，不需要重启 ES 实例。
<entry key="remote_ext_stopwords">location</entry>
</pre> have fun.
其中`location`是指一个url，比如`http://yoursite.com/getCustomDict`，该请求只需满足一下两点即可完成分词热更新。 常见问题
-------
1. 该http请求需要返回两个头部，一个是`Last-Modified`，一个是`ETags`，这两者都是字符串类型，只要有一个发生变化，该插件就会去抓取新的分词进而更新词库。
1.自定义词典为什么没有生效？
2. 该http请求返回的内容格式是一行一个分词，换行符用`\n`即可。
请确保你的扩展词典的文本格式为 UTF8 编码
满足上面两点要求就可以实现热更新分词了，不需要重启es实例。
2.如何手动安装，以 1.3.0 為例？（参考：https://github.com/medcl/elasticsearch-analysis-ik/issues/46）
常见问题：
------------- ```bash
1.自定义词典为什么没有生效？ git clone https://github.com/medcl/elasticsearch-analysis-ik
请确保你的扩展词典的文本格式为UTF8编码 cd elasticsearch-analysis-ik
mvn compile
2.如何手动安装，以 1.3.0 為例？（参考：https://github.com/medcl/elasticsearch-analysis-ik/issues/46） mvn package
plugin --install analysis-ik --url file:///#{project_path}/elasticsearch-analysis-ik/target/releases/elasticsearch-analysis-ik-1.3.0.zip
`git clone https://github.com/medcl/elasticsearch-analysis-ik` ```
`cd elasticsearch-analysis-ik`
`mvn compile` Thanks
`mvn package` ------
`plugin --install analysis-ik --url file:///#{project_path}/elasticsearch-analysis-ik/target/releases/elasticsearch-analysis-ik-1.3.0.zip` YourKit supports IK Analysis for ElasticSearch project with its full-featured Java Profiler.
\ No newline at end of file YourKit, LLC is the creator of innovative and intelligent tools for profiling
Java and .NET applications. Take a look at YourKit's leading software products:
<a href="http://www.yourkit.com/java/profiler/index.jsp">YourKit Java Profiler</a> and
<a href="http://www.yourkit.com/.net/profiler/index.jsp">YourKit .NET Profiler</a>.
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或