Java整合elasticsearch开发环境搭建
Elasticsearch是一个基于Lucene的搜索框架,它的结果都是通过json展示。最近需要用到(做为一个Java开发,现在才用有点晚吧,汗),这里记录一下搭建过程中的坑。
我的项目是一个基于Maven的springboot项目
本地安装es
Mac上安装es的方法很简单,但是过程很漫长。因为用到brew,下载太慢了。
在命令行执行brew install elasticsearch一直等着就可以了,大概十几分钟半个小时以后就安装好了。安装好以后,可以任意目录直接执行elasticsearch命令启动es服务:
SheldondeMacBook-Pro ~ > elasticsearch
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory
[2019-06-27T00:54:09,853][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [36.4gb], net total_space [233.4gb], types [apfs]
[2019-06-27T00:54:09,859][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [sKPNtnmYROOwLeNPFWKVLw]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node ] [node-1] version[6.8.1], pid[64336], build[oss/tar/1fad4e1/2019-06-18T13:16:52.517138Z], OS[Mac OS X/10.14.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node ] [node-1] JVM arguments [...]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService ] [node-1] loaded module [tribe]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2019-06-27T00:54:13,343][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and host providers [settings]
[2019-06-27T00:54:13,726][INFO ][o.e.n.Node ] [node-1] initialized
[2019-06-27T00:54:13,726][INFO ][o.e.n.Node ] [node-1] starting ...
[2019-06-27T00:54:18,917][INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-06-27T00:54:21,982][INFO ][o.e.c.s.MasterService ] [node-1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300}
[2019-06-27T00:54:21,987][INFO ][o.e.c.s.ClusterApplierService] [node-1] new_master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-06-27T00:54:22,007][INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-06-27T00:54:22,007][INFO ][o.e.n.Node ] [node-1] started
[2019-06-27T00:54:22,182][INFO ][o.e.g.GatewayService ] [node-1] recovered [3] indices into cluster_state
[2019-06-27T00:54:22,450][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[weather][1], [.kibana_1][0], [weather][0]] ...]).
可以看到es版本是6.8.1,也可以通过命令elasticsearch –version查看。
elasticsearch这个命令的位置可以通过which elasticsearch查看。但是elasticsearch的配置文件位置在/usr/local/Cellar/elasticsearch/6.8.1/libexec/config
启动es以后,可以通过http://localhost:9200查看状态:
{
"name" : "node-1",
"cluster_name" : "elasticsearch_xxx",
"cluster_uuid" : "JOzy6LrwRxCtmHqqemG2kw",
"version" : {
"number" : "6.8.1",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "1fad4e1",
"build_date" : "2019-06-18T13:16:52.517138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
注意看minimum_wire_compatibility_version这里,最低兼容到5.6.0,所以更小的版本客户端连接不上。
接下来安装kibana,这是es官方提供的一个es分析工具。执行brew install kibana即可,也要等好久。
这里最后出现了一个问题,brew安装的es和kibana版本和服务器的es版本不兼容,导致kibana无法连接服务器的es
kibana的配置文件在/usr/local/etc/kibana下面
通过kibana命令可以启动,启动后通过http://localhost:5601/访问。它默认连接的是本地es,因为刚安装没有数据,所以啥也干不了。
整合项目
配置文件
在配置文件application.properties中增加
spring.data.elasticsearch.clusterNodes=localhost:9300
spring.data.elasticsearch.clusterName=elasticsearch_xxx
clusterName可以在http://localhost:9200中看到。
maven 依赖
从其他项目复制了依赖
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.5.0</version>
<exclusions>
<exclusion>
<artifactId>elasticsearch</artifactId>
<groupId>org.elasticsearch</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
</dependency>
不知道为啥先排除又自己引用,难道想让transport和elasticsearch版本不一致?可以试一下版本不一样比如把elasticsearch版本设成5.6.0而transport是5.5.0会报错AbstractMethodError。
所以上面的可以写成
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.6.0</version>
</dependency>
这里使用了5.6.0,因为前面看到最低兼容5.6.0。
问题排查
- availableProcessors is already set to [8], rejecting [8]
如果出现这个问题(网上出现的挺多),需要在启动前执行System.setProperty(“es.set.netty.runtime.available.processors”, “false”)。我加在这里:
System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(Bootstrap.class, args);
- NoSuchMethodError: io.netty.util.internal.ObjectUtil.checkPositive
netty版本问题,统一项目内的netty版本:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.25.Final</version>
</dependency>
</dependencies>
</dependencyManagement>
结语
现在(应该)可以读写es了。出现的问题一般都是jar的版本不匹配,所以遇到问题尽量统一管理jar版本。
