Java整合elasticsearch开发环境搭建

发表于2019-06-27,长度7239, 492个单词, 20分钟读完
Flag Counter

Elasticsearch是一个基于Lucene的搜索框架,它的结果都是通过json展示。最近需要用到(做为一个Java开发,现在才用有点晚吧,汗),这里记录一下搭建过程中的坑。

我的项目是一个基于Maven的springboot项目

本地安装es

Mac上安装es的方法很简单,但是过程很漫长。因为用到brew,下载太慢了。

在命令行执行brew install elasticsearch一直等着就可以了,大概十几分钟半个小时以后就安装好了。安装好以后,可以任意目录直接执行elasticsearch命令启动es服务:

SheldondeMacBook-Pro ~ > elasticsearch
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory

[2019-06-27T00:54:09,853][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [36.4gb], net total_space [233.4gb], types [apfs]
[2019-06-27T00:54:09,859][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [sKPNtnmYROOwLeNPFWKVLw]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node               ] [node-1] version[6.8.1], pid[64336], build[oss/tar/1fad4e1/2019-06-18T13:16:52.517138Z], OS[Mac OS X/10.14.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-06-27T00:54:09,896][INFO ][o.e.n.Node               ] [node-1] JVM arguments [...]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2019-06-27T00:54:10,901][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2019-06-27T00:54:10,902][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [tribe]
[2019-06-27T00:54:10,903][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2019-06-27T00:54:13,343][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and host providers [settings]
[2019-06-27T00:54:13,726][INFO ][o.e.n.Node               ] [node-1] initialized
[2019-06-27T00:54:13,726][INFO ][o.e.n.Node               ] [node-1] starting ...
[2019-06-27T00:54:18,917][INFO ][o.e.t.TransportService   ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-06-27T00:54:21,982][INFO ][o.e.c.s.MasterService    ] [node-1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300}
[2019-06-27T00:54:21,987][INFO ][o.e.c.s.ClusterApplierService] [node-1] new_master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {node-1}{sKPNtnmYROOwLeNPFWKVLw}{dGM-m8BKS4yuXh9hXz64Jg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-06-27T00:54:22,007][INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-06-27T00:54:22,007][INFO ][o.e.n.Node               ] [node-1] started
[2019-06-27T00:54:22,182][INFO ][o.e.g.GatewayService     ] [node-1] recovered [3] indices into cluster_state
[2019-06-27T00:54:22,450][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[weather][1], [.kibana_1][0], [weather][0]] ...]).

可以看到es版本是6.8.1,也可以通过命令elasticsearch –version查看。

elasticsearch这个命令的位置可以通过which elasticsearch查看。但是elasticsearch的配置文件位置在/usr/local/Cellar/elasticsearch/6.8.1/libexec/config

启动es以后,可以通过http://localhost:9200查看状态:

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch_xxx",
  "cluster_uuid" : "JOzy6LrwRxCtmHqqemG2kw",
  "version" : {
    "number" : "6.8.1",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "1fad4e1",
    "build_date" : "2019-06-18T13:16:52.517138Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

注意看minimum_wire_compatibility_version这里,最低兼容到5.6.0,所以更小的版本客户端连接不上。

接下来安装kibana,这是es官方提供的一个es分析工具。执行brew install kibana即可,也要等好久。

这里最后出现了一个问题,brew安装的es和kibana版本和服务器的es版本不兼容,导致kibana无法连接服务器的es

kibana的配置文件在/usr/local/etc/kibana下面

通过kibana命令可以启动,启动后通过http://localhost:5601/访问。它默认连接的是本地es,因为刚安装没有数据,所以啥也干不了。

整合项目

配置文件

在配置文件application.properties中增加

spring.data.elasticsearch.clusterNodes=localhost:9300
spring.data.elasticsearch.clusterName=elasticsearch_xxx

clusterName可以在http://localhost:9200中看到。

maven 依赖

从其他项目复制了依赖

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>transport</artifactId>
    <version>5.5.0</version>
    <exclusions>
        <exclusion>
            <artifactId>elasticsearch</artifactId>
            <groupId>org.elasticsearch</groupId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
</dependency>

不知道为啥先排除又自己引用,难道想让transport和elasticsearch版本不一致?可以试一下版本不一样比如把elasticsearch版本设成5.6.0而transport是5.5.0会报错AbstractMethodError。

所以上面的可以写成

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>transport</artifactId>
    <version>5.6.0</version>
</dependency>

这里使用了5.6.0,因为前面看到最低兼容5.6.0。

问题排查

  1. availableProcessors is already set to [8], rejecting [8]

如果出现这个问题(网上出现的挺多),需要在启动前执行System.setProperty(“es.set.netty.runtime.available.processors”, “false”)。我加在这里:

System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(Bootstrap.class, args);
  1. NoSuchMethodError: io.netty.util.internal.ObjectUtil.checkPositive

netty版本问题,统一项目内的netty版本:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.25.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>

结语

现在(应该)可以读写es了。出现的问题一般都是jar的版本不匹配,所以遇到问题尽量统一管理jar版本。

Written on June 27, 2019
分类: dev, 标签: java elasticsearch
如果你喜欢,请赞赏! davelet