一、背景

公司有一批8萬的數(shù)據(jù)存儲(chǔ)在Mysql中，然后我使用多線程的方式調(diào)用Elasticsearch的bulk()方法推送到ES，但是在推送過程中出現(xiàn)了該問題，這屬于插入數(shù)據(jù)時(shí)產(chǎn)生的問題

二、異常

EVERE: Servlet.service() for servlet [default] in context with path [appBoot] threw exception [http://192.168.3.83:10014/api/kms-wiki/ElasticsearchService/createIndex] with root cause
ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1793)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1770)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1527)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
	at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/_bulk?timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]","bytes_wanted":511812774,"bytes_limit":510027366,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]","bytes_wanted":511812774,"bytes_limit":510027366,"durability":"TRANSIENT"},"status":429}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
		... 37 more

三、解決辦法

加大-Xms和-Xmx的值，比如docker-compose.yaml文件中可以這樣設(shè)置：

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    environment:
     - ES_JAVA_OPTS=-Xms1g -Xmx2g

四、解釋

1、異常分析

從上面錯(cuò)誤日志可以看出，現(xiàn)在是父熔斷器（parent）直接熔斷了，現(xiàn)在我們來介紹一下這幾種熔斷器作用

• parent circuit breaker（父熔斷器）：在所有斷路器上使用的總內(nèi)存量
• Field data circuit breaker（列數(shù)據(jù)熔斷器）
• Request circuit breaker(請(qǐng)求熔斷器)
• Accounting requests circuit breaker
• Script compilation circuit breaker（腳本編譯熔斷器）

2、查看父熔斷器占據(jù)JVM最大堆內(nèi)存的比例值

默認(rèn)值： 95%

要求： 大點(diǎn)好

查看方式：

請(qǐng)求方式：
GET

請(qǐng)求鏈接（ip和port都是Elasticsearch的；查看該接口返回值中的indices.breaker.total.limit參數(shù)值就是上面提到的比例值；如果沒有該值，那就是使用默認(rèn)值95%）：
http://ip:port/_cluster/settings?

設(shè)置方式：

請(qǐng)求方式：
PUT 

請(qǐng)求鏈接（其中ip和port都是Elasticsearch的）：
http://ip:port/_cluster/settings

請(qǐng)求體：
{
  "persistent": {
    "indices.breaker.total.limit": "95%"
  }
}

3、查看父熔斷器占據(jù)JVM堆的具體值

請(qǐng)求方式：
GET

請(qǐng)求鏈接（ip和port都是Elasticsearch的；查看該接口返回值中的parent的limit_size參數(shù)值，該值受到上面比例值的限制）：
http://ip:port/_nodes/stats/breaker

比如現(xiàn)在設(shè)置- ES_JAVA_OPTS=-Xms1g -Xmx2g，然后具體值是1.8g，我現(xiàn)在的父熔斷器占比是95%，所以1.8g是差不多的，如下：

文章來源地址http://www.zghlxwxcb.cn/news/detail-777564.html

4、解決查詢時(shí)導(dǎo)入的熔斷問題

• ES調(diào)優(yōu)2（bulk報(bào)錯(cuò)）

到了這里，關(guān)于解決Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large問題的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！