分組聚合及嵌套查詢
聚合查詢可以理解為SQL中的求和、求最大值、最小值以及求均值的需求
嵌套可以理解為es存值的某一個(gè)字段為對(duì)象屬性的值做處理.
Elasticsearch Java API分組與聚合結(jié)合
- 其中對(duì)字段field_one進(jìn)行分組,分組的別名為fieldOne,取2^31-1組數(shù)據(jù).如果不設(shè)置size,查詢的結(jié)果會(huì)返回默認(rèn)size大小.
AggregationBuilder oneAgg =
AggregationBuilders.terms(“fieldOne”) .field(“field_one”).size(2^31-1);
- 需要對(duì)多個(gè)字段分組
// 再創(chuàng)建一個(gè)
AggregationBuilder twoAgg
= AggregationBuilders.terms(“fieldTwo”) .field(“field_two”).size(2^31-1);
// 將兩個(gè)分組結(jié)合
searchSourceBuilder.aggregation(oneAgg.subAggregation(twoAgg));
- 需要將分組結(jié)果的其他字段再進(jìn)行統(tǒng)計(jì)的sum、min、max、avg聚合
AggregationBuilder threeAgg =
AggregationBuilders.sum(“fieldThree”).field(“field_three”);
// 將數(shù)據(jù)通過field_one、field_two字段進(jìn)行分組,對(duì)field_three進(jìn)行求和聚合
searchSourceBuilder.aggregation(oneAgg.subAggregation(twoAgg.subAggregation(threeAgg)));
- 只聚合不分組
searchSourceBuilder.aggregation(threeAgg);
簡(jiǎn)單的分組查詢
// 分頁(yè)設(shè)置
private Integer start = 0;
private Integer length = 20;
EsPage esPage = new EsPage(start,length);
// 排序設(shè)置
List<EsSort> esSorts = Lists.newArrayList(new EsSort("userId", SortOrder.ASC),
new EsSort(id, SortOrder.DESC));
// 查詢條件
BoolQueryBuilder queryBuilder = new BoolQueryBuilder();
queryBuilder.must(new TermQueryBuilder("userName","xxxx" ));
// 組裝查詢對(duì)象
SearchRequest searchRequest = new SearchRequest("user_list");
// 1、添加普通查詢條件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
.query(queryBuilder)
// 聚合無視query篩選出的數(shù)據(jù)
.size(0)
.from(0)
.fetchSource(false);
// 組裝聚合查詢的條件
TermsAggregationBuilder aggregationBuilder = AggregationBuilders.terms("group_by_userId")
// 需要查詢的字段
.field("userId")
// 分頁(yè)
.size(page.getPage() + page.getSize());
// 2、添加聚合查詢條件
searchSourceBuilder.aggregation(aggregationBuilder);
// 3、添加排序條件
searchSourceBuilder.sort(sort.getSortFiled(), sort.getSortOrder());
searchRequest.source(searchSourceBuilder);
try {
SearchResponse searchResponse = restClient.search(searchRequest, RequestOptions.DEFAULT);
// 獲取總數(shù)
long sumTotal = searchResponse.getHits().getTotalHits().value;
if (sumTotal >= MAX_PAGE_DEPTH) {
throw new EsDataException("目前暫不支持超過10000條的數(shù)據(jù)進(jìn)行視圖聚合，請(qǐng)修改聚合任務(wù)的數(shù)量");
}
// sumTotal:查詢出來的總數(shù)
// searchResponse.getAggregations():聚合之后查詢出來的數(shù)據(jù)
return Pair.of(sumTotal, searchResponse.getAggregations());
} catch (IOException e) {
EsExceptionUtil.dealIOException(e);
} catch (ElasticsearchException e1) {
EsExceptionUtil.dealElasticsearchException(e1);
}
解析查聚合結(jié)果
// 查詢出來的總數(shù)
int count = pair.getLeft().intValue();
// 聚合之后查詢出來的數(shù)據(jù)
Aggregations aggregations = pair.getRight();
// 根據(jù)聚合時(shí)的聚合名稱,獲取數(shù)據(jù)
Terms terms = aggregations.get("group_by_userId");
List<Long> userIds = Lists.newArrayList();
if (terms != null && CollectionUtils.isNotEmpty(terms.getBuckets())) {
// terms.getBuckets():就是聚合之后的每條數(shù)據(jù)桶
for (Terms.Bucket bucket : terms.getBuckets()) {
// 遍歷得到桶中的key,就是聚合后,查詢的字段L:userId
long key = NumberUtils.toLong(bucket.getKeyAsString(), 0);
if (key == 0) {
continue;
}
userIds.add(key);
}
}
上面是最簡(jiǎn)單的一層聚合查詢,其實(shí)其他復(fù)雜的聚合也都是大同小異,就是一個(gè)套娃的操作.
復(fù)雜聚合查詢
public Map<String, Map<String, Long>> aggTwoArgs(String indices, QueryBuilder queryBuilder, String args1, String args2, int i) throws IOException {
Map<String, Map<String, Long>> map = new HashMap<>();
//設(shè)置要查詢的索引
SearchRequest request = new SearchRequest().indices(indices);
//構(gòu)建搜索
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
//添加搜索長(zhǎng)度
sourceBuilder.size(0);
//添加搜索條件
sourceBuilder.query(queryBuilder);
//設(shè)置要聚合的字段以及條數(shù)
//設(shè)置該次聚合的名稱 terms(args1)
//以及要聚合的字段field(args1 + ".keyword") 添加keyword是對(duì)字段進(jìn)行不分詞查詢。
TermsAggregationBuilder agg1 = AggregationBuilders.terms(args1).field(args1 + ".keyword").size(i);
//設(shè)置子聚合以及條數(shù)，設(shè)置返回?cái)?shù)據(jù)中該聚合的名稱 terms(args2)，以及要聚合的字段field(args2 + ".keyword")
TermsAggregationBuilder agg2 = AggregationBuilders.terms(args2).field(args2 + ".keyword").size(i);
//合并子聚合
agg1.subAggregation(agg2);
//添加聚合查詢
sourceBuilder.aggregation(agg1);
//創(chuàng)建請(qǐng)求
request.source(sourceBuilder);
//發(fā)送請(qǐng)求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
Aggregations aggregations = response.getAggregations();
// 取第一個(gè)分組
Terms terms1 = aggregations.get(args1);
for (Terms.Bucket bucket1 : terms1.getBuckets()) {
// 取分組下的分組
Terms terms2 = bucket1.getAggregations().get(args2);
Map<String, Long> map1 = new HashMap<>();
for (Terms.Bucket bucket2 : terms2.getBuckets()) {
map1.put(bucket2.getKeyAsString(), bucket2.getDocCount());
}
// bucket1.getKeyAsString():第一個(gè)分組的值
// map1:第二個(gè)分組下的map值
map.put(bucket1.getKeyAsString(), map1);
}
return map;
}
ElasticSearch Java API嵌套文章來源：http://www.zghlxwxcb.cn/news/detail-503681.html
public static final String MAPPING = ""
+ "{\n"
+ " \"properties\": {\n"
+ " \"itemAttributes\": {\n"
+ " \"type\": \"nested\",\n"
+ " \"properties\": {\n"
+ " \"specId\": {\n"
+ " \"type\": \"long\"\n"
+ " },\n"
+ " \"value\": {\n"
+ " \"type\": \"text\",\n"
+ " \"analyzer\": \"ngram_analyzer\",\n"
+ " \"fields\": {\n"
+ " \"value_keyword\": {\n"
+ " \"type\": \"keyword\"\n"
+ " }\n"
+ " }\n"
+ " },\n"
+ " \"key\": {\n"
+ " \"type\": \"long\"\n"
+ " },\n"
+ " \"ip\": {\n"
+ " \"type\": \"long\"\n"
+ " }\n"
+ " }\n"
+ " }\n"
+ " }\n"
+ "}";
上面ES的mapping中的itemAttributes字段就是一個(gè)嵌套式的數(shù)據(jù)結(jié)構(gòu).
所以上面的分組和聚合對(duì)itemAttributes對(duì)象下的屬性不適用,需要再做一層處理.文章來源地址http://www.zghlxwxcb.cn/news/detail-503681.html
// 對(duì)于嵌套的聚合,需要新建一個(gè)NestedAggregationBuilder 對(duì)象”nestedAgg”為別名
NestedAggregationBuilder nestedAggregationBuilder =
AggregationBuilders.nested(“nestedAgg”, “itemAttributes”);
// 分別對(duì)nested對(duì)象下的屬性建分組
AggregationBuilder specIdAgg =
AggregationBuilders.terms(“fieldSpecId”).field(“itemAttributes.specId”).size(2 ^ 31 - 1);
AggregationBuilder valueAgg =
AggregationBuilders.terms(“fieldValue”).field(“itemAttributes.value”).size(2 ^ 31 - 1);
AggregationBuilder keyAgg =
AggregationBuilders.terms(“fieldKey”).field(“itemAttributes.key”).size(2 ^ 31 - 1);
AggregationBuilder ipAgg =
AggregationBuilders.terms(“fieldIp”).field(“itemAttributes.ip”).size(2 ^ 31 - 1);
// 再將上面寫的一個(gè)個(gè)聚合體放入nestedAggregationBuilder,需要將上面的specIdAgg、valueAgg、keyAgg聚合相互之間有關(guān)聯(lián)需要一層一層關(guān)聯(lián)如下
nestedAggregationBuilder.subAggregation(
specIdAgg.subAggregation(valueAgg.subAggregation(ipAgg.subAggregation(
AggregationBuilders.sum(“keyAgg”).field(“itemAttributes.key”)))));
searchSourceBuilder.query(boolQueryBuilder).size(0);
searchSourceBuilder.aggregation(nestedAggregationBuilder);
searchRequest.source(searchSourceBuilder);
// 注意searchSourceBuilder.aggregation(nestedAggregationBuilder);
// 這個(gè)段代碼這是定義聚合方式的。
// searchSourceBuilder
//.aggregation(specIdAgg)
//.aggregation(valueAgg)
//.aggregation(ipAgg)
//.aggregation(keyAgg);
// 和上方聚合方式完全不一樣,單一將數(shù)據(jù)分組,聚合相互之間沒有關(guān)聯(lián).
try {
SearchResponse searchResponse = restHighLevelClient.search(searchRequest);
Nested nested = searchResponse.getAggregations().get(“nestedAgg”);
Aggregations nestedAggregations = nested.getAggregations();
Aggregation nestedAggregation = nestedAggregations.asList().get(0);
// 取第一個(gè)分組
List<? extends Terms.Bucket> specIdBuckets = (
(ParsedStringTerms) nestedAggregation).getBuckets();
for (Terms.Bucket specIdBucket : specIdBuckets ) {
// 第一個(gè)分組的值
String specId = specIdBucket .getKey().toString();
// 取第二個(gè)分組
List<Aggregation> valueAggregations = specIdBucket .getAggregations().asList();
List<? extends Terms.Bucket> valueBuckets = ((ParsedStringTerms) valueAggregations .get(0)).getBuckets();
for (Terms.Bucket valueBucket : valueBuckets ) {
// 第二個(gè)分組的值
String vlaue = valueBucket .getKey().toString();
// 取第三個(gè)分組
List<Aggregation> ipAggregations = valueBucket.getAggregations()
.asList();
List<? extends Terms.Bucket> ipBuckets =
((ParsedStringTerms)ipAggregations .get(0)).getBuckets();
for (Terms.Bucket ipBucket : ipBuckets ) {
// 第三個(gè)分組的值
String ip= ipBucket .getKey().toString();
// 第四個(gè)分組
Aggregation keyeAggregation = ipBucket .getAggregations().asList().get(0);
// 第四個(gè)分組的值
Integer count = (int) ((ParsedSum) keyAggregation ).getValue();
System.out.print(fieldOne);
System.out.print(filedtwo);
System.out.print(ip);
System.out.print(count );
}
}
}
} catch (IOException e) {
log.error("數(shù)據(jù)解析錯(cuò)誤:{}", e);
}
到了這里，關(guān)于elasticsearch中的聚合分組查詢的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
