本文指出，將BM25，向量檢索Embedding模型后近似KNN相結(jié)合，可以讓搜索引擎既能理解用戶查詢的字面意義，又能捕捉到查詢的深層次語義，從而提供更全面、更精確的搜索結(jié)果。這種混合方法在現(xiàn)代搜索引擎中越來越普遍，因?yàn)樗Y(jié)合了傳統(tǒng)搜索的精確性和基于AI的搜索的語義理解能力。然后在8.8引入Learned Sparse Encoder新特性，因?yàn)閐ense vector search 密集向量搜索通常需要在領(lǐng)域內(nèi)進(jìn)行重新訓(xùn)練。如果沒有在領(lǐng)域內(nèi)進(jìn)行重新訓(xùn)練，它們甚至可能表現(xiàn)不如傳統(tǒng)的詞匯評分，比如Elastic的BM25。

How to get the best of lexical and AI-powered search with Elastic’s vector database 如何利用Elastic的向量數(shù)據(jù)庫實(shí)現(xiàn)詞匯和人工智能搜索的最佳效果

作者

Bernhard Suhm 伯恩哈德·蘇姆

2023年7月3日

Maybe you came across the term “vector database” and are wondering whether it’s the new kid on the block of data retrieval systems. Maybe you are confused by conflicting claims about vector databases. The truth is, the approach used by vector databases has been around for a few years. If you’re looking for the best retrieval performance, hybrid approaches that combine keyword-based search (sometimes referred to as lexical search) with vector-based approaches represent the state of the art. 也許你曾經(jīng)聽說過“向量數(shù)據(jù)庫”這個術(shù)語，想知道它是否是數(shù)據(jù)檢索系統(tǒng)中的新生力量。也許你對向量數(shù)據(jù)庫的爭議性說法感到困惑。事實(shí)是，向量數(shù)據(jù)庫所采用的方法已經(jīng)存在幾年了。如果你正在尋找最佳的檢索性能，將基于關(guān)鍵詞的搜索（有時被稱為詞匯搜索）與基于向量的方法相結(jié)合的混合方法代表了最先進(jìn)的技術(shù)。

In Elasticsearch?, you can get the best of both worlds: lexical and vector search. Elastic? made lexical columnar retrieval popular, implemented in Lucene, and has been perfecting that approach for more than 10 years. In addition, for several years, Elastic has been investing in vector database capabilities, such as a native implementation of approximate nearest neighbor search using hierarchical navigable small world (HNSW) (available in 8.0 and later releases). 在Elasticsearch中，您可以兼得詞法和向量搜索的優(yōu)勢。Elastic將詞法列式檢索變得流行，該方法在Lucene中實(shí)現(xiàn)，并已經(jīng)完善了這種方法超過10年。此外，多年來，Elastic一直在投資于向量數(shù)據(jù)庫功能，例如使用分層可導(dǎo)航小世界（HNSW）的近似最近鄰搜索的本機(jī)實(shí)現(xiàn)（在8.0及更高版本中可用）。

Figure 1: Timeline of search innovations delivered by Elastic, including building up vector search capabilities since 2019 圖1：Elastic提供的搜索創(chuàng)新時間表，包括自2019年以來構(gòu)建向量搜索能力

Elastic is positioned to be a leader in the rapidly evolving vector database market: Elastic定位為快速發(fā)展的向量數(shù)據(jù)庫市場的領(lǐng)導(dǎo)者

• Fully performant and scalable vector database functionality, including storing embeddings and efficiently searching for nearest neighbor 完全高性能和可擴(kuò)展的矢量數(shù)據(jù)庫功能，包括存儲嵌入并高效搜索最近鄰

• A proprietary sparse retrieval model that implements semantic search out of the box 一個專有的稀疏檢索模型，可以直接實(shí)現(xiàn)語義搜索

• Industry-leading relevance of all types — keyword, semantic, and vector 所有類型的行業(yè)領(lǐng)先相關(guān)性——關(guān)鍵詞、語義和向量

• The ability to apply generative AI and enrich large language models (LLMs) with proprietary, business-specific data as context 將生成式人工智能應(yīng)用于大型語言模型（LLMs），并豐富其專有的、業(yè)務(wù)特定的數(shù)據(jù)作為上下文的能力

• All capabilities in a single platform: execute vector search, embed unstructured data into vector representations applying off-the-shelf and custom models, and implement search applications in production, complete with solutions for observability and security 在一個平臺上擁有所有功能：執(zhí)行向量搜索，將非結(jié)構(gòu)化數(shù)據(jù)嵌入向量表示中并應(yīng)用現(xiàn)成和定制模型，以及在生產(chǎn)環(huán)境中實(shí)現(xiàn)搜索應(yīng)用，包括可觀察性和安全性解決方案

In this blog, learn more about the concepts relating to vector databases, how they work, which use cases they apply to, and how you can achieve superior search relevance with vector search. 在這篇博客中，了解有關(guān)向量數(shù)據(jù)庫相關(guān)概念，它們的工作原理，適用的使用案例以及如何通過向量搜索實(shí)現(xiàn)更高的搜索相關(guān)性。

The basics of vector databases 矢量數(shù)據(jù)庫的基礎(chǔ)

Why is there so much attention on vector databases? 為什么向量數(shù)據(jù)庫受到如此多的關(guān)注？

A vector database is a term for a system capable of executing vector search. So to understand vector databases, let’s start with vector search and why it has garnered so much attention lately. 矢量數(shù)據(jù)庫是指能夠執(zhí)行矢量搜索的系統(tǒng)。因此，要理解矢量數(shù)據(jù)庫，讓我們從矢量搜索開始，以及為什么最近引起了如此多的關(guān)注。

Vector search plays an important role in recent discussions about how AI is transforming literally everything, from business workflows to education. Why does vector search play such an important role on this topic? First, vector search enables fast and accurate semantic search of unstructured data — without extensive curation of metadata, keywords, and synonyms. Second, vector search contributes a piece to the recent excitement around generative AI because it can provide accurate context from proprietary sources outside what LLMs “know” (i.e., have seen during their training). 矢量搜索在最近的討論中扮演著重要角色，討論的內(nèi)容涉及人工智能如何從業(yè)務(wù)工作流程到教育等方面進(jìn)行轉(zhuǎn)變。為什么矢量搜索在這個話題上扮演如此重要的角色呢？首先，矢量搜索能夠快速準(zhǔn)確地對非結(jié)構(gòu)化數(shù)據(jù)進(jìn)行語義搜索，而無需對元數(shù)據(jù)、關(guān)鍵詞和同義詞進(jìn)行大量整理。其次，矢量搜索為最近圍繞生成式人工智能的興奮情緒做出了貢獻(xiàn)，因?yàn)樗梢詮膶Ｓ衼碓粗刑峁?zhǔn)確的上下文，超出了大型語言模型“知道”的范圍（即在它們的訓(xùn)練過程中見過的內(nèi)容）。

What are vector databases used for? 矢量數(shù)據(jù)庫用于什么？

Most standard databases let you retrieve related information by matching on structured fields, including matching keywords in descriptions, and values in numeric fields. By contrast, a vector database captures the meaning of unstructured text and finds you “what you mean” instead of matching text — also known as semantic search. 大多數(shù)標(biāo)準(zhǔn)數(shù)據(jù)庫都可以通過匹配結(jié)構(gòu)化字段來檢索相關(guān)信息，包括匹配描述中的關(guān)鍵詞和數(shù)值字段中的數(shù)值。相比之下，向量數(shù)據(jù)庫捕捉了非結(jié)構(gòu)化文本的含義，并找到了您所指的內(nèi)容，而不是匹配文本，也被稱為語義搜索。

Additionally, vector databases allow you to: 此外，矢量數(shù)據(jù)庫還允許您：

• Search unstructured data other than text, including images or audio. Searches that involve more than one type of data have been referred to as “multimodal search” — like searching for images using a textual description. 搜索非結(jié)構(gòu)化數(shù)據(jù)，包括圖像或音頻。涉及多種類型數(shù)據(jù)的搜索被稱為“多模式搜索”——比如使用文本描述搜索圖像。

• Personalize user experiences by modeling user characteristics or behaviors in a statistical (vector) model and matching others against that. 通過在統(tǒng)計(jì)（向量）模型中對用戶特征或行為進(jìn)行建模，并將其他用戶與之匹配，個性化用戶體驗(yàn)。

• Create “generative” experiences, where the system doesn’t just return a list of documents related to a query the user issued, but engages the user in conversations, explains multi-step processes, and generates an interaction that goes well beyond perusing related information. 創(chuàng)建“生成式”體驗(yàn)，系統(tǒng)不僅僅返回與用戶查詢相關(guān)的文檔列表，而是與用戶進(jìn)行對話，解釋多步驟過程，并生成遠(yuǎn)遠(yuǎn)超出瀏覽相關(guān)信息的互動。

What is a vector database, and how does it work? 什么是矢量數(shù)據(jù)庫，它是如何工作的？

A vector database consists of two primary components: 一個向量數(shù)據(jù)庫由兩個主要組成部分組成：

1. Indexing and storing embeddings, which is what the multi-dimensional numeric representation of unstructured data is generally called. Embeddings are generated by deep neural networks that were trained to classify that type of unstructured data and capture the meaning, context, and associations of unstructured data in a “dense” vector, usually hundreds to thousands dimensions deep — the secret sauce of vector search. 索引和存儲嵌入，通常指的是非結(jié)構(gòu)化數(shù)據(jù)的多維數(shù)值表示。嵌入是由經(jīng)過訓(xùn)練以分類該類型非結(jié)構(gòu)化數(shù)據(jù)并在“密集”向量中捕捉非結(jié)構(gòu)化數(shù)據(jù)的含義、上下文和關(guān)聯(lián)的深度神經(jīng)網(wǎng)絡(luò)生成的，通常是數(shù)百到數(shù)千個維度深的 — 向量搜索的秘密武器。

2. A search algorithm that efficiently finds nearest neighbors in the high dimensional “embedding space,” where vector proximity means similarity in meaning. Different ways to search indices exist, also known as approximate nearest neighbor (ANN) search, with HNSW being one of the most commonly utilized algorithms by vector database providers. 在高維的“嵌入空間”中高效地找到最近鄰的搜索算法，其中向量的接近意味著相似的含義。存在不同的搜索索引方法，也被稱為近似最近鄰（ANN）搜索，HNSW是向量數(shù)據(jù)庫提供者最常用的算法之一。

Figure 2: Key components for executing vector search 圖2：執(zhí)行向量搜索的關(guān)鍵組件

Some vector databases only provide the capability to store and search embeddings, depicted as A in Figure 2 above. However, this approach leaves developers with the challenge of how to generate those embeddings. Typically, that requires access to an embedding model (shown as C) and an API to apply it to your data and queries (B). And you may be able to store only very limited meta data along with the embeddings, making it more complex to provide comprehensive information in your user application. 一些向量數(shù)據(jù)庫只提供存儲和搜索嵌入的功能，如上圖中的A所示。然而，這種方法讓開發(fā)人員面臨如何生成這些嵌入的挑戰(zhàn)。通常，這需要訪問一個嵌入模型（如C所示）和一個API來將其應(yīng)用于您的數(shù)據(jù)和查詢（B）。您可能只能存儲與嵌入一起的非常有限的元數(shù)據(jù)，這使得在用戶應(yīng)用程序中提供全面信息變得更加復(fù)雜。

Further, dedicated vector databases leave you to figure out how to integrate the search capability into your application, as alluded to on the right side in Figure 2. Managing the components of your software architecture to solve those challenges involves evaluating many solutions offered from different vendors with varying quality levels and support. 此外，專用的向量數(shù)據(jù)庫讓您自行解決如何將搜索功能整合到您的應(yīng)用程序中，如圖2右側(cè)所示。管理軟件架構(gòu)的各個組件以解決這些挑戰(zhàn)，涉及評估來自不同供應(yīng)商提供的許多解決方案，其質(zhì)量和支持程度各不相同。

Elastic as a vector database 彈性作為矢量數(shù)據(jù)庫

Elastic provides all capabilities you should expect from a vector database and more! Elastic提供了您從向量數(shù)據(jù)庫中所期望的所有功能，甚至更多！

In contrast to dedicated vector databases, Elastic supports three capabilities in a single platform that are critical to implement applications powered by vector search: storing embeddings (A), efficiently searching for nearest neighbor (B), and embedding text into vector representations (C). 與專用的向量數(shù)據(jù)庫相比，Elastic在單個平臺上支持三種關(guān)鍵能力，這對于實(shí)現(xiàn)基于向量搜索的應(yīng)用程序至關(guān)重要：存儲嵌入（A）、高效搜索最近鄰居（B）和將文本嵌入向量表示（C）。

This approach eliminates inefficiencies and complexities compared to accessing them through APIs, as necessary with other vector databases. Elastic implements approximate nearest neighbor search using HNSW natively in Lucene and lets you apply filtering (as pre-filtering, for accurate results) using an algorithm that switches between brute force and approximate nearest neighbors as appropriate (i.e., falls back to brute force when the pre-filter removes a large portion of the candidate list). 這種方法消除了與通過API訪問它們相比的低效和復(fù)雜性，這在其他矢量數(shù)據(jù)庫中是必要的。Elastic在Lucene中本地實(shí)現(xiàn)了使用HNSW的近似最近鄰搜索，并允許您應(yīng)用過濾（作為預(yù)過濾，以獲得準(zhǔn)確的結(jié)果），使用在適當(dāng)時切換在蠻力和近似最近鄰之間的算法（即，在預(yù)過濾移除候選列表的大部分時退回到蠻力）。

Use our market-leading Learned Sparse Encoder model or bring your own embedding model. Learn more about loading transformers created in PyTorch into Elastic in this blog. 使用我們市場領(lǐng)先的學(xué)習(xí)稀疏編碼器模型，或者使用您自己的嵌入模型。在這篇博客中了解如何將在PyTorch中創(chuàng)建的transformers加載到Elastic中。

Figure 3: Elastic combines all the components of a vector database in a single platform 圖3：Elastic將矢量數(shù)據(jù)庫的所有組件合并到一個平臺中

How to get optimal retrieval performance with vector search 如何通過向量搜索獲得最佳檢索性能

Challenges with implementing vector search 實(shí)施向量搜索時面臨的挑戰(zhàn)

To the heart of how to implement superior semantic search, let’s understand the challenges with (dense) vector search: 要實(shí)現(xiàn)卓越的語義搜索，讓我們先了解（密集）向量搜索的挑戰(zhàn)：

• Picking the right embedding model: Standard embedding models deteriorate out of domain, like the ones available off the shelf on public repositories — as shown, for example, in Table 2 of our earlier blog on benchmarking vector search. If you’re lucky, a pre-trained model works well enough for your use case, but generally you have to adapt them with domain data, which requires annotated data and expertise in training deep neural networks. You can find a blueprint for the model adaptation process in this blog. 選擇合適的嵌入模型：標(biāo)準(zhǔn)嵌入模型在領(lǐng)域外表現(xiàn)不佳，比如在公共存儲庫中提供的模型，正如我們早期博客中的表2所示，用于基準(zhǔn)向量搜索。如果你很幸運(yùn)，預(yù)訓(xùn)練模型對你的用例效果良好，但通常你需要用領(lǐng)域數(shù)據(jù)對其進(jìn)行調(diào)整，這需要有標(biāo)注數(shù)據(jù)和深度神經(jīng)網(wǎng)絡(luò)訓(xùn)練方面的專業(yè)知識。你可以在這篇博客中找到模型調(diào)整過程的藍(lán)圖。

Figure 4: Pre-trained embedding models deteriorate out-of-domain, compared to BM25 圖4：與BM25相比，預(yù)訓(xùn)練的嵌入模型在域外表現(xiàn)下降

• Implementing efficient filtering: In search and recommender systems, you are typically not done returning a list of relevant documents; users want to apply filters. Filtering for metadata with vector search is challenging: if you filter after running the vector search, you risk being left with too few (or no) results matching the filter conditions (known as “post filtering”). Otherwise, if you filter first, the nearest neighbor search isn’t as efficient because it’s performed on a small subset of the data, whereas the data structure used during vector search (like the HNSW graph) was created for the whole data set. Restricting the search scope to relevant vectors is — for many use-cases — an absolute necessity for providing a better customer experience. 實(shí)現(xiàn)高效的過濾：在搜索和推薦系統(tǒng)中，通常不能僅返回相關(guān)文檔列表；用戶希望能夠應(yīng)用過濾器。使用向量搜索進(jìn)行元數(shù)據(jù)過濾具有挑戰(zhàn)性：如果在運(yùn)行向量搜索后進(jìn)行過濾，您可能會冒著剩下與過濾條件匹配的結(jié)果太少（或沒有）的風(fēng)險（稱為“后過濾”）。否則，如果首先進(jìn)行過濾，最近鄰搜索效率不高，因?yàn)樗窃跀?shù)據(jù)的一個小子集上執(zhí)行的，而向量搜索期間使用的數(shù)據(jù)結(jié)構(gòu)（如HNSW圖）是為整個數(shù)據(jù)集創(chuàng)建的。將搜索范圍限制在相關(guān)向量上對于提供更好的客戶體驗(yàn)來說，在許多用例中都是絕對必要的。

• Performing hybrid search: For best performance, you typically have to combine vector search with traditional lexical approaches 執(zhí)行混合搜索：為了獲得最佳性能，通常需要將向量搜索與傳統(tǒng)的詞匯方法相結(jié)合

Dense versus sparse vector retrieval 密集與稀疏向量檢索

There are two big families of retrieval approaches, often referred to as “dense” and “sparse.” Both use a vector representation of text, which encodes meaning and associations, and both perform a search for close matches as a second step, as indicated in Figure 5 below. All vector-based retrieval approaches have that in common. 檢索方法通常分為“密集”和“稀疏”兩大類。兩者都使用文本的向量表示，編碼了含義和關(guān)聯(lián)，并且都在第二步進(jìn)行接近匹配的搜索，如下圖5所示。所有基于向量的檢索方法都具有這一共同點(diǎn)。

Above we described what more specifically is known as “dense” vector search, where unstructured data is transformed into a numeric representation using an embedding model, and you find matches as a nearest neighbor to a query in embedding space. To deliver high relevance results, dense vector search generally requires in-domain retraining. Without in-domain retraining, they may underperform even traditional lexical scoring, such as Elastic’s BM25. The upside and reason why vector search has garnered so much attention is that it can outperform all other approaches when fine-tuned, and it allows you to search unstructured data other than text, like images or audio, which has become known as “multimodal search.” The vector is considered “dense” because most of its values are non-zero. 上面我們描述了更具體被稱為“密集”向量搜索的內(nèi)容，其中非結(jié)構(gòu)化數(shù)據(jù)通過嵌入模型轉(zhuǎn)換為數(shù)值表示，然后在嵌入空間中找到與查詢最近的鄰居匹配。為了提供高相關(guān)性的結(jié)果，密集向量搜索通常需要在領(lǐng)域內(nèi)進(jìn)行重新訓(xùn)練。如果沒有在領(lǐng)域內(nèi)進(jìn)行重新訓(xùn)練，它們甚至可能表現(xiàn)不如傳統(tǒng)的詞匯評分，比如Elastic的BM25。向量搜索受到如此多的關(guān)注的好處和原因是，當(dāng)進(jìn)行精細(xì)調(diào)整時，它可以勝過所有其他方法，并且它允許您搜索除文本之外的非結(jié)構(gòu)化數(shù)據(jù)，比如圖像或音頻，這被稱為“多模態(tài)搜索”。向量被認(rèn)為是“密集”的，因?yàn)樗拇蟛糠种刀际欠橇愕摹?/p>

By contrast to “dense” vectors described above, “sparse” representations contain very few non-zero values. For example, lexical search that made Elasticsearch popular (BM25) is an example of a sparse retrieval method. It uses a bag-of-words representation for text and achieves high relevance by modifying the basic relevance scoring method known as TF-IDF (term frequency, inverse document frequency) for factors like length of the document. 與上文描述的“密集”向量相比，“稀疏”表示法包含非常少的非零值。例如，使Elasticsearch流行的詞匯搜索（BM25）就是一種稀疏檢索方法的例子。它使用詞袋表示文本，并通過修改基本相關(guān)性評分方法（稱為TF-IDF，即詞項(xiàng)頻率-逆文檔頻率）來考慮文檔長度等因素，從而實(shí)現(xiàn)高相關(guān)性。

Figure 5: Understanding the commonalities and differences between dense and sparse “vector” search 圖5：理解密集和稀疏“向量”搜索之間的共同點(diǎn)和差異

Learned sparse retrievers: Highly performant semantic search out of the box 學(xué)習(xí)稀疏檢索器：開箱即用的高性能語義搜索

The latest sparse retrieval approaches use learned sparse representations that offer multiple advantages over other approaches: 最新的稀疏檢索方法使用了學(xué)習(xí)到的稀疏表示，相比其他方法具有多種優(yōu)勢：

• High relevance without any in-domain retraining: They can be used out of the box without adapting the model on the specific domain of the documents. 高相關(guān)性，無需進(jìn)行領(lǐng)域內(nèi)的再訓(xùn)練：它們可以直接使用，無需在文檔的特定領(lǐng)域上調(diào)整模型。

• Interpretability: You can follow along which terms are matched, and the score attached by sparse encoders indicates how relevant a term is to a query — very interpretable — whereas dense vector search relies on the numeric representations of meaning that were derived by applying an embedding model, which is “black box” like many machine learning approaches. 可解釋性：您可以跟蹤匹配的術(shù)語，稀疏編碼器附加的分?jǐn)?shù)表示一個術(shù)語對查詢的相關(guān)性有多高，非常易解釋；而密集向量搜索依賴于通過應(yīng)用嵌入模型得出的意義的數(shù)值表示，這類似于許多機(jī)器學(xué)習(xí)方法的“黑匣子”。

• Fast: Sparse vectors fit right into inverted indices that have made established sparse retrievers like Lucene and Elasticsearch so fast. But sparse retrievers only apply to text data — not to images or other types of unstructured data. 快速：稀疏向量完全適用于倒排索引，這使得像Lucene和Elasticsearch這樣的稀疏檢索器變得如此快速。但稀疏檢索器僅適用于文本數(shù)據(jù)，而不適用于圖像或其他類型的非結(jié)構(gòu)化數(shù)據(jù)。

Key tradeoffs between sparse and dense vector-based retrieval 稀疏和密集向量檢索之間的關(guān)鍵權(quán)衡

Elastic 8.8 introduced our own learned sparse retriever, included with the Elasticsearch Relevance EngineTM (ESRETM), which expands any text with related relevant words. Here’s how it works: a structure is created to represent terms found in the document, as well as their synonyms. In a process called term expansion, the model adds terms based on their relevance to the document, from a static vocabulary of 30K fixed tokens, words, and sub-word units. Elastic 8.8引入了我們自己學(xué)習(xí)的稀疏檢索器，包含在Elasticsearch相關(guān)性引擎（ESRE）中，它可以擴(kuò)展任何文本的相關(guān)相關(guān)詞。它的工作原理是：創(chuàng)建一個結(jié)構(gòu)來表示文檔中發(fā)現(xiàn)的術(shù)語及其同義詞。在一個稱為術(shù)語擴(kuò)展的過程中，模型根據(jù)它們與文檔的相關(guān)性，從30K個固定標(biāo)記、單詞和子詞單元的靜態(tài)詞匯中添加術(shù)語。

This is similar to vector embedding in that an auxiliary data structure is created and stored in each document, which then can be used for just-in-time semantic matching within a query. Each term also has an associated score, which captures its contextual importance within the document and therefore is interpretable — unlike embeddings. 這類似于向量嵌入，因?yàn)樵诿總€文檔中創(chuàng)建并存儲了一個輔助數(shù)據(jù)結(jié)構(gòu)，然后可以用于查詢中的即時語義匹配。每個術(shù)語還有一個相關(guān)的分?jǐn)?shù)，捕捉了它在文檔中的上下文重要性，因此是可解釋的 — 不像嵌入。

Our pre-trained sparse encoder lets you implement semantic search out of the box and also addresses the other challenges with vector-based retrieval described above: 我們的預(yù)訓(xùn)練稀疏編碼器可以讓您立即實(shí)現(xiàn)語義搜索，并解決上述基于向量檢索的其他挑戰(zhàn)：

• You don’t need to worry about picking an embedding model — Elastic’s Learned Sparse Encoder model comes pre-loaded into Elastic, and you can activate it with a single click. 您不需要擔(dān)心選擇嵌入模型——Elastic的Learned Sparse Encoder模型已預(yù)裝到Elastic中，您只需點(diǎn)擊一次即可激活它。

• There are multiple ways to implement hybrid search, including reciprocal rank fusion (RRF) and linear combination. 有多種實(shí)現(xiàn)混合搜索的方法，包括互惠排名融合（RRF）和線性組合。

• Keep memory and storage in check by using quantized (byte-size) vectors and leveraging all the recent innovations in Elasticsearch that reduce data storage requirements. 通過使用量化（字節(jié)大?。┫蛄坎⒗肊lasticsearch中所有最新的創(chuàng)新技術(shù)來減少數(shù)據(jù)存儲需求，來控制內(nèi)存和存儲。

• Get all that in a hardened platform that can handle petabyte scale. 在一個能處理百萬億字節(jié)規(guī)模的硬化平臺上獲得所有這些。

You can learn about the model’s architecture, how we trained it, and how it outperforms alternative approaches in this blog describing the Elastic Learned Sparse Encoder. 您可以在描述彈性學(xué)習(xí)稀疏編碼器的博客中了解模型的架構(gòu)、我們是如何訓(xùn)練它的，以及它如何勝過替代方法。

Why choose Elastic as your vector database? 為什么選擇 Elastic 作為您的向量數(shù)據(jù)庫？

Elastic’s vector database is a strong offering in the fast developing vector search market. It provides: Elastic的向量數(shù)據(jù)庫是快速發(fā)展的向量搜索市場中的一個強(qiáng)大選擇。它提供：

• Semantic search out of the box 開箱即用的語義搜索

• Best-in-class retrieval performance: Hybrid search with Elastic’s Learned Sparse Encoder combined with BM25 outperforms SPLADE, ColBERT, and high-end embedding models offered by OpenAI. 最佳檢索性能：彈性學(xué)習(xí)稀疏編碼器與BM25的混合搜索優(yōu)于SPLADE、ColBERT和OpenAI提供的高端嵌入模型。

Figure 6: Elastic’s sparse retrieval model outperforms other popular vector databases, evaluated using NDCG@10 over a subset of the public BEIR benchmark 圖6：Elastic的稀疏檢索模型在公共BEIR基準(zhǔn)測試的子集上，使用NDCG@10評估，表現(xiàn)優(yōu)于其他流行的向量數(shù)據(jù)庫

• The flexibility of using our market leading learned sparse encoder model, picking any off-the-shelf model, or bringing your own optimized model — so you can keep up with innovations in this rapidly evolving space 利用我們市場領(lǐng)先的學(xué)習(xí)稀疏編碼器模型的靈活性，選擇任何現(xiàn)成模型，或者使用自己優(yōu)化的模型 —— 這樣你就能跟上這個快速發(fā)展的領(lǐng)域的創(chuàng)新

• Efficient pre-filtering on HNSW using a practical algorithm that appropriately trades off speed against loss in relevancy 在HNSW上使用高效的預(yù)過濾，使用一個實(shí)用的算法，適當(dāng)?shù)卦谒俣群拖嚓P(guān)性損失之間進(jìn)行權(quán)衡

• Capabilities needed in most search applications that dedicated vector databases do not provide, like aggregation, filtering, faceted search, and auto-complete 大多數(shù)搜索應(yīng)用程序需要的功能，專用向量數(shù)據(jù)庫無法提供，如聚合、過濾、分面搜索和自動完成文章來源地址http://www.zghlxwxcb.cn/news/detail-773278.html

到了這里，關(guān)于使用ElasticSearch完成大模型+本地知識庫：BM25+Embedding模型+Learned Sparse Encoder 新特性的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！