usearch的簡單使用
- usearch是快速開源搜索和聚類引擎×，用于C++、C、Python、JavaScript、Rust、Java、Objective-C、Swift、C#、GoLang和Wolfram ??中的向量和??字符串×
// https://github.com/unum-cloud/usearch/blob/main/python/README.md
$ pip install usearch
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting usearch
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/f4/24124f65ea3e940e54af29d55204ddfbeafa86d6b94b63c2e99baff2f7d6/usearch-2.8.14-cp38-cp38-manylinux_2_28_x86_64.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 17.0 MB/s eta 0:00:00
Requirement already satisfied: numpy in /home/ubuntu/anaconda3/envs/vglm2/lib/python3.8/site-packages (from usearch) (1.23.1)
Requirement already satisfied: tqdm in /home/ubuntu/anaconda3/envs/vglm2/lib/python3.8/site-packages (from usearch) (4.66.1)
Installing collected packages: usearch
Successfully installed usearch-2.8.14
- 一個簡單的例子（注：本例子在運行時向index中不斷添加項目，并將最后的index持久化為一個文件，在運行時由于添加項目內(nèi)存占用會不斷增加）
import numpy as np
from usearch.index import Index, MetricKind, Matches
ndim = 131072
index_path = "test.usearch"
index = Index(
ndim=ndim, # Define the number of dimensions in input vectors
metric='cos', # Choose 'l2sq', 'haversine' or other metric, default = 'ip'
dtype='f32', # Quantize to 'f16' or 'i8' if needed, default = 'f32'
connectivity=16, # How frequent should the connections in the graph be, optional
expansion_add=128, # Control the recall of indexing, optional
expansion_search=64, # Control the quality of search, optional
)# index = Index(ndim=ndim, metric=MetricKind.Cos)
for i in range(1,10):
vector = np.random.random((1000, ndim)).astype('float32')
index.add(None, vector, log=True)
index.save(index_path)
vector = np.random.random((1, ndim)).astype('float32')
matches: Matches = index.search(vector, 10)
ids = matches.keys.flatten()
print(matches)
# test.usearch大小： 10*1000*131072 =>2.2G (如果dtype='f32'=>4G+)
usearch-images
- https://github.com/ashvardanian/usearch-images
運行效果
數(shù)據(jù)獲取
- https://huggingface.co/datasets/unum-cloud/ann-unsplash-25k/tree/main
依賴 ucall
- Requires: Python >=3.9
- https://pypi.org/project/ucall/#files
OSError: [Errno 28] inotify watch limit reached
File "/home/ubuntu/anaconda3/envs/usearch/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 428, in _raise_error
raise OSError(errno.ENOSPC, "inotify watch limit reached")
OSError: [Errno 28] inotify watch limit reached
這個錯誤表明在使用 watchdog 庫時超過了 Linux 系統(tǒng)對 inotify 監(jiān)視的文件數(shù)或目錄數(shù)的限制。Linux 對于每個進程的 inotify 能夠監(jiān)視的文件和目錄有一個限制，當達到這個限制時，會出現(xiàn)像上面的錯誤一樣的問題?？梢試L試增加系統(tǒng)對 inotify 的資源限制?？梢酝ㄟ^修改 /etc/sysctl.conf 文件來增加 fs.inotify.max_user_watches 參數(shù)的值。例如：
```bash
sudo sysctl -w fs.inotify.max_user_watches=65536
```
然后，運行下面的命令使更改生效：
```bash
sudo sysctl -p
```
注意：增加監(jiān)視數(shù)可能會對系統(tǒng)資源產(chǎn)生一些影響，因此請根據(jù)實際情況慎重調(diào)整。文章來源：http://www.zghlxwxcb.cn/news/detail-787699.html
模型加載
文章來源地址http://www.zghlxwxcb.cn/news/detail-787699.html
- https://huggingface.co/unum-cloud/uform-vl-multilingual-v2/tree/main
///home/ubuntu/anaconda3/envs/usearch/lib/python3.10/site-packages/uform/__init__.py
def get_checkpoint(model_name, token) -> Tuple[str, Mapping, str]:
model_path = snapshot_download(repo_id=model_name, token=token)
config_path = f"{model_path}/torch_config.json"
state = torch.load(f"{model_path}/torch_weight.pt")
return config_path, state, f"{model_path}/tokenizer.json"
def get_model(model_name: str, token: Optional[str] = None) -> VLM:
config_path, state, tokenizer_path = get_checkpoint(model_name, token)
with open(config_path, "r") as f:
model = VLM(load(f), tokenizer_path)
model.image_encoder.load_state_dict(state["image_encoder"])
model.text_encoder.load_state_dict(state["text_encoder"])
return model.eval()
- 修改成如下，調(diào)用時使用
_model = get_model("你的下載路徑")
def get_checkpoint(model_name, token) -> Tuple[str, Mapping, str]:
model_path = model_name#snapshot_download(repo_id=model_name, token=token)
config_path = f"{model_path}/torch_config.json"
state = torch.load(f"{model_path}/torch_weight.pt")
return config_path, state, f"{model_path}/tokenizer.json"
def get_model(model_name: str, token: Optional[str] = None) -> VLM:
config_path, state, tokenizer_path = get_checkpoint(model_name, token)
with open(config_path, "r") as f:
model = VLM(load(f), tokenizer_path)
model.image_encoder.load_state_dict(state["image_encoder"])
model.text_encoder.load_state_dict(state["text_encoder"])
return model.eval()
其他細微的修改
數(shù)據(jù)源的修改
_datasets = {
name: _open_dataset(os.path.join("/home/ubuntu/userfile/***/Usearch/usearch-images-main/data", name))
for name in (
"unsplash-25k",
# "cc-3m",
# "laion-4m",
)
}
dataset_names: str = st.multiselect(
"Datasets",
[
dataset_unsplash_name,
# dataset_cc_name,
# dataset_laion_name,
],
[dataset_unsplash_name],#, dataset_cc_name],
format_func=lambda x: x.split(":")[0],
)
- 也可下載cc-3m數(shù)據(jù)：
數(shù)據(jù)讀取的修改
# uris: Strs = File(os.path.join(dir, "images.txt")).splitlines()
file_path = os.path.join(dir, "images.txt")
with open(file_path, 'r') as file:
uris = file.read().splitlines()
CG
- “usearch” 通常指的是一個生物信息學工具，用于對DNA和蛋白質(zhì)序列進行搜索和比對。具體來說，它是由Qiime軟件包提供的一個用于序列分析的工具，主要用于對微生物群落的高通量測序數(shù)據(jù)進行處理和分析。Qiime（Quantitative Insights Into Microbial Ecology）是一個用于分析和解釋微生物群落結(jié)構(gòu)的開源軟件包。在Qiime中，usearch被用于處理和比對DNA序列，以便進行物種注釋、多樣性分析等。USEARCH —— 最簡單易學的擴增子分析流程
到了這里，關(guān)于向量數(shù)據(jù)庫:usearch的簡單使用+實現(xiàn)圖片檢索應用的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
