假设已基于docker搭建elasticsearch,详细过程参考如下链接。

https://blog.csdn.net/liliang199/article/details/151581138

这里先验证ES运行是否正常,然后导入向量数据,并示例查询过程。

1 安装验证

设置ES密码环境变量

export ELASTIC_PASSWORD=xxxxxx

如果忘记密码,可以重新生成一份

docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

把证书http_ca.crt从容器复制一份到宿主机,保存在当前目录,路径为"./http_ca.crt"

docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .

运行命令查看restful api是否运行正常

curl --cacert http_ca.crt -u elastic:$ELASTIC_PASSWORD "https://localhost:9200"

如果收到如下输出,说明ES运行正常

{
  "name" : "xxxf4ac4xxx",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "xxxx1x9j3qQ_xxx",
  "version" : {
    "number" : "9.1.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "dasdfa1a2f57de895a73a1391ff8426c0153c8d",
    "build_date" : "2025-05-24T22:05:04.526302670Z",
    "build_snapshot" : false,
    "lucene_version" : "10.2.2",
    "minimum_wire_compatibility_version" : "8.19.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}

2 python连接

pip安装elasticsearch包

pip install elasticsearch

python访问ES代码如下所示,证书文件“./http_ca.crt'”在当前目录。

from elasticsearch import Elasticsearch
import ssl
import asyncio

passwd = "espassword" # es环境获取

ssl_context = ssl.create_default_context(cafile='./http_ca.crt') # ./http_ca.crt获取见上文
es = Elasticsearch(
    ['https://localhost:9200'],
    http_auth=('elastic', passwd),
    # scheme="https",
    ssl_context=ssl_context
)

def main():
    info = es.info()
    print(info)

# 运行主函数
main()

输出如下,说明ES运行正常。

{'name': 'c2542f4ac460', xxxx 'tagline': 'You Know, for Search'}

3 建立索引

3.1 新建索引

建立search-ahz2索引,包含两个字段: vector,text,代码如下。

from elasticsearch import Elasticsearch

client = Elasticsearch(
    ['https://localhost:9200'],
    http_auth=('elastic', passwd),
    # scheme="https",
    ssl_context=ssl_context
)

index_name = "search-ahz2"

mappings = {
    "mappings": {
        "properties": {
            "source": {
                "type": "text"
            },
            "type": {
                "type": "text"
            },
            "vector": {
                "type": "dense_vector",
                "dims": 3
            },
            "text": {
                "type": "text"
            },
        }
    }   
}

client.indices.create(index=index_name, body=mappings)

res2 = client.indices.get(index=index_name)
print(res2)

输出如下,说明索引正确建立

{'acknowledged': True}

3.2 更新索引

如果想更新索引,运行如下代码。

mappings = {
    "properties": {
        "source": {
            "type": "text"
        },
        "type": {
            "type": "text"
        },
        "vector": {
            "type": "dense_vector",
            "dims": 3
        },
        "text": {
            "type": "text"
        },
    }
}

mapping_response = client.indices.put_mapping(index=index_name, body=mappings)
print(mapping_response)

输出如下,说明索引更新成功。

{'acknowledged': True}

需要注意的是:

1)因为是索引更新,所以更新的索引index_name必须存在,否则会报错。

2)保证es对象client的唯一性,比如程序空间不能同时存在两个es对象client,否则会报错。

如果不确定,可以先关闭之前创建的es对象client,然后重新创建新的client,代码示例如下

client.close()

client = Elasticsearch(
    [es_url],
    basic_auth=('elastic', passwd),
    ssl_context=ssl_context
)

4 数据导入

数据导入代码如下所示

from elasticsearch import Elasticsearch, helpers

docs = [
    {
        "source": "news",
        "type": "vec",
        "text": "Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acress across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site.",
        "vector": [
            4.667,
            0.145,
            3.07
        ]
    },
    {
        "source": "news",
        "type": "vec",
        "text": "Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face.",
        "vector": [
            4.222,
            3.251,
            6.634
        ]
    },
    {
        "source": "news",
        "type": "vec",
        "text": "Rocky Mountain National Park  is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site.",
        "vector": [
            6.504,
            9.081,
            0.003
        ]
    }
]

bulk_response = helpers.bulk(client, docs, index=index_name)
print(bulk_response)
print("done")

5 knn查询

5.1 直接查询

基于向量"query_vector": [-5, 9, -12],采用最近邻knn方式查询,并采用query进行过滤。

from elasticsearch import Elasticsearch
  
knn = {
    "field": "vector",
    "query_vector": [-5, 9, -12],
    "k": 30,
    "num_candidates": 100,
}

query = {
    "bool": {
        "must":  [
            {"term": {"source": "news"}},
            {"terms": {"type": ["vec"]}}
        ]
    }
}

search_response = client.search(
    index=index_name,
    knn = knn,
    query=query,
)
print(search_response)

输出示例

{'took': 3, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 3, 'relation': 'eq'}, 'max_score': 1.7727437, 'hits': [{'_index': 'search_multi_tags', '_id': 'RrSvbJoBzVHvZ0eui0CW', '_score': 1.7727437, '_source': {'source': 'news', 'type': 'vec', 'text': 'Rocky Mountain National Park  is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site.', 'vector': [6.504, 9.081, 0.003]}}, {'_index': 'search_multi_tags', '_id': 'RbSvbJoBzVHvZ0eui0CW', '_score': 1.3679638, '_source': {'source': 'news', 'type': 'vec', 'text': 'Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face.', 'vector': [4.222, 3.251, 6.634]}}, {'_index': 'search_multi_tags', '_id': 'RLSvbJoBzVHvZ0eui0CW', '_score': 1.3003887, '_source': {'source': 'news', 'type': 'vec', 'text': 'Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acress across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site.', 'vector': [4.667, 0.145, 3.07]}}]}}

knn参数说明如下

k,返回的最近邻对象的数量。k的值需要小于num_candidates。

num_candidates,每个分片上需查找的最近邻候选对象的个数,不能超过10000。增加num_candidates的值可提高最终K值的准确性,但相应搜索速度会变慢。

query_vector,查询向量,必须与field指定的向量数据具有相同的维度。

feild,要检索的向量字段名称。

reference

---

使用Elasticsearch的向量近邻检索(kNN)功能

https://help.aliyun.com/zh/es/use-cases/use-the-knn-search-feature-of-elasticsearch

docker搭建elasticsearch并使用python连接

https://zhuanlan.zhihu.com/p/660380507

kNN 搜索

https://elastic.ac.cn/docs/solutions/search/vector/knn

Logo

码道开发者社区,聚焦华为云码道 CodeArts 代码智能体,沉淀 Agent、Skill、鸿蒙开发实战内容,供开发者查阅资料、交流技术、分享工程实践

更多推荐