Details
-
Task
-
Resolution: Done
-
Major
-
7.6.2
-
Couchbase Server Enterprise Edition 7.6.2 build 3581
-
0
Description
Memory usage while indexing vector_data field present in xattrs (metadata) is significantly lower compared to indexing vector_data field present in document data.
Aim:
To get isolated logs, to compare the memory usage and other profilings when indexing vector field within a document and within xattrs field.
Overall Procedure :
- Create a five node cluster ( 3 nodes -> KV , 2 nodes -> Search )
- Push 1 M ( 1000000 ) documents into the cluster (type 1 : no xattrs )
- The document schema as follows : { sno : id, sname : <random_name>, id : documentID, vector_data : <vector_embedding_1536_dimensions> }
- Create a search index.
- Once all docs are processed, run a query to verify the index.
- Now, download the logs ( cbcollect_info ) to your local machine
- Use promtimer package to visualize metrics in grafana dashboard
- Now recreate the entire cluster ( flash all the nodes and reinstall Couchbase in all the nodes)
- Do the same steps as above, except this time we store the vector_data inside xattrs section in metadata instead of data field in the document.
- The document schema as follows :
data : { sno : id, sname : <random_name>, id : documentID }
metadata : { xattrs : { vector_data : <vector_embedding_1536_dimensions> }}
- Create a search index. Once all docs are processed, run a query to verify the index.
- Now, download the logs ( cbcollect_info ) to your local machine
- Use promtimer package to visualize metrics in grafana dashboard
Global details :
Cluster information
Build : Couchbase Server Enterprise Edition 7.6.2 build 3581
Bucket type : couchbase buckets
Storage backend : couchstore
Compression mode : passive
Ejection : value only
Ram used for data service (post indexing) : 8.57 GiB / 18.8 GiB
Resident : 100%
Disk used : 1.45 GiB
Documents count : 1M (1000000) docs
Global Search Index specifics :
Dimension : 1536
Similarity_metric : dot_product
Optimized for : recall
Phase 1 :
Pushing docs with vector data in the document only ( not in xattrs )
Indexing time elapsed : 04 mins, 23 secs ( +- 10 secs )
Index definition:
curl -XPUT -H "Content-Type: application/json" -u <username>:<password> \ |
http://172.23.106.201:8094/api/bucket/nishanthb1/scope/test/index/search_index -d \ |
'{
|
"type": "fulltext-index", |
"name": "nishanthb1.test.search_index", |
"uuid": "ea0f3fd32c0dad45", |
"sourceType": "gocbcore", |
"sourceName": "nishanthb1", |
"sourceUUID": "61845a4a2d0d3a16bd4c7caf514bf490", |
"planParams": { |
"maxPartitionsPerPIndex": 1024, |
"indexPartitions": 1 |
},
|
"params": { |
"doc_config": { |
"docid_prefix_delim": "", |
"docid_regexp": "", |
"mode": "scope.collection.type_field", |
"type_field": "type" |
},
|
"mapping": { |
"analysis": {}, |
"default_analyzer": "standard", |
"default_datetime_parser": "dateTimeOptional", |
"default_field": "_all", |
"default_mapping": { |
"dynamic": true, |
"enabled": false |
},
|
"default_type": "_default", |
"docvalues_dynamic": false, |
"index_dynamic": true, |
"store_dynamic": false, |
"type_field": "_type", |
"types": { |
"test.test": { |
"dynamic": false, |
"enabled": true, |
"properties": { |
"vector_data": { |
"dynamic": false, |
"enabled": true, |
"fields": [ |
{
|
"dims": 1536, |
"index": true, |
"name": "vector_data", |
"similarity": "dot_product", |
"type": "vector", |
"vector_index_optimized_for": "recall" |
}
|
]
|
}
|
}
|
}
|
}
|
},
|
"store": { |
"indexType": "scorch", |
"segmentVersion": 16 |
}
|
},
|
"sourceParams": {} |
}'
|
Sample query:
curl -XPOST -H "Content-Type: application/json" -u Administrator:password \ |
http://172.23.106.201:8094/api/bucket/nishanthb1/scope/test/index/search_index/query \ |
-d '{
|
"query": { |
"match_none": {} |
},
|
"explain": true, |
"knn": [ |
{
|
"field": "vector_data", |
"k": 6, |
"vector":[13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19] |
}
|
],
|
"size": 10, |
"from": 0 |
}'
|
Response:
{"status":{"total":1,"failed":0,"successful":1,"errors":{}},"request":{"query":{"boost":null,"match_none":{}},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":true,"sort":["-_score"],"includeLocations":false,"search_after":null,"search_before":null,"knn":[{"field":"vector_data","vector":[13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19],"vector_base64":"","k":6}],"knn_operator":""},"hits":[{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre10","score":3100054.25,"explanation":{"value":3100054.25,"message":"sum of:","children":[{"value":3100054.25,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0003J'), score of:","children":[{"value":3100054.25,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u0003J') with similarity_metric(dot_product)=3.100054e+06"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre18","score":3053276.75,"explanation":{"value":3053276.75,"message":"sum of:","children":[{"value":3053276.75,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u000cA�), score of:","children":[{"value":3053276.75,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u000cA�) with similarity_metric(dot_product)=3.053277e+06"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre8","score":2982702.25,"explanation":{"value":2982702.25,"message":"sum of:","children":[{"value":2982702.25,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0018�), score of:","children":[{"value":2982702.25,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0018�) with similarity_metric(dot_product)=2.982702e+06"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre29","score":2934896.75,"explanation":{"value":2934896.75,"message":"sum of:","children":[{"value":2934896.75,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000�Y), score of:","children":[{"value":2934896.75,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u0000�Y) with similarity_metric(dot_product)=2.934897e+06"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre20","score":2878318.5,"explanation":{"value":2878318.5,"message":"sum of:","children":[{"value":2878318.5,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0008�I), score of:","children":[{"value":2878318.5,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u0008�I) with similarity_metric(dot_product)=2.878318e+06"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_ea0f3fd32c0dad45_4c1c5584","id":"pre780451","score":2839517.5,"explanation":{"value":2839517.5,"message":"sum of:","children":[{"value":2839517.5,"message":"fieldWeight(vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0005|D), score of:","children":[{"value":2839517.5,"message":"vector(field(vector_data:\u0000\u0000\u0000\u0000\u0000\u0005|D) with similarity_metric(dot_product)=2.839518e+06"}]}]},"sort":["_score"]}],"total_hits":6,"cost":0,"max_score":3100054.25,"took":1630364544,"facets":null} |
Logs :
Since, the attachments sections are filled with log files, to prevent confusion, the logs of this phase is stored as a zip file -> logs_noxattrs.zip
Run promtimer on these logs to get complete visualization.
Phase 2:
Pushing docs with vector data in the xattrs only ( not in document )
Indexing time elapsed : 04 mins, 15 secs ( +- 10 secs )
Index definition:
**
|
|
curl -XPUT -H "Content-Type: application/json" -u <username>:<password> \ |
http://172.23.106.201:8094/api/bucket/nishanthb1/scope/test/index/search_index -d \ |
'{
|
"type": "fulltext-index", |
"name": "nishanthb1.test.search_index", |
"uuid": "314c017150e32347", |
"sourceType": "gocbcore", |
"sourceName": "nishanthb1", |
"sourceUUID": "5672ed22735e849e91f5687faee612f9", |
"planParams": { |
"maxPartitionsPerPIndex": 1024, |
"indexPartitions": 1 |
},
|
"params": { |
"doc_config": { |
"docid_prefix_delim": "", |
"docid_regexp": "", |
"mode": "scope.collection.type_field", |
"type_field": "type" |
},
|
"mapping": { |
"analysis": {}, |
"default_analyzer": "standard", |
"default_datetime_parser": "dateTimeOptional", |
"default_field": "_all", |
"default_mapping": { |
"dynamic": true, |
"enabled": false |
},
|
"default_type": "_default", |
"docvalues_dynamic": false, |
"index_dynamic": true, |
"store_dynamic": false, |
"type_field": "_type", |
"types": { |
"test.test": { |
"dynamic": false, |
"enabled": true, |
"properties": { |
"_$xattrs": { |
"dynamic": false, |
"enabled": true, |
"properties": { |
"vector_data": { |
"dynamic": false, |
"enabled": true, |
"fields": [ |
{
|
"dims": 1536, |
"index": true, |
"name": "vector_data", |
"similarity": "dot_product", |
"type": "vector", |
"vector_index_optimized_for": "recall" |
}
|
]
|
}
|
}
|
}
|
}
|
}
|
}
|
},
|
"store": { |
"indexType": "scorch", |
"segmentVersion": 16 |
}
|
},
|
"sourceParams": { |
"includeXAttrs": true |
}
|
}'
|
|
|
Sample query:
curl -XPOST -H "Content-Type: application/json" -u Administrator:password \ |
http://172.23.106.201:8094/api/bucket/nishanthb1/scope/test/index/search_index/query \ |
-d '{
|
"query": { |
"match_none": {} |
},
|
"explain": true, |
"knn": [ |
{
|
"field": "_$xattrs.vector_data", |
"k": 6, |
"vector":[13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19] |
}
|
],
|
"size": 10, |
"from": 0 |
}'
|
Response
{"status":{"total":1,"failed":0,"successful":1,"errors":{}},"request":{"query":{"boost":null,"match_none":{}},"size":10,"from":0,"highlight":null,"fields":null,"facets":null,"explain":true,"sort":["-_score"],"includeLocations":false,"search_after":null,"search_before":null,"knn":[{"field":"_$xattrs.vector_data","vector":[13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19,13,60,15,4,5,1,1,9,11,72,114,30,2,1,9,25,2,29,114,17,2,12,11,0,2,33,56,11,2,35,10,2,86,45,8,1,0,0,9,27,113,114,78,17,4,4,8,47,2,10,97,102,40,26,12,2,4,0,21,87,32,46,13,8,114,29,7,9,16,2,2,23,107,38,15,10,48,41,114,109,7,2,8,28,33,72,114,74,2,0,10,65,60,49,4,7,109,10,3,19,15,1,5,37,37,6,6,59,94,20,37,50,38,33,12,11,3,1,30,72,9,1,2,23,26,10,6,19],"vector_base64":"","k":6}],"knn_operator":""},"hits":[{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre346231","score":3.9186734193247027e-07,"explanation":{"value":3.9186734193247027e-07,"message":"sum of:","children":[{"value":3.9186734193247027e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012~), score of:","children":[{"value":3.9186734193247027e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012~) with similarity_metric(l2_norm)=3.918673e-07"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre403616","score":3.8553295304362845e-07,"explanation":{"value":3.8553295304362845e-07,"message":"sum of:","children":[{"value":3.8553295304362845e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012\u000f), score of:","children":[{"value":3.8553295304362845e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012\u000f) with similarity_metric(l2_norm)=3.855330e-07"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre291907","score":3.849096924879485e-07,"explanation":{"value":3.849096924879485e-07,"message":"sum of:","children":[{"value":3.849096924879485e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012o), score of:","children":[{"value":3.849096924879485e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012o) with similarity_metric(l2_norm)=3.849097e-07"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre392426","score":3.848990255896268e-07,"explanation":{"value":3.848990255896268e-07,"message":"sum of:","children":[{"value":3.848990255896268e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012f), score of:","children":[{"value":3.848990255896268e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012f) with similarity_metric(l2_norm)=3.848990e-07"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre212996","score":3.8230333169707507e-07,"explanation":{"value":3.8230333169707507e-07,"message":"sum of:","children":[{"value":3.8230333169707507e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012), score of:","children":[{"value":3.8230333169707507e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012) with similarity_metric(l2_norm)=3.823033e-07"}]}]},"sort":["_score"]},{"index":"nishanthb1.test.search_index_314c017150e32347_4c1c5584","id":"pre295660","score":3.796680182848118e-07,"explanation":{"value":3.796680182848118e-07,"message":"sum of:","children":[{"value":3.796680182848118e-07,"message":"fieldWeight(_$xattrs.vector_data in doc \u0000\u0000\u0000\u0000\u0000\u0000\u0012), score of:","children":[{"value":3.796680182848118e-07,"message":"vector(field(_$xattrs.vector_data:\u0000\u0000\u0000\u0000\u0000\u0000\u0012) with similarity_metric(l2_norm)=3.796680e-07"}]}]},"sort":["_score"]}],"total_hits":6,"cost":0,"max_score":3.9186734193247027e-07,"took":796093512,"facets":null} |
Logs :
Since, the attachments sections are filled with log files, to prevent confusion, the logs of this phase is stored as a zip file -> logs_xattrs.zip
Run promtimer on these logs to get complete visualization.
