Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Insert float16 vector to bfloat16 field could be successful #2018

Open
1 task done
binbinlv opened this issue Apr 2, 2024 · 4 comments
Open
1 task done

[Bug]: Insert float16 vector to bfloat16 field could be successful #2018

binbinlv opened this issue Apr 2, 2024 · 4 comments
Assignees
Labels
kind/bug Something isn't working

Comments

@binbinlv
Copy link

binbinlv commented Apr 2, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

Insert float16 vector to bfloat16 field could be successful

Expected Behavior

Report error

Steps/Code To Reproduce behavior

from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
import time
import h5py

import json
from pymilvus import CollectionSchema, FieldSchema, DataType, utility, connections, Collection
collection_name = "test_f16"
connections.connect()

dim = 128
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
bfloat16_vector = FieldSchema(name="bfloat16_vector", dtype=DataType.BFLOAT16_VECTOR, dim=dim)


schema = CollectionSchema(fields=[int64_field, bfloat16_vector])
if utility.has_collection(collection_name):
    old_collection = Collection(collection_name)
    old_collection.drop()
    print("drop")

collection = Collection(collection_name, schema=schema)

def gen_bf16_vectors(data):
      bf16_vectors = []
      for i in range(data.shape[0]):
            raw_vector = data[i]
            bf16_vector = tf.cast(raw_vector, dtype=tf.bfloat16).numpy()
            bf16_vectors.append(bf16_vector)
      return bf16_vectors

def gen_fp16_vectors(data):
       fp16_vectors = []
       for i in range(data.shape[0]):
             raw_vector =  data[i]
             fp16_vector = np.array(raw_vector, dtype=np.float16)
             fp16_vectors.append(fp16_vector)
       return fp16_vectors

import numpy as np
import tensorflow as tf

fname = "/home/sift-128-euclidean.hdf5"
f = h5py.File(fname, 'r')
data = f['train']
test = f['test']
neighbors = f['neighbors']
print(data.shape)
print(neighbors.shape)
print(test.shape)

print("begin insert...")
counter = 0
block_num = 100
block_size = int(data.shape[0]/block_num)
start = time.time()

for t in range(block_num):
    print("inserting: ", counter, " to ", counter + block_size )
    #bytes_data = gen_bf16_vectors(data[counter: counter + block_size])
    bytes_data = gen_fp16_vectors(data[counter: counter + block_size])
    #bytes_data = data[counter: counter + block_size]
    entities = [
            [i for i in range(counter, counter + block_size)],
            bytes_data
            ]
    insert_result =  collection.insert(entities)
    print(insert_result)
    counter = counter + block_size
print ("end of insert, cost: ", time.time()-start)

Environment details

- Hardware/Softward conditions (OS, CPU, GPU, Memory):
- Method of installation (Docker, or from source):
- Milvus version (v0.3.1, or v0.4.0):
- Milvus configuration (Settings you made in `server_config.yaml`):

Anything else?

No response

@binbinlv binbinlv added the kind/bug Something isn't working label Apr 2, 2024
@binbinlv
Copy link
Author

binbinlv commented Apr 2, 2024

/assign @XuanYang-cn

XuanYang-cn added a commit to XuanYang-cn/pymilvus that referenced this issue Apr 9, 2024
See also: milvus-io#2018, milvus-io#2004, milvus-io#2016

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Apr 10, 2024
See also: #2018, #2004, #2016

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn
Copy link
Contributor

/assign @binbinlv
Please help verify 2.5.0rc8

@XuanYang-cn
Copy link
Contributor

/unassign

XuanYang-cn added a commit to XuanYang-cn/pymilvus that referenced this issue Apr 10, 2024
sre-ci-robot pushed a commit that referenced this issue Apr 10, 2024
#2029)

See also: #2018, #2004, #2016

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn
Copy link
Contributor

pymilvus-2.4.1rc7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants