High MTTR with many tenants

[etiennedi]

How to reproduce this bug?

• Have many tenants, 100s of thousands
• Restart node

What is the expected behavior?

Startup time should be more or less instant (with lazy shard loading)

What is the actual behavior?

We see one write operation per tenant that slows down restarting. It took about 8 minutes on a clsuter with between 100k-200k tenants

Supporting information

No response

Server Version

1.24.x

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct

[etiennedi]

My guess is that the motivation behind the many writes is that we write back the schema after startup in case there was a migration. Could probably be optimized by checking if anything was actually changed. If not, don't store the schema back.

Possible Steps to reproduce:

• Import 200k tenants, they need to be cold otherwise you run out of file descriptors on a single node
• restart server