[Reliability] Add guardrails for memory mappings

[etiennedi]

If we run out of memory mappings, Go throws a fatal error which can't be caught like a regular panic. It crashes the whole node. We need to prevent this from happening and fail more gracefully.

fatal error: runtime: cannot allocate memory

runtime stack:
runtime.throw({0x1a0df42?, 0xc000056000?})
	/usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0x7f05bcb738a0 sp=0x7f05bcb73870 pc=0x43d81c
runtime.newArenaMayUnlock()
	/usr/local/go/src/runtime/mheap.go:2243 +0x98 fp=0x7f05bcb738d0 sp=0x7f05bcb738a0 pc=0x42fe38
runtime.newMarkBits(0xdb1025e?)
	/usr/local/go/src/runtime/mheap.go:2163 +0x125 fp=0x7f05bcb73918 sp=0x7f05bcb738d0 pc=0x42fb05
runtime.newAllocBits(...)
	/usr/local/go/src/runtime/mheap.go:2198
runtime.(*mheap).initSpan(0x2addcc0, 0x7efb4426dba8, 0x0, 0x8, 0x7f05bcb739b8?, 0x1)
	/usr/local/go/src/runtime/mheap.go:1413 +0x165 fp=0x7f05bcb73958 sp=0x7f05bcb73918 pc=0x42e5c5
runtime.(*mheap).allocSpan(0x2addcc0, 0x1, 0x0, 0x0?)
	/usr/local/go/src/runtime/mheap.go:1344 +0x43a fp=0x7f05bcb739f8 sp=0x7f05bcb73958 pc=0x42e2da
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:968 +0x5c fp=0x7f05bcb73a40 sp=0x7f05bcb739f8 pc=0x42dafc
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f05bcb73a50 sp=0x7f05bcb73a40 pc=0x471baa

Note that the error message is a bit misleading, it sounds like OOM, but it's not. It's actually the vm.max_map_count setting/limit.

Relates to #4501

Acceptance Criteria

1. We have some observability on open file descriptors and memory mappings. Note: This doesn't have to come from Weaviate itself if there are other ways to extract this (e.g. from Kuberenets metrics). The important thing is that we see both the limit and current usage on our dashboards for both settings
2. Any operation that leads to more file descriptors and memory mappings (mainly creating or activating tenants) first checks if we still have enough room for both open file descriptors and memory mappings. If not, the request is denied.
   • Note: This is non-trivial on a replicated setup because the coordinator node would be the one doing the validation, but other nodes could be out of file descriptors/mappings as well. Maybe a good compromise here would be to just use the coordinator for now to decide for all replicas and in the future we could make sure that every node knows about the capacity of every other node (similar to how we currently do it with free disk space)
3. If the cluster is already in a state where more tenants are marked active per node than the node can handle, it will eventually crash when lazy shard loading loads all shards in the background. Rather than keep going, lazy shard loading should also abort before it comes to that.

[dirkkul]

First PR adds a basis but doesn't completely resolve this issue: #4667

With the PR, you can:

• set a memory mapping limit for weaviate. You need to figure of yourself what a good number is. The guardrails work with shard lazy loading => point 3 should be covered
• it does NOT take any multi-node into account
• it does NOT add observability - I think this should be done outside of weaviate