Batched inference API and support for float16 inference

[salvaba94]

This branch adds support for half precision inference and a batched inference API (BatchedModel). Additionally, it includes a short demo showing how to use this API.

[ZachOBrien]

@salvaba94 I ran this batch inference demo but did not see any performance benefit to batching. Batch size 1 took 223ms on average, batch size 2 took 447ms on average, etc. It scaled linearly in the batch size.

Did you observe the same behavior? Or were you able to get better throughput via batching?

[salvaba94]

Hi @ZachOBrien, I've just checked it and yes, I see marginal improvements by using batching.

Here are the results:

• Batch 1 and float32: 363 ms
• Batch 2 and float32: 668 ms
• Batch 1 and float16: 217 ms
• Batch 2 and float16: 351 ms

I guess the improvement depends on the GPU (this was tested with RTX 2060).