[FEATURE] distilabel server

[gabrielmbmb]

Description

Currently distilabel can be only executed locally executing either a python script with the pipeline or using the CLI.

For future integrations with Argilla UI or deploying distilabel in services like HF Inference Endpoints, it would be nice to have a distilabel server that it's waiting for requests to execute certain pipeline.

The server will have a queue of pipeline executions. The queue will be filled from a POST requests to an HTTP server, that will contain the pipeline and the runtime parameters needed to execute the pipeline.

Adding a layer in distilabel allowing to generate statistics about the pipeline execution could allow to send back the pipeline execution information to a UI using SSE.