Skip to content

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies. uForwarder aims to address several pain points while using Apache Kafka for pub-sub message queueing at scale, including partition scalability and head-of-line blocking.

Notifications You must be signed in to change notification settings

uber/uForwarder

Repository files navigation

UForwarder

Overview

Uforwarder is a proxy that transfers data from kafka to message consumer through RPC protocol. Uforwarder aims to achieve reliable message delivery in realtime at scale through simple RPC protocol between Kafka and consumers.

Properties and Traits:

  • Out of order Message delivery
  • At least once message delivery
  • Parallel message processing of each kafka topic partition
  • Independent scaling of kafka topic partition and consumer instances
  • Retry queue and dead letter queue for error handling
  • Flow control / throttling
  • Head of line blocking detection and auto recovery (TODO: add link to more details)

RPC protocol supported

  • GRPC

Overfiew

For an overview of the project, please read the Uber Engineering blog Enabling Seamless Kafka Async Queuing with Consumer Proxy

Table of Contents

  1. Build
  2. Contributing
  3. License
  4. Acknowledgments
  5. Contact

Build

Prerequisite

  • Java 11 and above
  • Docker 24.0 and above

Steps

  1. Build binary
./gradlew jar
  1. Run unit test
./gradlew test
  1. Run integration test
./gradlew integrationTest
  1. Run code coverage verification
./gradlew check

Usage

Prerequisite

  • Kafka 2.8 and above
  • Zookeeper 3.4 and above

Steps

  1. Create a new docker network
docker network create docker-network
  1. Start Zookeeper and Kafka broker
docker run --env ALLOW_ANONYMOUS_LOGIN=yes --network docker-network -p 2181:2181 --name zookeeper zookeeper:3.8.0
docker run --env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 --env KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 --env KAFKA_LISTENERS=DOCKER://0.0.0.0:9092,HOSTER://0.0.0.0:9093 --env KAFKA_ADVERTISED_LISTENERS=DOCKER://kafka:9092,HOSTER://localhost:9093 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=DOCKER:PLAINTEXT,HOSTER:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=DOCKER --network docker-network -p 9093:9093 --name kafka confluentinc/cp-kafka:5.2.1
  1. Run Uforwarder controller and worker
docker run --env UFORWARDER_PROFILE=uforwarder-controller --env UFORWARDER_KAFKA_CONNECT=kafka:9092 --env UFORWARDER_ZOOKEEPER_CONNECT=zookeeper:2181/uforwarder --network docker-network --name controller -p 8087:8087 uforwarder:latest
docker run --env UFORWARDER_PROFILE=uforwarder-worker --env UFORWARDER_KAFKA_CONNECT=kafka:9092 --env UFORWARDER_CONTROLLER_CONNECT=controller:8087 --network docker-network --name worker uforwarder:latest
  1. Run Sample Consumer
./gradlew uforwarder-sample-consumer:bootRun

Contributing

We accept contributions through Pull Requests (PR's). Please sign our Contributing License Agreement (CLA) after you submit the PR, so that it can be accepted.

License

Apache License Version 2.0

Contact

uforwarder-committers@googlegroups.com

About

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies. uForwarder aims to address several pain points while using Apache Kafka for pub-sub message queueing at scale, including partition scalability and head-of-line blocking.

Resources

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages