GitHub - ybyzek/kafka-github-actions: Example GitHub Actions for Apache Kafka client application development for local and Confluent Cloud (original) (raw)
ybyzek / kafka-github-actions Public
- Notifications You must be signed in to change notification settings
- Fork9
- Star 15
Example GitHub Actions for Apache Kafka client application development for local and Confluent Cloud
15 stars 9 forks Branches Tags Activity
Notifications You must be signed in to change notification settings
Additional navigation options
Folders and files
| Name | Name | Last commit message | Last commit date |
|---|---|---|---|
| Latest commitHistory9 Commits | |||
| .github/workflows | .github/workflows | ||
| checkstyle | checkstyle | ||
| configs | configs | ||
| images | images | ||
| schemas | schemas | ||
| src | src | ||
| tests | tests | ||
| .gitignore | .gitignore | ||
| Dockerfile | Dockerfile | ||
| README.md | README.md | ||
| pom.xml | pom.xml | ||
| requirements.txt | requirements.txt |
Repository files navigation
README
Overview
This demo uses GitHub Actions for developing and valiating Apache Kafka client application, focusing on a Python producer and consumer using Kafka Python Client, plus examples for testing schema compatibility using both Python and Java.
The code defines an GitHub Actions workflow kafka-app with jobs to demonstrate a few capabilities:
- build: Build a Docker image for the Kafka Python client application
- sanity: Sanity test the app with rdkafka mock
- local: Locally test the app using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
- local-avro: Locally test the app with Schema Registry and Avro using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
- ccloud: Test the app to Confluent Cloud
- ccloud-avro: Test the app to Confluent Cloud with Confluent Cloud Schema Registry and Avro
- ccloud-python-schema-compatibility: Use SchemaRegistryClient.test_compatibility() as shown in src/test_schema_compatibility.py to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Python)
- ccloud-java-schema-compatibility: Use kafka-schema-registry-maven-plugin as shown in the pom.xml to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Java)
Instructions
- Provision a Kafka cluster, enable Schema Registry and create an API key and secret on Confluent Cloud. Use the quickstart, Confluent Terraform Provider, or ccloud-stack.
- In your GitHub account, create Encrypted Secrets for your Confluent Cloud cluster and credentials:
Confluent Cloud Kafka cluster
CONFLUENT_BOOTSTRAP_SERVERS CONFLUENT_API_KEY CONFLUENT_API_SECRET
Confluent Cloud Schema Registry
CONFLUENT_SCHEMA_REGISTRY_URL CONFLUENT_BASIC_AUTH_USER_INFO
- Fork this repository and push a change to trigger the workflow.
Resources
- Blog: more details on Apache Kafka CI/CD with GitHub Actions
- Confluent Cloud: Cloud-native service for Apache Kafka
- GitHub Actions for ksqlDB: demonstrates schema migrations upon a ksqlDB cluster running in Confluent Cloud
- Kafka Python Client
- GitHub Actions
About
Example GitHub Actions for Apache Kafka client application development for local and Confluent Cloud
Topics
python java schema-registry apache-kafka librdkafka confluent-cloud github-actions schema-registry-maven-plugin avro-schema-registry
Resources
Stars
Watchers
Forks
Releases
No releases published