Questions tagged [apache-kafka-connect]
Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.
apache-kafka-connect
3,860
questions
0
votes
0
answers
6
views
Writing debezium CDC events from Kafka Connect to Iceberg
We want to ingest Debezium CDC events into Iceberg Tables on Azure Blob. We aim to do so using Kafka Connect Iceberg Sink Connector, and a Snowflake Catalog for Iceberg.
Questions that have surfaced -
...
- 3
-1
votes
0
answers
35
views
Why does debezium add a null bytes (\u0000) at the beginning of the message?
As in the question why does debezium add a null bytes (\u0000) at the beginning of the output message.
How can I remove null bytes (\u0000) from the beginning of the message?
- 784
0
votes
0
answers
11
views
Kafka Connect connector won't link up to a topic
In my work project I have some ksqlDB queries that rely on data provided by a topic that's supposed to be populated by a connector. Said connector isn't created through the ksqlDB CLI, but through the ...
0
votes
0
answers
27
views
Cheaper Alternatives to Confluent PostgreSQL Sink Connector for Consuming Kafka Data [closed]
I am currently working on a hobby project where we plan to consume data from Confluent Kafka and insert it into a PostgreSQL database. We began using the Confluent PostgreSQL Sink Connector, but we ...
- 358
0
votes
0
answers
20
views
Using the Confluent BigQuery sink Kafka connector, why are my messages/records not being written to new BQ tables, but existing tables are?
I am running Kafka Connect locally using Docker, but also in AWS MSK. Both are producing the same results.
My connector configuration is as follows:
{
"name": "connector-name",
&...
- 440
0
votes
0
answers
20
views
S3 Sink Confluent Kafka Connector JsonParser feature
I have json data in kafka, getting below stack trace
at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:331)
... 18 more Caused by: com.fasterxml.jackson.core....
- 305
-1
votes
1
answer
36
views
Strimzi Kafka Connect Cluster - Resource planning [closed]
I'm planing resources which will be dedicated to run Kafka Connect Clusters in Strimzi Distribution.
My workloads are pretty small, there will be probably only 1 task per connector needed.
I would go ...
- 134
1
vote
1
answer
24
views
Can debezium/Kafka connect ignore or resize too large messages?
We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL.
For 99% of our data this all works fine, but very occasionally debezium fails with:
[Worker-...
- 196
-1
votes
0
answers
18
views
Custom vault kafka configProvider for Mongo sink Connector
I implemented a custom vault configProvider with dynamic credentials rotation using configData(Map<String, String> data, Long ttl) for a MongoSink Kafka connect.
The first version of plugin ...
- 1
0
votes
1
answer
31
views
Getting Timeout Kafka exception org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA)
I am facing kafka timeout exception. Error is:
partition: B3-0 error: org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA), deadlineMs=1721138713548, tries=721498,...
- 155
0
votes
1
answer
26
views
How many kafka connect clusters are optimal?
I use Kafka with Strimzi operator and I am wondering if there is a optimal number of kafka connect clusters I should make. I have a medium sized infrastructure with around 50 different business logic ...
- 91
0
votes
0
answers
21
views
Minimal MirrorSourceConnector configurations to replicate data to another Kafka instance
I am trying to replicate topics from one Kafka cluster to another using Mirror Maker.
I am using docker compose to create the following resources:
networks:
local_kafka:
name: local_kafka
...
- 307
0
votes
0
answers
16
views
Add headers from kafka records to mongo field using sink connector
Im using a mongo sink connector to sink data from kafka topic to mongo collection, all i need is to add headers from these Kafka record to some field in the mongo record.
Its ok if I can map each ...
- 141
0
votes
1
answer
21
views
SQL connector to KSQLDB , how to do it in docker.yml?
I found myself struggling to get documentation updated on how to put SQL server DB to KSQLDB ?
I would like to know what would be the easiest way to get the connection toward Azure SQL Server and ...
- 3
0
votes
0
answers
60
views
Kafka Connect to Snowflake connection via JDBC error
I've been trying to send data from Kafka to Snowflake using the JDBC driver with Kafka Connect.
Some details about the environment:
Kafka is running in a Cloudera private cluster (Base 7.1.9).
The ...
- 1