Questions tagged [aws-msk-connect]
The aws-msk-connect tag has no usage guidance.
aws-msk-connect
45
questions
0
votes
2
answers
245
views
Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler
We are trying to use AWS serverless MSK for Apache Druid ingestion with IAM authentication. I am getting following error on Druid console even though I added aws-msk-iam-auth jar in Druid/libs folder.
...
- 317
-1
votes
2
answers
653
views
MSK with SASL/SCRAM authentication. I can access the cluster but cant perform any action
We're currently building an MSK cluster. We use 2 types of authentication for 2 different clients.
The IAM authentication works fine.
But for the SASL/SCRAM authentication that it's not the case, we ...
- 1,774
0
votes
1
answer
80
views
MSK Connector log level update
I am using the Kafka Snowflake connector as the MSK Connector. Default setting of the log level for logs is INFO, I could not find a way to change that.
Does the AWS MSK Connect API / CLI allow this ...
- 115
0
votes
1
answer
76
views
AWS MSK Connect inserting data to s3 without additional folder of {topic_name}
I am consuming a kafka topic (say, topic1) by using AWS MSK connect and sending to AWS s3.
The following is a simplified AWS MSK Connector config..
topics.dir=folder1/folder2
topics=topic1
The topic ...
- 758
0
votes
1
answer
122
views
AWS MSK connector role configuration
I'm creating a Neo4jSource for kafka on a aws Msk to connect with a Neo4j Cluster on AWS.
I have the custom plugin created, but when I start to configure the connector on the tab "Access ...
0
votes
1
answer
89
views
Error when create "java-pubsub-group-kafka-connector" to send messages from AWS MSK to Pub/sub
I'm using java-pubsub-group-kafka-connector to send messages from AWS MSK to GCP pub/sub. I created AWs MSK connector and status is running. But when send message to AWS MSK, after some time see this ...
0
votes
1
answer
89
views
GCP credentials for Pub/Sub to Apache Kafka Connector
I want to transfer messages from AWS MSK to GCP Pub/Sub using this connector.
Uploaded connector(pubsub-group-kafka-connector-1.2.0.jar ) to the s3 bucket, created custum plugin, and want to create ...
0
votes
1
answer
91
views
How to setup and configure mongodb kafka soruce connector using aws msk connect?
I have setup a MSK cluster using public subnets. I also have turned on public access for this MSK cluster. The cluster is up and running. I am trying to setup MongoDB Kafka Source Connector using MSK ...
- 157
0
votes
0
answers
165
views
How to deploy pyspark sturcture streaming in AWS EMR with MSK Connect mongdb kafka source connector plugin using aws console?
I am working on apache kafka and pyspark to run some data analytics job. I am using Mongodb Kafka Source Connector to produce data in a pyspark structured streaming job. In pyspark script I am using ...
- 157
0
votes
1
answer
159
views
Kafka to BigQuery MSK Connector BigQuerySinkConnector Failing with Invalid Configuration Error
I am facing a challenge with the setup of a Kafka to BigQuery connector using MSK (Managed Streaming for Kafka). The connector is failing with an InvalidInput.InvalidConnectorConfiguration error. ...
- 417
0
votes
0
answers
99
views
How to send data to API gateway with kafka intergartion proxy
So i would like to send data to my kafka topics,
i've created a EC2 instance with Role based authentication
On my EC2 client, download the Confluent.io Amazon S3 Connector and copy it to the S3 ...
- 56
0
votes
1
answer
96
views
AWS MSK Connect MirrorMaker2 - Failed to Flush
Using MirrorMaker2 on MSK Connect (2.7.1) the source connector is throwing the following error:
[Worker-0d8c5a576b5ef6e99] [2023-12-22 16:01:00,771] ERROR [msk-dev-conv-mm2-sourceconnector|task-1|...
- 1,414
0
votes
0
answers
51
views
Kinesis Confluent Connector not reaching MSK Cluster
I am trying to add a Kinesis connector to connect my Kinesis Stream with my AWS MSK Cluster, so that I can send messages from the Kinesis to a certain topic in the Kafka cluster.
This is the connector ...
- 117
1
vote
1
answer
937
views
How to handle nested arrays of struct in Kafka JDBC Sink Connector
I'm trying to handle nested arrays of struct via my Kafka Connect JDBC Sink Connector while writing records to Postgres.
Im using Kafka Connect (MSK Connect) build in AWS MSK. (I cannot use KSQL or ...
- 134
1
vote
0
answers
854
views
Reading AVRO messages fails for AWS glue schema registry deserializer
I have built a kafka pipeline using MSK, MSK Connect, Debezium Postgres Source connector and AWS Glue schema registry.
On producer side, I am able to publish AVRO records with schemas in Glue schema ...
- 33