Questions tagged [aws-msk]
For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
aws-msk
437
questions
0
votes
0
answers
21
views
Confluent-Kafka: no broker available for coordinator query: intervaled in state query-coord
I am using python-confluent-kafka to create a producer.
The Kafka cluster is on MSK 3.7.x KRaft with IAM enabled and TLS enabled, both within the cluster and between clients and brokers.
Any ideas on ...
- 305
1
vote
1
answer
23
views
Can debezium/Kafka connect ignore or resize too large messages?
We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL.
For 99% of our data this all works fine, but very occasionally debezium fails with:
[Worker-...
- 196
0
votes
0
answers
13
views
High producer latency in AWS MSK Kafka
I am trying to run performance test script against my test kafka cluster in AWS. It looks like something is restricting the performance to about 2 MB/sec.
I have tried following:
Increased the type ...
- 1,897
0
votes
1
answer
24
views
How can i connect my springboot micro service to an MSK with IAM authentiaction?
Im trying to connect my project with this configuration
@Bean
public KafkaAdmin kafkaAdmin() {
Map<String, Object> configs = new HashMap<>();
configs.put(...
0
votes
0
answers
21
views
I have created a custom S3-Sink-Connector which reads Avro messages, validates with the schema and dumps into S3 Bucket in the parquet format
This is my S3SinkTask code where I'm facing these errors, please help me resolve the errors
C:\\Users\\ADMIN\\Downloads\\Projects\\custom-s3-sink-connector\>curl -X GET http://localhost:8083/...
- 1
0
votes
0
answers
19
views
Decrease the number of brokers in Kafka AWS MSK
I am trying to decrease the number of brokers in AWS MSK
When i go to Edit number of brokers, and try to decrease it by one, i get the following error:
In the table below, AZs with an error status ...
- 67
0
votes
1
answer
61
views
MSK connector failed to find class in MSK cluster with KRaft mode
I have a Kafka 3.7.x cluster using ZooKeeper mode which works well. Now I am trying new Kafak in KRaft mode:
However, for this Kafka cluster in KRaft mode, the same connector failed to start.
Here ...
- 48.7k
0
votes
0
answers
26
views
Troubleshooting Kafka Integration with Spark Streaming on Amazon EMR Serverless
Objective:
To set up a streaming job on Amazon EMR Serverless to process weather data from Amazon MSK (Managed Streaming for Apache Kafka) and write the word count results to an S3 bucket.
Steps Taken:...
0
votes
0
answers
14
views
AWS MSK Private Broker with Public NLB
I am trying to provide accesss to MY MSK in internal Network private VPC AWS. I have created Public NLB and associate the target group. In target group the instance is in halthy state and i can ...
0
votes
0
answers
33
views
Connect kafdrop with secure AWS MSK broker
I can connect kafka cli producer to AWS MSK broker secured with IAM Authentication & Authorization using the following steps:
Setup AWS CLI locally
Place aws-msk-iam-auth-2.1.1-all.jar in the ...
- 1,897
0
votes
0
answers
16
views
aws kafka get-bootstrap-brokers cli command returns nothin
I am running following command.
It executes without any error but returns nothing.
What can be the reason?
aws kafka get-bootstrap-brokers --cluster-arn --region eu-central-1 --output json
I am ...
- 1
0
votes
0
answers
15
views
Does it required donwtime while decreasing insync replica value in AWS MSK Cluster?
I have a MSK Cluster with 2 RF and 2 minISR values and there are multiple topics on which many producers and consumers are consuming messages.
As we should always have minISR value as RF-1, so I need ...
- 1,205
0
votes
1
answer
92
views
Create Topic Authorization Problem on Amazon MSK (Public Kafka)
We have a Kafka server running on Amazon AWS. This is a public server with SASL/SCRAM enabled. We can successfully authenticate but when we try to create a new topic, we get an authorization error.
...
- 2,769
0
votes
0
answers
29
views
AWS MSK ServerlessCluster did not stabilize
I am using serverless framework to deploy my cloudformation resources. I am trying to create an MSK cluster, but I'm getting back
Resource handler returned message: "Resource of type 'AWS::MSK::...
- 623
0
votes
0
answers
34
views
Kafkaconnect not able to find S3Sinkconnector
I have downloaded and added S3connector libs from confluent into folder
/usr/share/java/kafka/s3-connect
I can see all necessary jars under above location
Also added
/config/connect-standalone....
- 29