Questions tagged [aws-batch]
AWS Batch enables you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure.
aws-batch
452
questions
1
vote
1
answer
37
views
Snakemake running as an AWS Batch or an AWS Fargate task raises MissingInputException on the inputs stored on a S3 bucket
We have a Dockerized Snakemake pipeline with the input data stored on a S3 bucket snakemake-bucket:
Snakefile:
rule bwa_map:
input:
"data/genome.fa"
output:
"...
0
votes
0
answers
25
views
How to Run 40+ Jobs Concurrently in AWS Batch Without Exceeding Time Limits
I'm currently using AWS Batch to run multiple jobs concurrently, but I'm facing issues with job timeouts. My goal is to run more than 40 jobs at a time, but some jobs are failing due to exceeded time ...
1
vote
1
answer
43
views
AWS deployment Options: ECS, Batch, Step Functions, which to use?
I am currently having a relatively huge batch logic analyzing transactions and purchase data, and applying dynamic pricing accordingly. It runs twice daily and takes around 2 hours to run. I am ...
- 55
0
votes
0
answers
19
views
AWS Batch Healthcheck
Looks like AWS Batch Job Definition does not support health checks.
I am scheduling long running jobs (containers) using AWS Batch on EC2. Without health checks there is a possibility of jobs stuck in ...
- 835
0
votes
0
answers
35
views
422 Unprocessable Entity on SageMaker Batch Transform Job
I'm trying to do a Batch Transform Job using my own container. I don't know what could be the problem on the invocation of the job.
Here is my Dockerfile
FROM python:3.7
COPY requirements.txt /opt/...
0
votes
1
answer
34
views
Design Pattern AWS Batch calling a REST endpoint using cron
I am inexperienced with AWS, so if this is a basic question please excuse my lack of knowledge here.
This question is an architectural question.
I have set up the following on AWS so far which is ...
- 8,729
1
vote
0
answers
29
views
Unable to create AWS Batch Job-Definition of Fargate type
I am referring to aws_batch_job_definition – Manage AWS Batch Job Definitions — Ansible Documentation page in-order to create aws job-definition.
As mentioned, we can create a container type of job-...
0
votes
0
answers
24
views
AWS Batch Job Metric confusion
This is our current workflow:
File is dropped in S3 Bucket => a Lambda is triggered => Lambda then starts a Batch Job to process the file =>
Batch Job starts a task in ECS => an Instance ...
- 2,450
0
votes
0
answers
30
views
How do Nextflow and AWS Batch are working together on an architectural level?
I am new to both AWS Batch and nf and I have some doubts. I have a question about the interaction between nf and aws. Think that you want to run the nf horse process locally and you want to run the ...
- 1
0
votes
0
answers
25
views
Debugging AWS batch job
I have submitted a job to AWS Batch running on Fargate. The job description was added without problem, the job was submitted, but it never moves out of runnable status. The problem is there is no, or ...
- 9,333
0
votes
0
answers
79
views
How to take AWS Batch Job ID from Step Function?
I am setting up a pipeline where I take celery message from SQS, decode it and run AWS Step Function. Inside Step Function I am submitting AWS Batch Job that runs my python script.
The issue I am ...
- 1
0
votes
0
answers
30
views
AWS Batch get all failed job name on specific Job Queue?
Is there a way to use AWS SDK, AWS Lambda or some python script to get all the job names that failed on specific job queue on AWS Batch? After extracting them, I would like to upload this to AWS S3 or ...
0
votes
0
answers
27
views
Extract AWS Batch and CloudWatch data for Amazon QuickSight?
I am trying to make a dashboard using data from CloudWatch logs. On AWS Batch, I have a job queue (I'll call this A). On this queue A, I have some jobs that succeeded and some jobs failed. When I ...
0
votes
1
answer
54
views
ModuleNotFoundError: No module named 'torch' on AWS Batch GPU instance
I have a job that runs on AWS Batch on a GPU instance, my application uses torch, i.e.
import torch
The Compute Environment has only one GPU instance, I was able to confirm that torch is available ...
- 3,524
1
vote
1
answer
74
views
Handling Long-Running Tasks and Subtasks with RabbitMQ and AWS Batch
Question:
I'm working on a system where I have a producer that initiates multiple tasks, each of which can take a significant amount of time to complete. Here's an overview of the system architecture:
...
- 75