Questions tagged [airflow]
Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.
airflow
10,724
questions
1
vote
1
answer
8
views
How to add trigger for run dag into another dag (with decorators) in Airflow?
I got 2 dags in same dags folder:
dag_update_database (dag_update_database.py)
dag_add_client_loyalty (dag_updade_clients_loyalty.py)
I need run second dag right after successfully run first dag.
I'...
- 327
1
vote
0
answers
15
views
GKEStarPodOperator/Kubectl API: Error 500 "No agent available"
try to launch operator GKEStartPodOperator, to launch a simple test again GKE cluster in GCP.
Using service account with these roles:
Kubernetes Engine Admin
Kubernetes Engine Developer
Kubernetes ...
- 127
0
votes
0
answers
24
views
How can I pause a GKE autopilot nigthly?
How can I pause my GKE Autopilot cluster nightly in GCP? This cluster supports the Cloud Composer service in our Data stage area.
We know that it's not possible to do this natively because the service ...
0
votes
0
answers
13
views
DAG: How can I Transform an unformatted file into a CSV in the same Bucket?
I'm trying to create a DAG (airflow) that processes an unformatted file uploaded to a GCP bucket and generates a new .csv file with specific transformations. The input file format is "CARTOLAS....
0
votes
0
answers
22
views
importing airflow automatically creates airflow directory
I've noticed that whenever I import airflow in Python, it automatically creates an airflow directory in my home directory. Literally just this
$ python
Python 3.11.9 | packaged by conda-forge | (main, ...
- 1,287
0
votes
1
answer
14
views
Passing Airflow Variables to a constructor
I have a PythonOperator task that requires arguments to be passed to the constructor before the python callable is invoked.
PythonOperator(task_id='mytask',
provide_context=True,
...
0
votes
0
answers
15
views
Airflow can't fetch logs from remote docker workers. Name or service not known
I have two nodes airflow setup, on one node there is webserver, several workers, scheduler. Node number two have only workers.For workers I use celery. But webserver can't fetch logs from node number ...
- 63
0
votes
0
answers
20
views
Downloading Files via FTP While Get Stuck
The main logic and code I use:
step 1:
check whether the existing files locally have the same file size as them on FTP:
ftp.nlist() # get file names on ftp
for file in ftp_files:
response = ftp....
0
votes
1
answer
10
views
How to pass airflow connection credentials to DatabricksSubmitNowOperator without exposing them?
I'm rather new to airflow and couldn't find information anywhere. Currently in my DAG i'm reading an existing Postgres connection configured in airflow thus taking the credentials for it and passing ...
- 1
0
votes
0
answers
10
views
How CeleryExecutor executes KubernetesPodOperator?
I have an Airlfow instance spinning in Azure Kubernetes Service. Deployed with official Helm chart. And I noticed CeleryExecutor is used there. I have KubernetesPodOperator tasks, and wondering how ...
0
votes
0
answers
15
views
Running dbt run command and print some logs message
I use airflow to run a dbt command, which is a long running query, sometime take more than 1 hour. We have airflow config scheduler_zombie_task_threshold set to 600 which is 10 mins. This is leading ...
- 191
2
votes
0
answers
15
views
Is there a way to control maximum dag concurrency from UI param?
In airflow is it possible to control the concurrency of the DAG with an UI param?
As far as I know DAG run parameters are only know as task runtime, so it does not look like a good approach.
...
0
votes
1
answer
14
views
How to pass in-operator parameters to built-in templating for Airflow?
Have recently started to use Airflow for various jobs, trying to get handle on templating.
I was very impressed by EmailOperator, in which you could specify a html_content to be a HTML file, which ...
- 4,287
0
votes
0
answers
15
views
Airflow 2.7.2 Trigger badld-written code block Triggerer's async thread
I am running Airflow version 2.7.2, and I constantly see the issue in my trigger log.
Triggerer's async thread was blocked for 0.50 seconds, likely by a badly-written trigger. Set PYTHONASYNCIODEBUG=1 ...
- 17
0
votes
1
answer
21
views
Is there a way to trigger DAG when the current run fails?
Can I trigger a DAG immediately when the current run fails (it doesn't matter which of the tasks failed), without scheduling it?
