subdag tasks don't get picked up when dag concurrency set to 1

[dstandish]

They do get picked up like once or twice but pretty quickly somehow things get into a state where the scheduler thinks there is a task runninging in the dag (the subdag operator) so it won't schedule any more tasks (i.e. the real tasks, in the subdag)

i know subdags are on the way out and i am no fan of subdags (i took this opportunity to remove all of them for our team) but it looks like a bug so i'm reporting it.

[halilduygulu]

I think in airflow 2 subdags are considered as a task instance and a dagrun, so when i have 15 subdags in a dag and set scheduler level general concurrency to 15, I have no running task instance slot left for actual tasks since subdag operator is holding up scheduler to create task instances.

[dstandish]

the dags (where the subdags lived) had a dag concurrency of 1. it wasn't general scheduler concurrency (i.e. cluster parallelism or whatever it is)

but so it sounds like you are saying these simply needed to be updated to have a concurrency of 2.... for example. these seems a bit risky what if after finishing first task in a subdag the scheduler launches a new subdag instead of another task in the first subdag.... not that this happens just thinking it through.

anyway i don't see any change to concurrency handling in https://github.com/apache/airflow/blob/master/UPDATING.md .... and these subdag dags did run properly before upgrading, (though there is mention of scheduler vs backfill) so it seems there may be something undesirable going on.

[ernest-kr]

which version of the airflow are you using ?

[dstandish]

2.0.1

[jscheffl]

I believe this is rather a technical limitation where documentation might need to be improved or we need to have a warning raised in logs. Each task that is being executed requires a worker slot. A call to a subdag also allocates one worker slot until the execution of the subdag is completed.

I tried to re-produce the issue with the example_subdag_operator.py that is shipped with the example DAGs in Airflow (2.4.0) and was not able to re-produce when setting concurrency=1 in the DAG. Alternatively also with the non-deprecated option max_active_tasks=1.

My setup consisted out of LocalExecutor with a development setup of Airflow. What I was able to re-produce was forcing the Airflow scheduler to run SequentialExecutor which limits task processing to a single task. In this case (irrespective of setting of concurrency) only one task slot is provided as executor. In this case SubDagOperator can not be used because the run of the subdag allocates one task slot and there is no worker slot possible to process tasks from within the subdag.

To be able to work on this issue, can you please:

• Do a regression on a current Airflow release
• If you can re-produce also try it with the example_subdag_operator.py
• If you can re-produce in your environment using a current version, please describe:
  • Which Airflow-Version
  • Which Environment
  • Which Executor is used
  • A copy of the DAG (or a neutralized DAG) which can be used to re-produce in an external environment

[eladkal]

By now SubDags are not something we will look into. It's been deprecated for 2.5+ years and we are no accepting PRs for this area of the code.

See https://github.com/apache/airflow/pull/31545#issuecomment-1582468856

Users who still use SubDags are encouraged to migrate to TaskGroups.

Closing as out of scope / won't fix.