项目场景：

项目场景：airflow scheduler在运行一段时间后突然不工作或者空转，后台进程存在，但是不工作

问题描述

查看日志发现scheduler无心跳

================================================================================
[2022-06-14 14:51:06,301] {manager.py:1065} INFO - Finding 'running' jobs without a recent heartbeat
[2022-06-14 14:51:06,304] {manager.py:1069} INFO - Failing jobs without heartbeat after 2022-06-14 06:46:06.304241+00:00
[2022-06-14 14:51:16,495] {manager.py:1065} INFO - Finding 'running' jobs without a recent heartbeat
[2022-06-14 14:51:16,497] {manager.py:1069} INFO - Failing jobs without heartbeat after 2022-06-14 06:46:16.497371+00:00

查看scheduler错误日志

Traceback (most recent call last):File "/home/airflow/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 697, in _finalize_fairyfairy._reset(pool)File "/home/airflow/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 893, in _resetpool._dialect.do_rollback(self)File "/home/airflow/env/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2513, in do_rollbackdbapi_connection.rollback()
MySQLdb._exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')
(env) *@:/home/>tail airflow/conf/airflow-scheduler.err
Traceback (most recent call last):File "/home/airflow/env/lib/python3.9/threading.py", line 950, in _bootstrap_innerself.run()File "/home/airflow/env/lib/python3.9/concurrent/futures/process.py", line 317, in runresult_item, is_broken, cause = self.wait_result_broken_or_wakeup()File "/home/airflow/env/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeupworker_sentinels = [p.sentinel for p in self.processes.values()]File "/home/airflow/env/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration

原因分析：

追踪栈堆可见

Process 13749: python3.9 /home/airflow/env/bin/airflow scheduler -D
Python v3.9.0 (/home/airflow/env/bin/python3.9)Thread 13749 (idle): "MainThread"wait (threading.py:312)result (concurrent/futures/_base.py:435)result_iterator (concurrent/futures/_base.py:600)_chain_from_iterable_of_lists (concurrent/futures/process.py:559)_send_tasks_to_celery (airflow/executors/celery_executor.py:325)_process_tasks (airflow/executors/celery_executor.py:277)trigger_tasks (airflow/executors/celery_executor.py:268)heartbeat (airflow/executors/base_executor.py:158)_run_scheduler_loop (airflow/jobs/scheduler_job.py:734)_execute (airflow/jobs/scheduler_job.py:651)run (airflow/jobs/base_job.py:246)_run_scheduler_job (airflow/cli/commands/scheduler_command.py:46)scheduler (airflow/cli/commands/scheduler_command.py:70)wrapper (airflow/utils/cli.py:92)command (airflow/cli/cli_parser.py:48)main (airflow/__main__.py:48)<module> (airflow:8)
Thread 2111 (idle): "QueueFeederThread"wait (threading.py:312)_feed (multiprocessing/queues.py:233)run (threading.py:888)_bootstrap_inner (threading.py:950)_bootstrap (threading.py:908)

发现是线程空转导致scheduler进程空等，一直饥饿，检查发现这其实是python3.9的bug
详见issue43498

解决方案：

一次性方案

通过web端或者命令行
airflow jobs check --job-type SchedulerJob --allow-multiple --limit 100
监控scheduler的状态
如果进程无心跳则重启

永久方案

升级python版本

python >=3.9.10或3.10.1

airflow scheduler stuck相关推荐

大数据调度平台Airflow（八）：Airflow分布式集群搭建及测试
目录 Airflow分布式集群搭建及测试一.节点规划二.airflow集群搭建步骤 1.在所有节点安装python3.7 2.在所有节点上安装airflow 三.初始化Airflow 1.每台节点 ...
大数据调度平台Airflow（三）：Airflow单机搭建
目录 Airflow单机搭建一.安装Anconda及python3.7 1.官网下载Anconda ,选择linux版本,并安装 2.将下载好的anconda安装包上传至mynode4节点,进行安装 ...
airflow sql_alchemy_conn mysql_airflow的安装和使用 - 完全版
之前试用了azkaban一小段时间,虽然上手快速方便,但是功能还是太简单,不够灵活. Airflow使用代码来管理任务,这样应该是最灵活的,决定试一下. 我是python零基础,在使用airflow的 ...
centos 7 ssh 安装mysql,Centos 7 安装Airflow
Airflow 安装环境 : centos7 python3.7.0 mysql-8.0.22 一.安装mysql-8.0.22 二.安装python3.7.0 三.安装配置 [root@had ...
Airflow安装部署
Airflow安装部署新闻信息是通过爬虫获取,使用scrapy框架进行爬虫任务:使用airflow工作流监控平台对爬虫任务进行管理.监控(可使用CeleryExecutor分布式,也可使用Local ...
python etl 大猩猩_Airflow教程-使用Airflow实现ETL调度
一.Airflow是什么 airflow 是一个编排.调度和监控workflow的平台,由Airbnb开源,现在在Apache Software Foundation 孵化.airflow 将work ...
airflow sql_alchemy_conn mysql_搭建AirFlow—— 一段波折后的总结
声明:本人借鉴了众多同行的博客,所以总结了这篇博客,用来致敬他们,非常感谢他们,转载请注明出处. 基础环境:[CentOS 6.9](cat /etc/redhat-release) python2. ...
bitnami如何使用_使用Bitnami获取完全配置的Apache Airflow Docker开发堆栈
bitnami如何使用 I've been using it for around 2 years now to build out custom workflow interfaces, like ...
airflow mysql_Airflow 使用及原理分析
Airflow 入门及使用什么是 Airflow?Airflow 是一个使用 Python 语言编写的 Data Pipeline 调度和监控工作流的平台. Airflow 是通过 DAG(Dire ...

airflow scheduler stuck