Warning error formatting crashes the scheduler

[AvisDX]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.1

What happened and how to reproduce it?

We have our JWT secret which 44 bytes long and it gives a warning on the startup of the process via:
warnings.warn(key_length_msg, InsecureKeyLengthWarning, stacklevel=2)
but some formatting error of the warning crashes the process altogether

2026-04-30T10:08:50.242838Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.schemas
2026-04-30T10:08:50.243212Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.tables
2026-04-30T10:08:50.243472Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.types
2026-04-30T10:08:50.243713Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.constraints
2026-04-30T10:08:50.243953Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.defaults
2026-04-30T10:08:50.244189Z | INFO | 7 | plugins.py:37 | setup plugin alembic.autogenerate.comments
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
2026-04-30T10:08:51.463132Z | WARNING | 7 | default_celery.py:60 | No visibility_timeout configured in [celery_broker_transport_options]. Using default of 86400 seconds (24 hours). Celery tasks running longer than this will be redelivered by the broker, which terminates the original task. If you have long-running tasks, increase this value in your Airflow configuration: [celery_broker_transport_options] visibility_timeout = <seconds>
2026-04-30T10:08:51.499480Z | WARNING | 7 | default_celery.py:60 | No visibility_timeout configured in [celery_broker_transport_options]. Using default of 86400 seconds (24 hours). Celery tasks running longer than this will be redelivered by the broker, which terminates the original task. If you have long-running tasks, increase this value in your Airflow configuration: [celery_broker_transport_options] visibility_timeout = <seconds>
2026-04-30T10:08:51.505461Z | INFO | 7 | executor_loader.py:376 | Loaded executor: :CeleryExecutor:
2026-04-30T10:08:51.953394Z | INFO | 7 | scheduler_job_runner.py:1442 | Starting the scheduler
2026-04-30T10:08:51.955599Z | INFO | 7 | scheduler_job_runner.py:2715 | Adopting or resetting orphaned tasks for active dag runs
2026-04-30T10:08:52.320332Z | INFO | 7 | scheduler_job_runner.py:665 | 4 tasks up for execution:
	<TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]>
	<TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]>
	<TaskInstance: <dag>.send_summary_emails scheduled__2026-04-30T07:00:00+00:00 [scheduled]>
	<TaskInstance: <dag>.reparse_dags manual__2026-04-30T08:03:41.487312+00:00 [scheduled]>
2026-04-30T10:08:52.320646Z | INFO | 7 | scheduler_job_runner.py:749 | DAG <dag> has 0/16 running and queued tasks
2026-04-30T10:08:52.320842Z | INFO | 7 | scheduler_job_runner.py:749 | DAG <dag> has 0/16 running and queued tasks
2026-04-30T10:08:52.321055Z | INFO | 7 | scheduler_job_runner.py:749 | DAG <dag> has 0/16 running and queued tasks
2026-04-30T10:08:52.321257Z | INFO | 7 | scheduler_job_runner.py:749 | DAG <dag> has 0/16 running and queued tasks
2026-04-30T10:08:52.321553Z | INFO | 7 | scheduler_job_runner.py:902 | Setting the following tasks to queued state (scheduler job_id=31):
	<TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]> (id=<uuid>, try_number=1)
	<TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]> (id=<uuid>, try_number=1)
	<TaskInstance: <dag>.send_summary_emails scheduled__2026-04-30T07:00:00+00:00 [scheduled]> (id=<uuid>, try_number=1)
	<TaskInstance: <dag>.reparse_dags manual__2026-04-30T08:03:41.487312+00:00 [scheduled]> (id=<uuid>, try_number=1)
2026-04-30T10:08:52.327927Z | INFO | 7 | scheduler_job_runner.py:1030 | Trying to enqueue tasks: [<TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]>, <TaskInstance: <dag>.run_<dag> scheduled__2026-04-30T02:00:00+00:00 [scheduled]>, <TaskInstance: <dag>.send_summary_emails scheduled__2026-04-30T07:00:00+00:00 [scheduled]>, <TaskInstance: <dag>.reparse_dags manual__2026-04-30T08:03:41.487312+00:00 [scheduled]>] for executor: CeleryExecutor(parallelism=32)
2026-04-30T10:08:52.337837Z | ERROR | 7 | scheduler_job_runner.py:1468 | Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1463, in _execute
    self._run_scheduler_loop()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1602, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1747, in _do_scheduling
    num_queued_tis = self._critical_section_enqueue_task_instances(session=session)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1036, in _critical_section_enqueue_task_instances
    self._enqueue_task_instances_with_queued_state(queued_tis_per_executor, executor, session=session)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 982, in _enqueue_task_instances_with_queued_state
    workload = workloads.ExecuteTask.make(
        ti,
        generator=executor.jwt_generator,
        sentry_integration=_get_sentry_integration(executor),
    )
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/executors/workloads/task.py", line 99, in make
    token=cls.generate_token(str(ti.id), generator),
          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/executors/workloads/base.py", line 77, in generate_token
    return generator.generate({"sub": sub_id}) if generator else ""
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/auth/tokens.py", line 474, in generate
    return jwt.encode(claims, self.signing_arg, algorithm=self.algorithm, headers=headers)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/jwt/api_jwt.py", line 147, in encode
    return self._jws.encode(
           ~~~~~~~~~~~~~~~~^
        json_payload,
        ^^^^^^^^^^^^^
    ...<4 lines>...
        sort_headers=sort_headers,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/airflow/.local/lib/python3.13/site-packages/jwt/api_jws.py", line 197, in encode
    warnings.warn(key_length_msg, InsecureKeyLengthWarning, stacklevel=2)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.13/warnings.py", line 110, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       msg.file, msg.line)
       ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/structlog.py", line 678, in _showwarning
    warning_log.warning(str(message), category=category.__name__, filename=filename, lineno=lineno)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/structlog.py", line 96, in meth
    return self._proxy_to_logger(name, event, **kw)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/structlog/_base.py", line 222, in _proxy_to_logger
    args, kw = self._process_event(method_name, event, event_kw)
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/structlog/_base.py", line 173, in _process_event
    event_dict = proc(self._logger, method_name, event_dict)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/percent_formatter.py", line 161, in __call__
    sio.write(self._fmt % params)
              ~~~~~~~~~~^~~~~~~~
TypeError: %d format: a real number is required, not str
2026-04-30T10:08:52.346415Z | INFO | 7 | scheduler_job_runner.py:1480 | Exited execute loop
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/__main__.py", line 55, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/utils/cli.py", line 113, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/cli/commands/scheduler_command.py", line 66, in scheduler
    run_command_with_daemon_option(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        args=args,
        ^^^^^^^^^^
    ...<2 lines>...
        should_setup_logging=True,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
    callback()
    ~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/cli/commands/scheduler_command.py", line 69, in <lambda>
    callback=lambda: _run_scheduler_job(args),
                     ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/utils/memray_utils.py", line 59, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/cli/commands/scheduler_command.py", line 48, in _run_scheduler_job
    run_job(job=job_runner.job, execute_callable=job_runner._execute)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)  # type: ignore[arg-type]
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/job.py", line 355, in run_job
    return execute_job(job, execute_callable=execute_callable)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/job.py", line 384, in execute_job
    ret = execute_callable()
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1463, in _execute
    self._run_scheduler_loop()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1602, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1747, in _do_scheduling
    num_queued_tis = self._critical_section_enqueue_task_instances(session=session)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 1036, in _critical_section_enqueue_task_instances
    self._enqueue_task_instances_with_queued_state(queued_tis_per_executor, executor, session=session)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/jobs/scheduler_job_runner.py", line 982, in _enqueue_task_instances_with_queued_state
    workload = workloads.ExecuteTask.make(
        ti,
        generator=executor.jwt_generator,
        sentry_integration=_get_sentry_integration(executor),
    )
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/executors/workloads/task.py", line 99, in make
    token=cls.generate_token(str(ti.id), generator),
          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/executors/workloads/base.py", line 77, in generate_token
    return generator.generate({"sub": sub_id}) if generator else ""
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/auth/tokens.py", line 474, in generate
    return jwt.encode(claims, self.signing_arg, algorithm=self.algorithm, headers=headers)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/jwt/api_jwt.py", line 147, in encode
    return self._jws.encode(
           ~~~~~~~~~~~~~~~~^
        json_payload,
        ^^^^^^^^^^^^^
    ...<4 lines>...
        sort_headers=sort_headers,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/airflow/.local/lib/python3.13/site-packages/jwt/api_jws.py", line 197, in encode
    warnings.warn(key_length_msg, InsecureKeyLengthWarning, stacklevel=2)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.13/warnings.py", line 110, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       msg.file, msg.line)
       ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/structlog.py", line 678, in _showwarning
    warning_log.warning(str(message), category=category.__name__, filename=filename, lineno=lineno)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/structlog.py", line 96, in meth
    return self._proxy_to_logger(name, event, **kw)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/structlog/_base.py", line 222, in _proxy_to_logger
    args, kw = self._process_event(method_name, event, event_kw)
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.13/site-packages/structlog/_base.py", line 173, in _process_event
    event_dict = proc(self._logger, method_name, event_dict)
  File "/home/airflow/.local/lib/python3.13/site-packages/airflow/_shared/logging/percent_formatter.py", line 161, in __call__
    sio.write(self._fmt % params)
              ~~~~~~~~~~^~~~~~~~
TypeError: %d format: a real number is required, not str

What you think should happen instead?

No response

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.18.0

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[FTS18]

I've opened a PR (#66230) to fix this.
The issue was indeed a formatting mismatch in PercentFormatRender when callsite info is missing during startup. My PR ensures these parameters default to 0 instead of a string, preventing the crash.

[AvisDX]

@FTS18 Thanks!

[FTS18]

Just an update: The PR now includes a fix for the root cause in the logging bridge, so you'll see the correct Process and Thread IDs in your logs instead of just zeros or 'unknown'.

[vatsrahul1001]

Fixing PR still in draft state moving to next milestone