wait-for-airflow-migrations fails after updating to v1.13

[chris922]

Checks

• I have checked for existing issues.
• This report is about the Openmetadata Helm Chart.
• This report is about the Openmetadata Dependencies Helm Chart.

Description

I updated from open-metadata v1.12.x to v1.13.1 and I am deploying airflow using the Dependencies Helm Chart.

The airflow db migration job completes successfully and the alembic_version is set to 1d6611b6ab7c. According to this list this revision id belongs to airflow 3.2.0: https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html
Unfortunately, all my airflow pods will not start afterwards and fail in wait-for-airflow-migrations container with this message: There are still unapplied migrations after 60 seconds. MigrationHead(s) in DB: {'1d6611b6ab7c'} | Migration Head(s) in Source Code: {'509b94a1042d'}

I debugged the issue and the issue seems to arise out of this extra PIP installation: apache-airflow-providers-fab==2.4.4
Without downgrading this package to 2.4.4 the airflow db check-migrations script works.
Here is the source of this:
https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml#L103-L104

Can you please double check if it's still required to downgrade to 2.4.4? How can we solve the issue so that airflow pods can start again?

Steps to reproduce

Step 1: Install openmetadata deps with airflow (v1.13.0 or v1.13.1).
Step 2: Verify if pods are able to start (should fail)

---

Fix issue:

Deploy again overriding the envs to remove _PIP_ADDITIONAL_REQUIREMENTS:

  # Environment variables for Airflow configuration
  env:
    # This is required for OpenMetadata UI to fetch status of DAGs
    - name: AIRFLOW__API__AUTH_BACKENDS
      value: "airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth"
    # OpenMetadata Airflow Apis Plugin DAGs Configuration
    - name: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS
      value: "/opt/airflow/dags"
    # OpenMetadata Airflow Secrets Manager Configuration
    - name: AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION
      value: ""
    - name: AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID
      value: ""
    - name: AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY
      value: ""
    # Workaround for Airflow 3 + MySQL compatibility issue
    # Downgrade FAB provider to avoid CREATE INDEX IF NOT EXISTS issue
    #- name: _PIP_ADDITIONAL_REQUIREMENTS
    #  value: "apache-airflow-providers-fab==2.4.4"

Chart Version

1.13.1

App Version

1.13.1

Kubernetes Version

1.31.14+rke2r1

Helm Version

doesn't matter

Relevant Logs

!!!!!  Installing additional requirements: 'apache-airflow-providers-fab==2.4.4' !!!!!!!!!!!!

WARNING: This is a development/test feature only. NEVER use it in production!
         Instead, build a custom image as described in

         https://airflow.apache.org/docs/docker-stack/build.html

         Adding requirements at container startup is fragile and is done every time
         the container starts, so it is only useful for testing and trying out
         of adding dependencies.

WARNING: connexion 2.15.1 does not provide the extra 'flask'
WARNING: connexion 2.15.0 does not provide the extra 'flask'
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
clickhouse-sqlalchemy 0.3.2 requires sqlalchemy<2.1.0,>=2.0.0, but you have sqlalchemy 1.4.54 which is incompatible.
databricks-sqlalchemy 2.0.9 requires sqlalchemy>=2.0.21, but you have sqlalchemy 1.4.54 which is incompatible.
iomete-sqlalchemy 1.0.22 requires sqlalchemy>=2.0.30, but you have sqlalchemy 1.4.54 which is incompatible.
openmetadata-ingestion 1.13.1.1 requires sqlalchemy<3,>=2.0.0, but you have sqlalchemy 1.4.54 which is incompatible.
openmetadata-managed-apis 1.13.1.1 requires Flask==3.1.3, but you have flask 2.2.5 which is incompatible.
sqlalchemy-cockroachdb 2.0.4 requires SQLAlchemy<2.1,>=2.0.47, but you have sqlalchemy 1.4.54 which is incompatible.
sqlalchemy-exasol 6.0.0 requires sqlalchemy<3.0.0,>=2.0.0, but you have sqlalchemy 1.4.54 which is incompatible.
sqlalchemy-hana 4.6.0 requires sqlalchemy<3,>=2, but you have sqlalchemy 1.4.54 which is incompatible.
sqlalchemy-redshift 1.0.0 requires SQLAlchemy<3,>=2.0.0, but you have sqlalchemy 1.4.54 which is incompatible.
2026-07-01T09:14:59.268556Z [info     ] setup plugin alembic.autogenerate.schemas [alembic.runtime.plugins] loc=plugins.py:37
2026-07-01T09:14:59.268802Z [info     ] setup plugin alembic.autogenerate.tables [alembic.runtime.plugins] loc=plugins.py:37
2026-07-01T09:14:59.268960Z [info     ] setup plugin alembic.autogenerate.types [alembic.runtime.plugins] loc=plugins.py:37
2026-07-01T09:14:59.269093Z [info     ] setup plugin alembic.autogenerate.constraints [alembic.runtime.plugins] loc=plugins.py:37
2026-07-01T09:14:59.269207Z [info     ] setup plugin alembic.autogenerate.defaults [alembic.runtime.plugins] loc=plugins.py:37
2026-07-01T09:14:59.269304Z [info     ] setup plugin alembic.autogenerate.comments [alembic.runtime.plugins] loc=plugins.py:37
/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:108 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
2026-07-01T09:15:05.086695Z [info     ] Waiting for migrations... 0 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:06.089592Z [info     ] Waiting for migrations... 1 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:07.092576Z [info     ] Waiting for migrations... 2 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:08.095323Z [info     ] Waiting for migrations... 3 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:09.098658Z [info     ] Waiting for migrations... 4 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:10.101987Z [info     ] Waiting for migrations... 5 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:11.104970Z [info     ] Waiting for migrations... 6 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:12.115622Z [info     ] Waiting for migrations... 7 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:13.120486Z [info     ] Waiting for migrations... 8 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:14.123445Z [info     ] Waiting for migrations... 9 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:15.126431Z [info     ] Waiting for migrations... 10 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:16.129305Z [info     ] Waiting for migrations... 11 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:17.132289Z [info     ] Waiting for migrations... 12 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:18.135246Z [info     ] Waiting for migrations... 13 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:19.138750Z [info     ] Waiting for migrations... 14 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:20.141829Z [info     ] Waiting for migrations... 15 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:21.144858Z [info     ] Waiting for migrations... 16 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:22.147826Z [info     ] Waiting for migrations... 17 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:23.151017Z [info     ] Waiting for migrations... 18 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:24.154265Z [info     ] Waiting for migrations... 19 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:25.157339Z [info     ] Waiting for migrations... 20 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:26.160162Z [info     ] Waiting for migrations... 21 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:27.163959Z [info     ] Waiting for migrations... 22 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:28.166830Z [info     ] Waiting for migrations... 23 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:29.169984Z [info     ] Waiting for migrations... 24 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:30.172968Z [info     ] Waiting for migrations... 25 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:31.176037Z [info     ] Waiting for migrations... 26 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:32.178915Z [info     ] Waiting for migrations... 27 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:33.181969Z [info     ] Waiting for migrations... 28 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:34.185161Z [info     ] Waiting for migrations... 29 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:35.188029Z [info     ] Waiting for migrations... 30 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:36.191621Z [info     ] Waiting for migrations... 31 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:37.195114Z [info     ] Waiting for migrations... 32 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:38.198168Z [info     ] Waiting for migrations... 33 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:39.200997Z [info     ] Waiting for migrations... 34 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:40.203956Z [info     ] Waiting for migrations... 35 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:41.206775Z [info     ] Waiting for migrations... 36 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:42.209763Z [info     ] Waiting for migrations... 37 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:43.213013Z [info     ] Waiting for migrations... 38 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:44.216117Z [info     ] Waiting for migrations... 39 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:45.219242Z [info     ] Waiting for migrations... 40 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:46.222591Z [info     ] Waiting for migrations... 41 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:47.225973Z [info     ] Waiting for migrations... 42 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:48.228823Z [info     ] Waiting for migrations... 43 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:49.232358Z [info     ] Waiting for migrations... 44 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:50.235308Z [info     ] Waiting for migrations... 45 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:51.238363Z [info     ] Waiting for migrations... 46 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:52.241464Z [info     ] Waiting for migrations... 47 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:53.244772Z [info     ] Waiting for migrations... 48 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:54.247888Z [info     ] Waiting for migrations... 49 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:55.251128Z [info     ] Waiting for migrations... 50 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:56.254282Z [info     ] Waiting for migrations... 51 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:57.257148Z [info     ] Waiting for migrations... 52 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:58.258909Z [info     ] Waiting for migrations... 53 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:15:59.261969Z [info     ] Waiting for migrations... 54 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:16:00.264055Z [info     ] Waiting for migrations... 55 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:16:01.266966Z [info     ] Waiting for migrations... 56 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:16:02.269983Z [info     ] Waiting for migrations... 57 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:16:03.272922Z [info     ] Waiting for migrations... 58 second(s) [airflow.utils.db] loc=db.py:811
2026-07-01T09:16:04.276124Z [info     ] Waiting for migrations... 59 second(s) [airflow.utils.db] loc=db.py:811
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 6, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 219, in check_migrations
    db.check_migrations(timeout=args.migration_wait_timeout)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 812, in check_migrations
    raise TimeoutError(
TimeoutError: There are still unapplied migrations after 60 seconds. MigrationHead(s) in DB: {'1d6611b6ab7c'} | Migration Head(s) in Source Code: {'509b94a1042d'}

Custom Helm Values

mysql:
  enabled: false
opensearch:
  enabled: false
airflow:
  enabled: true

  createUserJob:
    useHelmHooks: false
    applyCustomEnv: false
    
  migrateDatabaseJob:
    useHelmHooks: false
    applyCustomEnv: false
    jobAnnotations:
      "argocd.argoproj.io/hook": Sync

Have you joined Openmetadata Slack community?

No

[wintertag]

Same problem here! There's also an open issue on Airflow's side about it, unfortunately, it's unaddressed since: apache/airflow#68483

[axcawef]

We are seeing what appears to be the same dependency conflict on OpenMetadata / openmetadata-dependencies chart 1.13.3, although with a different symptom.

Our Airflow pods start successfully, but the OpenMetadata Pipeline Service Client reports:

Platform Service Client Unavailable

In the running openmetadata-dependencies-api-server pod we have:

openmetadata-ingestion: 1.13.3.0
openmetadata-managed-apis: 1.13.3.0
apache-airflow: 3.1.8
apache-airflow-providers-fab: 2.4.4
SQLAlchemy: 1.4.54

However, openmetadata-ingestion==1.13.3.0 requires:

sqlalchemy >=2.0.0,<3

The openmetadata_managed_apis Airflow plugin then fails to import with:

ImportError: cannot import name 'DeclarativeBase' from 'sqlalchemy.orm'

As a consequence:

airflow plugins

returns:

No plugins loaded

The OpenMetadata managed API endpoint:

/pluginsv2/api/v2/openmetadata/health-auth

does not reach the OpenMetadata managed API plugin. Instead, it falls through to the Airflow UI route.

The OpenMetadata server then tries the available Airflow API variants and eventually reports that no working Airflow API endpoint can be detected.

This results in:

Platform Service Client Unavailable

and prevents actions such as Test Connection and AutoPilot / ingestion agent deployment.

As an additional comparison, we tested the pristine image directly:

docker.getcollate.io/openmetadata/ingestion:1.13.3

The untouched image contains:

SQLAlchemy: 2.0.49

and the following import succeeds:

import openmetadata_managed_apis.plugin

Result:

OPENMETADATA PLUGIN IMPORT: OK

In contrast, the pod deployed through the openmetadata-dependencies Helm chart ends up with:

SQLAlchemy: 1.4.54

and the same plugin import fails.

The chart currently configures:

_PIP_ADDITIONAL_REQUIREMENTS=apache-airflow-providers-fab==2.4.4

The runtime environment also contains:

Flask-AppBuilder: 4.6.3

which declares:

SQLAlchemy <1.5

while openmetadata-ingestion==1.13.3.0 requires:

SQLAlchemy >=2.0.0,<3

This strongly suggests that the runtime installation of:

apache-airflow-providers-fab==2.4.4

through _PIP_ADDITIONAL_REQUIREMENTS is causing SQLAlchemy to be downgraded from the 2.x version included in the official ingestion:1.13.3 image to SQLAlchemy 1.4.x.

This results in an internally inconsistent Python environment and prevents the openmetadata_managed_apis Airflow plugin from loading.

So, in addition to the Airflow migration failure originally reported in this issue, the same dependency pin appears to cause another failure mode in 1.13.3:

1. Airflow starts successfully.
2. SQLAlchemy is downgraded to 1.4.54.
3. openmetadata_managed_apis fails to import because DeclarativeBase is unavailable.
4. Airflow reports No plugins loaded.
5. The OpenMetadata managed Airflow API is unavailable.
6. OpenMetadata reports Platform Service Client Unavailable.

For reference, our deployment versions are:

OpenMetadata server: 1.13.3
OpenMetadata Helm chart: 1.13.3
OpenMetadata dependencies Helm chart: 1.13.3

Server image:
docker.getcollate.io/openmetadata/server:1.13.3

Airflow / ingestion image:
docker.getcollate.io/openmetadata/ingestion:1.13.3

Runtime:
openmetadata-ingestion: 1.13.3.0
openmetadata-managed-apis: 1.13.3.0
apache-airflow: 3.1.8
apache-airflow-providers-fab: 2.4.4
Flask-AppBuilder: 4.6.3
SQLAlchemy: 1.4.54

The pristine docker.getcollate.io/openmetadata/ingestion:1.13.3 image, before the Helm-configured runtime dependency installation, has:

SQLAlchemy: 2.0.49

and successfully imports openmetadata_managed_apis.plugin.

This looks like the same underlying dependency issue reported here, with a different runtime symptom.

[wintertag]

btw there's also an open issue on Airflow side with the same assessment: apache/airflow#68483
So the root cause is probably not fixable on OpenMetadata side but would it be possible to ship the v1.13 Helmchart with an older Airflow version?

[axcawef]

In my case, it was caused by an Airflow regression requested in the Openmetadata-dependencies helm chart value:

airflow:
  env:
    - name: _PIP_ADDITIONAL_REQUIREMENTS
      value: "apache-airflow-providers-fab==2.4.4"

That was the key culprit.

What happened was:

• The openmetadata/ingestion:1.13.3 image itself was healthy and contained the newer dependency stack.
• At pod startup, _PIP_ADDITIONAL_REQUIREMENTS caused pip to install apache-airflow-providers-fab==2.4.4.
• That pulled the FAB/Flask-AppBuilder dependency tree backwards and resulted in SQLAlchemy 1.4.54 in the running pod.
• OpenMetadata 1.13.3's ingestion/plugin code expects SQLAlchemy 2.x and imports sqlalchemy.orm.DeclarativeBase, which does not exist in SQLAlchemy 1.4.
• Therefore openmetadata_managed_apis.plugin failed to import, so Airflow loaded no OpenMetadata plugin endpoints.
• Consequently OpenMetadata's AirflowRESTClient could not detect /pluginsv2/api/v2/openmetadata/..., producing the UI symptom "Platform Service Client Unavailable".

Why is this _PIP_ADDITIONAL_REQUIREMENTS necessary in the 1.13.x release? Especially when it is running fine without this additional requirement.