LXDInstance.exists() can error if project doesn't exist in LXD 6.6

[jonathan-conder]

Bug Description

When a project doesn't exist, lxc --project <project> list now fails instead of succeeding with no output. This causes downstream commands like *craft clean to fail sometimes in our CI.

To Reproduce

$ sudo snap remove --purge lxd
lxd removed
$ sudo snap install --revision 36020 lxd
lxd 6.5-ccdfb39 from Canonical✓ installed
$ uv run python -c 'from craft_providers.lxd import LXDInstance; LXDInstance(name="u1", project="p1").exists()'
$ sudo snap refresh --channel 6/stable lxd
lxd (6/stable) 6.6-3c9aa6d from Canonical✓ refreshed
$ uv run python -c 'from craft_providers.lxd import LXDInstance; LXDInstance(name="u1", project="p1").exists()'
Traceback (most recent call last):
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 917, in list
    proc = self._run_lxc(
           ^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 206, in _run_lxc
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['lxc', '--project', 'p1', 'list', 'local:', '--format=yaml']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxd_instance.py", line 286, in exists
    return self._get_instance_information() is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxd_instance.py", line 316, in _get_instance_information
    instances = self.lxc.list(project=self.project, remote=self.remote)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 921, in list
    raise LXDError(
craft_providers.lxd.errors.LXDError: Failed to list instances for project 'p1'.
* Command that failed: 'lxc --project p1 list local: --format=yaml'
* Command exit code: 1
* Command standard error output: 'Error: Failed to get project: Project not found\n'

part yaml

Relevant log output

Traceback (most recent call last):
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 917, in list
    proc = self._run_lxc(
           ^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 206, in _run_lxc
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['lxc', '--project', 'p1', 'list', 'local:', '--format=yaml']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxd_instance.py", line 286, in exists
    return self._get_instance_information() is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxd_instance.py", line 316, in _get_instance_information
    instances = self.lxc.list(project=self.project, remote=self.remote)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jono/code/craft-providers/craft_providers/lxd/lxc.py", line 921, in list
    raise LXDError(
craft_providers.lxd.errors.LXDError: Failed to list instances for project 'p1'.
* Command that failed: 'lxc --project p1 list local: --format=yaml'
* Command exit code: 1
* Command standard error output: 'Error: Failed to get project: Project not found\n'

[tomponline]

One way this could be fixed is to avoid using lxc list and calling lxc config show <instance> instead.

This would also avoid the potentially expensive lxc list with full recursion that is being done today.

This would translate to:

GET /1.0/instances/c1

Alternatively if you need to stick with lxc list then consider adding --fast to avoid recursion (if you don't need instance state info) and use server-side filtering with --name=<instance name>.

This would translate to:

GET /1.0/instances?filter=name+eq+c1&recursion=1

Rather than the current call which returns all instances and gathers full status info for each:

/1.0/instances?recursion=2

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-4932.

This message was autogenerated

[lengau]

Thanks for the report and the suggestion! This might be a good opportunity to swap this particular method to using pylxd as well.

[tomponline]

Indeed, although keep in mind pylxd deb package wont be in 26.04 so use https://pypi.org/project/pylxd/

[tigarmo]

Another confirmation point here is that this started happening on Snapcraft's spread tests, which afaik use lxd from latest/stable (we should probably fix) that:

2025-12-03T13:47:59.2960203Z 2025-12-03 13:47:58.910   File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/services/provider.py", line 406, in _clean_instance
2025-12-03T13:47:59.2962049Z 2025-12-03 13:47:58.910     provider.clean_project_environments(instance_name=instance_name)
2025-12-03T13:47:59.2963143Z 2025-12-03 13:47:58.910   File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/provider.py", line 78, in clean_project_environments
2025-12-03T13:47:59.2964504Z 2025-12-03 13:47:58.910     if environment.exists():
2025-12-03T13:47:59.2965083Z 2025-12-03 13:47:58.910        ^^^^^^^^^^^^^^^^^^^^
2025-12-03T13:47:59.2965881Z 2025-12-03 13:47:58.910   File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_instance.py", line 286, in exists
2025-12-03T13:47:59.2967291Z 2025-12-03 13:47:58.910     return self._get_instance_information() is not None
2025-12-03T13:47:59.2968066Z 2025-12-03 13:47:58.910            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-12-03T13:47:59.2969159Z 2025-12-03 13:47:58.910   File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_instance.py", line 316, in _get_instance_information
2025-12-03T13:47:59.2970323Z 2025-12-03 13:47:58.910     instances = self.lxc.list(project=self.project, remote=self.remote)
2025-12-03T13:47:59.2971287Z 2025-12-03 13:47:58.910                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-12-03T13:47:59.2972214Z 2025-12-03 13:47:58.910   File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 921, in list
2025-12-03T13:47:59.2973088Z 2025-12-03 13:47:58.910     raise LXDError(
2025-12-03T13:47:59.2974393Z 2025-12-03 13:47:58.910 craft_providers.lxd.errors.LXDError: Failed to list instances for project 'snapcraft'.
2025-12-03T13:47:59.2975402Z 2025-12-03 13:47:58.910 * Command that failed: 'lxc --project snapcraft list local: --format=yaml'
2025-12-03T13:47:59.2976303Z 2025-12-03 13:47:58.910 * Command exit code: 1
2025-12-03T13:47:59.2978459Z 2025-12-03 13:47:58.910 * Command standard error output: 'To start your first container, try: lxc launch ubuntu:24.04\nOr for a virtual machine: lxc launch ubuntu:24.04 --vm\n\nError: Failed to get project: Project not found\n'
2025-12-03T13:47:59.2980827Z 2025-12-03 13:47:58.910 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20251203-134758.040290.log'

[tomponline]

@lengau are you able to run your daily tests against LXD 6/candidate to get early warning of similar changes in the future?

[lengau]

@tomponline I'd like to get us running against a matrix of candidate and edge releases going back to 4.0/candidate on Jammy — the primary issue has been time to get that written.

[lengau]

@tomponline — given all this I made a quick and dirty version of the workflow that we can revisit to do it properly another time. #857

[tomponline]

@tomponline — given all this I made a quick and dirty version of the workflow that we can revisit to do it properly another time. #857

Excellent thank you!