backend: (llvm) add ctypes conversion registry [STACKED PR]

[sueszli]

Essentially just follows the PyASTContext pattern but with CTYPEs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.86%. Comparing base (895964b) to head (4034d1b).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6183   +/-   ##
=======================================
  Coverage   86.85%   86.86%           
=======================================
  Files         428      429    +1     
  Lines       64253    64294   +41     
  Branches     7369     7373    +4     
=======================================
+ Hits        55808    55849   +41     
+ Misses       6880     6879    -1     
- Partials     1565     1566    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[superlopuh]

This is great, but I'm not sure that this is the optimal design for the mechanism. As it stands, the system is closed, and only the types added in the function can be mapped. I think two other options would be better:

A) have an abstract CTypeConvertible abstract superclass, that all the relevant attributes subclass,
B) have some kind of mapping ctx thing that lets users register conversion functions like we do for the frontend.

My gut feeling says to go for B, even if it's more infrastructure, as then the builtin dialect doesn't have to know about ctypes conversion, and the ctypes conversion doesn't need to know about the builtin dialect, only a third file that can bridge the two.

What do you think?

Also, it feels like we should have a bidirectional conversion, not sure whether this is a conversation for now or for later.

BTW This is the sort of thing where it would first be useful to have a bigger PR that would have an example of the final use-case, to be able to have informed API design discussions based on usage. In this case, we have the JIT prototype in the filecheck folder, maybe that could be updated in this PR also?

[sueszli]

Sasha Feedback:

I think it's fair game to have a bunch of NotImplementedError under the hood.

[sueszli]

Thank you for reviewing and providing clear feedback. Really appreciate it.

BTW This is the sort of thing where it would first be useful to have a bigger PR that would have an example of the final use-case, to be able to have informed API design discussions based on usage.
In this case, we have the JIT prototype in the filecheck folder, maybe that could be updated in this PR also?

Is it OK for me to turn this into an initial draft / stacked PR that we then later break down into smaller PRs?

B) have some kind of mapping ctx thing that lets users register conversion functions like we do for the frontend.

So something similar to PyASTContext?

xdsl/xdsl/frontend/pyast/context.py

Lines 49 to 100 in a44f0a7

	class PyASTContext:
	"""Encapsulate the mapping between Python and IR types and operations."""
	type_registry: TypeRegistry = field(default_factory=TypeRegistry)
	"""Mappings between source code and IR type."""
	function_registry: FunctionRegistry = field(default_factory=FunctionRegistry)
	"""Mappings between functions and their operation types."""
	literal_registry: LiteralRegistry = field(default_factory=LiteralRegistry)
	"""Mappings between literal types and their operation constructors."""
	post_transforms: list[ModulePass] = field(
	default_factory=lambda: [FrontendDesymrefyPass()]
	)
	"""An ordered list of passes to apply to the built module."""
	post_callback: PassPipelineCallbackType | None = default_pipeline_callback
	"""Callback to run between post transforms."""
	ir_context: Context = field(
	default_factory=lambda: Context(allow_unregistered=True)
	)
	"""The xDSL context to use when applying transformations to the built module."""
	def register_type(
	self,
	source_type: type,
	ir_type: TypeAttribute,
	) -> None:
	"""Associate a type in the source code with its type in the IR."""
	self.type_registry.insert(source_type, ir_type)
	def register_function(
	self, function: Callable[..., Any], ir_constructor: Callable[..., Operation]
	) -> None:
	"""Associate a method on an object in the source code with its IR implementation."""
	self.function_registry.insert(function, ir_constructor)
	def register_literal(
	self, value_type: type[object], ir_constructor: Callable[[Any], Operation]
	) -> None:
	"""Associate a Python literal type with an IR constructor."""
	self.literal_registry.insert(value_type, ir_constructor)
	def register_post_transform(self, transform: ModulePass) -> None:
	"""Add a module pass to be run on the generated IR."""
	self.post_transforms.append(transform)
	def register_dialect(self, dialect: Dialect) -> None:
	"""Add a dialect to the context used for transformation."""
	self.ir_context.load_dialect(dialect)

Also, it feels like we should have a bidirectional conversion, not sure whether this is a conversation for now or for later.

Sure, we could come back to this at a later point.

Thanks

[sueszli]

I vibed this first draft (and rigerously reviewed).

Does this kind of align with what you had in mind?

Thanks

[sueszli]

Do you prefer comments or named conditions?

[superlopuh]

I don't know what you mean

[sueszli]

a) having the var is_converter = callable(value) and not isinstance(value, type)

b) inlining callable(value) and not isinstance(value, type) with a comment that explains what it is

[superlopuh]

note that these are not ctypes.c_double, so it's not just translation to/from ctypes that we need here. Although it seems like there's some magic for floats specifically.

[sueszli]

Ok will keep in mind when we work on non-scalar types. For now, should I add a comment?

[superlopuh]

What I mean is that for this example technically it's a regression because previously the jit function had float in the type, which is handled. Now that it's P, R, we've lost any kind of support to detect what's handled and what isn't. Ideally we'd not have this kind of regression, and have a context that would map python objects to ctypes and back to make sure that the thing will either be ill-typed or raise an error if it's used for things that aren't handled.

[sueszli]

Oh, this flew right past me. I added a comment.

that would map python objects to ctypes and back

Now it makes sense. And it has to be both ways because we also then marshall the result back (unpack the result from its ctypes form back into a Python value), right?

[superlopuh]

Sorry, I think I wasn't clear before. This is exactly what I meant. In a way, the ctypes mapping of types is still crucial, but it would help to also complement it with a typing of python values to ctypes values to complete this translation.

[sueszli]

Where would the python_type_context be placed (xdsl.jit.? jit.llvm.?)

[superlopuh]

no, let's do this in this PR, but keep it minimal