Add an OpenGL API-Agnostic OpenGL Widget

[corranwebster]

What is the problem or limitation you are having?

OpenGL is a widely-used and reasonably common standard known and understood by a large number of developers, even though it is not the easiest API to learn. There should be a widget that allows a user to perform OpenGL drawing using whatever OpenGL API they prefer or is available on their platform.

Describe the solution you'd like

The solution would be widgets on each platform that set up an OpenGL view which call back to user functions for initialisation and rendering code. What the user does in these call-back is completely up to them: the widget makes no assumptions about the library which is being uses to interface with OpenGL, permitting the user to choose between libraries like PyOpenGL or ModernGL, to use the native OpenGL interface for their platform if there is one, or potentially to call out to wrapped OpenGL code written in C or C++.

Describe alternatives you've considered

A similar widget, but where Toga has an OpenGL compatibility layer which provides a common OpenGL interface that works cross platform, choosing a particular "best" OpenGL API for each platform which is wrapped by the compatibility layer.

Or possibly something more structured. For example, rather than arbitrary rendering callbacks on the interface widget, the user supplies shader programs, buffer data, etc. which are then used in a pre-defined, standard rendering pipeline defined completely in the interface layer.

Additional context

This isn't necessarily the end-state: as discussed in #51, it probably makes sense to have more smaller, more user-friendly GL API similar to WebGL, which is built on top of this widget.

Tests for this API are probably going to be fairly simple (eg. do the callbacks get called when we expect, etc.) since actual drawing is not the widget's responsibility.

[corranwebster]

In terms of design, what I'm looking at implementing looks like:

• An OpenGLView widget which takes a Renderer as an argument.
• The Renderer has on_init and on_render methods which are given the widget instance and any additional useful state (eg. the on_render method gets passed the size of the widget).
• The OpenGLView is generally similar to Canvas: can't take focus, always enabled, has a redraw method which tells the backend to schedule a redraw.
• The Renderer can't be changed after initialisation (it's not clear how to safely schedule re-initialization). It may be possible to relax this once we have better use cases. But the renderer itself can have state and can change what it's doing, so I don't think this is a big issue.

It would be good to be able to provide mouse/touch data to the renderer, but it's not clear whether it's better to have Canvas-style callbacks and let the user manually hook things up if needed, or pass immediate pointer state through as a parameter to on_render, or both. I'm leaning towards immediate mode, but it needs investigation.

Having a renderer object with two methods rather than passing init and render handlers makes sense because the initialisation and rendering are necessarily intertwined. Plus we don't want async handlers for these.

For testing, care has to be taken that everything OpenGL-related happens on the main thread, so fixtures and tests need to be async in most cases.

[freakboy3742]

This seems like it could be a good approach. OpenGL is (more or less) a cross-platform compatibility API; so leaning into that fact and not re-inventing this particular wheel makes sense to me. If there's an opportunity for a more "Python" OpenGL interface, it might make sense to do that at the PyOpenGL layer, rather than Toga trying to invent one.

However, there's three things that make me wonder whether this should be part of Toga core, or a standalone widget:

1. PyOpenGL doesn't support iOS or Android yet - I'm guessing we'd be able to upstream patches to add that support; but until such time as we have, any usage of the OpenGL widget will require the sort of "if current_platform ..." handling that you've got in the example app, which isn't a good look for Toga code
2. While it's not an especially heavyweight library, it isn't insignificant. Having a dependency on an OpenGL library as part of every Toga application strikes me as somewhat unnecessary.
3. OpenGL and 3d graphics are important for the set of users who need it - but does it need to be part of the core feature set of Toga?

There's the additional benefit that it would provide additional proof that widgets don't have to be in core to be useful.

[corranwebster]

This seems like it could be a good approach. OpenGL is (more or less) a cross-platform compatibility API; so leaning into that fact and not re-inventing this particular wheel makes sense to me. If there's an opportunity for a more "Python" OpenGL interface, it might make sense to do that at the PyOpenGL layer, rather than Toga trying to invent one.

The purpose of this suggestion is just to provide the widget, not the OpenGL drawing - so the user has (some) choice of how they actually call out to OpenGL - PyOpenGL is one solution on desktop, but there's also potentially Pyglet and ModernGL or VisPy's Gloo, and the Qt backend has its own API. On Android, the android.opengl wrappers are available. Currently iOS doesn't have any Python OpenGL wrapper, but ctypes can be used. Web presumably has WebGL.

This works because of the way OpenGL is very "set everything up and then issue a handful of actual drawing commands" working in low-level C code: there is a fairly seamless hand-off between the code in the backend that manages the drawing contexts and

Leaving the OpenGL API layer unspecified makes the base widget more useful for a wider group of uses and integration with existing codebases.

So there's no dependency on PyOpenGL being proposed here. The bare widget should work in the sense of "here is a blank slate, draw how you will", it's up to the user to add the dependencies they need if they need them. And yes, that means that a program that is intended to be cross platform might need a renderer for desktop, and another for android and another for iOS, but given the differences between OpenGL and OpenGL ES, that's likely needed anyway. Also by having the Renderer object be self-contained, it's only one switch, if it's needed.

However, there's three things that make me wonder whether this should be part of Toga core, or a standalone widget:

1. PyOpenGL doesn't support iOS or Android yet - I'm guessing we'd be able to upstream patches to add that support; but until such time as we have, any usage of the OpenGL widget will require the sort of "if current_platform ..." handling that you've got in the example app, which isn't a good look for Toga code

This sort of branching code is likely needed anyway, even if there was a cross-platform compatibility API, because of the difference in shader languages between desktop and mobile.

3. While it's not an especially heavyweight library, it isn't insignificant. Having a dependency on an OpenGL library as part of every Toga application strikes me as somewhat unnecessary.

The code in the #4279 so far doesn't need any extra dependencies unless you create a widget and do some drawing. Even just creating the widget with an empty renderer that does nothing doesn't require extra dependencies. All the Widget-level code currently is using core backend functionality of the respective backends.

5. OpenGL and 3d graphics are important for the set of users who need it - but does it need to be part of the core feature set of Toga?

I think it's low-cost to have the base widget be in Toga (if you look at #4279 the actual widget code is very small; I didn't even have to add any OpenGL dependencies for the testbed), and the same argument could be made about the Canvas widget: 2d vector graphics are important for the users who need it... but not everyone does.

A cross-platform, Pythonic API on top of that could well be better as a separate thing, particularly as it may have utility outside of Toga. And the more I'm messing around with things, the more that cross-platform layer looks to be a big thing, probably bigger than the Canvas widget's drawing API.

The other reason for not having a separate library for the widget is testing: right now we don't have a good story for testing widgets outside of the main toga library (other than copying testbed wholesale and removing the bits that aren't used).

Although I just realised, that having the OpenGL widget in its own library, but part of the Toga repo could work, given the new widget entrypoint system. Although I still think I'm leaning towards the widget being in the main backend codebases.

[corranwebster]

This sort of branching code is likely needed anyway, even if there was a cross-platform compatibility API, because of the difference in shader languages between desktop and mobile.

An additional thought here: there may be ways we can allow the Renderer to hook in to the get_factory look-up mechanism via entrypoints if we want to avoid the branching code issue.

[freakboy3742]

Leaving the OpenGL API layer unspecified makes the base widget more useful for a wider group of uses and integration with existing codebases.

So there's no dependency on PyOpenGL being proposed here. The bare widget should work in the sense of "here is a blank slate, draw how you will", it's up to the user to add the dependencies they need if they need them.

I think I understand the broad direction you're suggesting here, but it's not clear to me what this looks like to the user (even looking at the example app for #4273). It seems like the practice of actually implementing a renderer needs a fairly low level exposure to platform internals.

The example code accepts a widget argument to the callbacks, but then doesn't actually use that argument. Is this all assuming that there only ever be a single OpenGL context? If there isn't... as a user, how do I know that I've got the "right" context, and know that it's in the right format to use on the renderer object that I have?

5. OpenGL and 3d graphics are important for the set of users who need it - but does it need to be part of the core feature set of Toga?

I think it's low-cost to have the base widget be in Toga (if you look at #4279 the actual widget code is very small; I didn't even have to add any OpenGL dependencies for the testbed),

There's two aspects of "cost" here - code size, and conceptual cost. Code size is evidently less of a concern than I first thought; but the conceptual cost of a widget that doesn't have a "full cross platform story" is another matter - that makes it a lot less "like" the rest of Toga, which means some explanations are required.

and the same argument could be made about the Canvas widget: 2d vector graphics are important for the users who need it... but not everyone does.

That's a fair criticism.

The hesitance/pushback here mostly comes from my background with Django. The less that is in the core, the easier it is to maintain the core; and when users actually see useful things outside the core, it seeds the idea that things don't have to be in the core to be useful or good.

The other reason for not having a separate library for the widget is testing: right now we don't have a good story for testing widgets outside of the main toga library (other than copying testbed wholesale and removing the bits that aren't used).

True - but that's also something we can fix :-)

Although I just realised, that having the OpenGL widget in its own library, but part of the Toga repo could work, given the new widget entrypoint system.

Also true - but it would definitely make CI testing complicated - changes in an external repo would need to trigger CI on Toga, or we'd need to duplicate testing... or we'd need to make the testbed app (or parts of it) a re-usable testing artefact...

Although I still think I'm leaning towards the widget being in the main backend codebases.

FWIW - I can still be convinced it should be in core; details on the final product will be the deciding factor.

[corranwebster]

I think I understand the broad direction you're suggesting here, but it's not clear to me what this looks like to the user (even looking at the example app for #4273). It seems like the practice of actually implementing a renderer needs a fairly low level exposure to platform internals.

Implementing a renderer requires access to the "low-level" OpenGL API (there isn't really a "high-level" OpenGL - maybe WebGL or ModernGL), which someone writing OpenGL code is presumably on-board with. Where there's any difference between platforms it's because of the OpenGL API(s) that is/are available. There's no platform dependency at the renderer level beyond that, so the PyOpenGL renderer from the examples works unmodified on Cocoa, Qt and (presumably, I don't have a VM where I can test it right now) Gtk. Possibly even on Windows, depending on what the OpenGL widget there ends up looking like. If, in the future, PyOpenGL is ported to android or iOS, it should Just Work there too.

The code of the rendered for different OpenGL APIs is very similar too: most of the differences are around how to convert Python objects to the appropriate sort of C array objects and so the calling conventions and signatures are different (eg. whether or not you need to pass the length of a list/array of numbers along with the list/array itself, and how you provide that array), made a little more complex by Android going through Java first. Plus a little bit of OpenGL vs OpenGL ES.

The example code accepts a widget argument to the callbacks, but then doesn't actually use that argument. Is this all assuming that there only ever be a single OpenGL context? If there isn't... as a user, how do I know that I've got the "right" context, and know that it's in the right format to use on the renderer object that I have?

No, the contract of the OpenGLView widget is that the implementation does all the set-up so that OpenGL is ready to set-up/render into the correct context before the Renderer is called (some backends do this for you automatically, some others need the implementation to get the widget's context and make it current) and makes sure the rendered pixels get displayed on the screen (some backends need an explicit step to do this). So all the Renderer needs to worry about is OpenGL drawing set-up (compiling shaders, populating buffers, setting options) and drawing itself. But it also means that you can have as many widgets as you want, in principle.

I included the widget as part of the renderer API very early on before I'd actually written any Renderers. In fact, the earliest attempts had init and render handlers so they had to take a widget as a first argument, but that caused problems (a) if they got mismatched, and (b) they need to be immediate calls, not async, so regular handlers risked a user trying to use an async handler.

My original thought was that things like sizing, mouse and button state would be attached to the widget, but the experience with the shadertoy code is that its better to just pass that through as extra parameters to the render method.

In any case, it may not be needed at all - if the PR ends up basically complete and nothing needs the widget, then we can probably remove it from the signature, and anything which needs the widget can pass it to the Renderer directly at set-up time.

Although I just realised, that having the OpenGL widget in its own library, but part of the Toga repo could work, given the new widget entrypoint system.

Also true - but it would definitely make CI testing complicated - changes in an external repo would need to trigger CI on Toga, or we'd need to duplicate testing... or we'd need to make the testbed app (or parts of it) a re-usable testing artefact...

I think I may not have communicated what I meant. I was considering having a new top-level project in the toga repo (like Travertino or Positron) called something like toga_opengl_widget or similar which provides all the backend widgets and the test probes with appropriate entry-points. This is its own package, eventually published on PyPI.

When testing in testbed, this is just added to the requirements for each backend that supports OpenGL and testbed has tests for the OpenGL widget.

But when installing toga via pip, we add a extra/optional dependency for OpenGL support which adds the toga_opengl_widget package.

That way an install doesn't have to include OpenGL stuff, but if a change to core breaks the OpenGL widgets its immediately apparent, and we don't run the risk of things getting out of sync.

That's significantly more complex than just having another widget in each backend, though.

Although I still think I'm leaning towards the widget being in the main backend codebases.

FWIW - I can still be convinced it should be in core; details on the final product will be the deciding factor.

I hope I can convince you. FWIW the only backend which doesn't seem to have an out-of-the-box OpenGL widget is Winforms, so OpenGL widgets are fairly "core" things for most GUI toolkits.

For the people who really care about OpenGL, "here's a widget, go do your low-level stuff with the library you're used to" is probably what they want. And if we label the widget as Beta until we see how people use it (and possibly how Python OpenGL wrappers develop) and get the inevitable kinks ironed out then that's a reasonable way forward. People may forgive some "un-toga-like" sins if it is clearly labeled as something being worked on.

But yes, all of this is a bit moot until the current PR is in a finished-ish state.

[freakboy3742]

Implementing a renderer requires access to the "low-level" OpenGL API (there isn't really a "high-level" OpenGL - maybe WebGL or ModernGL),

As a side note - WebGL is likely going to be a good candidate for a "cross platform" 3d API if/when we get there, if only because it's the API that is going to exist out of the box for web deployments (much as we've targeted the HTML5 Canvas API for the Canvas widget).

No, the contract of the OpenGLView widget is that the implementation does all the set-up so that OpenGL is ready to set-up/render into the correct context before the Renderer is called (some backends do this for you automatically, some others need the implementation to get the widget's context and make it current) and makes sure the rendered pixels get displayed on the screen (some backends need an explicit step to do this). So all the Renderer needs to worry about is OpenGL drawing set-up (compiling shaders, populating buffers, setting options) and drawing itself. But it also means that you can have as many widgets as you want, in principle.

Sure - but the bit that I'm missing is the glue between the native widget and the thing that PyOpenGL is operating on.

A widget has some "native" widget representation - that's a different actual native widget on every platform. That's the same pattern we have for every other widget; so that makes sense.

PyOpenGL does some abstraction on the platforms so there's a common external API surface. That makes sense as well.

The bit I'm not seeing is how the PyOpenGL API "attaches" to the native widget. There's no references to "widget" (other than input arguments) in any of the example Renderer code, and no references to "native" at all. How is the Renderer finding the native OpenGL context to operate on? How does that implementation know it's go the right instance?

I don't have any issue with the idea that each OpenGLView widget is responsible for setting up the context, or that the PyOpenGL widget can abstract the platform specific - it's literally the "how does PyOpenGL know what it's operating on" that I'm not seeing.

(To be clear - this might be me being unfamiliar with the details of OpenGL. It's been long time since I've done any 3D graphics in anger... as an indication, I used an SGI IndyStation at the time...)

I have a vague recollection of needing to create an OpenGL Context; is this case of the native widget enabling a specific context in the native-appropriate way before invoking on_render, and on_render then being able to assume that it can just operate on the current GL context? If that is the case - is there any risk of the GL context getting switched out underneath the renderer?

I included the widget as part of the renderer API very early on before I'd actually written any Renderers.
...
In any case, it may not be needed at all - if the PR ends up basically complete and nothing needs the widget, then we can probably remove it from the signature, and anything which needs the widget can pass it to the Renderer directly at set-up time.

Even if it isn't needed, it's likely desirable to retain it for consistency purposes. Plus I'm sure someone will want to tweak something or retrieve some property of the triggering widget.

Also true - but it would definitely make CI testing complicated - changes in an external repo would need to trigger CI on Toga, or we'd need to duplicate testing... or we'd need to make the testbed app (or parts of it) a re-usable testing artefact...

I think I may not have communicated what I meant. I was considering having a new top-level project in the toga repo (like Travertino or Positron) called something like toga_opengl_widget or similar which provides all the backend widgets and the test probes with appropriate entry-points. This is its own package, eventually published on PyPI.

Ah - ok. Agreed that would be one possible approach. However, if there's no extra dependencies, and the code for the widget isn't that excessive, I think I'd lean to including the widget in core over that approach. If we're going to make it standalone, we might as well make it fully standalone.

For the people who really care about OpenGL, "here's a widget, go do your low-level stuff with the library you're used to" is probably what they want. And if we label the widget as Beta until we see how people use it (and possibly how Python OpenGL wrappers develop) and get the inevitable kinks ironed out then that's a reasonable way forward. People may forgive some "un-toga-like" sins if it is clearly labeled as something being worked on.

That's also fair. Clearly labelling something as beta does gives us coverage for removing it.

[corranwebster]

As a side note - WebGL is likely going to be a good candidate for a "cross platform" 3d API if/when we get there, if only because it's the API that is going to exist out of the box for web deployments (much as we've targeted the HTML5 Canvas API for the Canvas widget).

Yep, I've been thinking that way - the biggest advantage is that it has distilled out the 20+ years of backwards compatibility cruft and is presenting a fairly clean API - but like the Canvas API there are some tweaks we might make to make it more Pythonic.

The bit I'm not seeing is how the PyOpenGL API "attaches" to the native widget. There's no references to "widget" (other than input arguments) in any of the example Renderer code, and no references to "native" at all. How is the Renderer finding the native OpenGL context to operate on? How does that implementation know it's go the right instance?

I don't have any issue with the idea that each OpenGLView widget is responsible for setting up the context, or that the PyOpenGL widget can abstract the platform specific - it's literally the "how does PyOpenGL know what it's operating on" that I'm not seeing.

In some sense it doesn't. Unlike with the 2D drawing APIs there isn't a "context" which is needed for every call, instead OpenGL has a global current context (it may be global per thread rather than app global), and that's what the drawing calls are working on implicitly. There are OpenGL functions that allow you to query the state of the context you have right now.

(To be clear - this might be me being unfamiliar with the details of OpenGL. It's been long time since I've done any 3D graphics in anger... as an indication, I used an SGI IndyStation at the time...)

For full disclosure, I'm not an OpenGL expert by any means - I've played with it, and have fixed bugs in a 2D drawing backend that used it, but all serious 3D work I've done has been through higher-level scene graph APIs like VTK.

I have a vague recollection of needing to create an OpenGL Context; is this case of the native widget enabling a specific context in the native-appropriate way before invoking on_render, and on_render then being able to assume that it can just operate on the current GL context? If that is the case - is there any risk of the GL context getting switched out underneath the renderer?

Yes, that's exactly it. For example, the core of the Cocoa code looks like this (inside the regular "draw the contents" method)):

    @objc_method
    def drawRect_(self, rect: NSRect) -> None:
        ...
        self.openGLContext.makeCurrentContext()
        try:
            self.interface.renderer.on_render(self.interface, ...)
        finally:
            # show what was drawn up to any error
            self.openGLContext.flushBuffer()

There is some state on the Cocoa NSOpenGLContext object, but it's the sort of thing that would be set via the widget if we needed to expose it (like how big the back-buffer is, for example), not things that need to be changed during actual drawing and anything relevant for drawing can be queried via the OpenGL APIs I think.

Anyway, as long as there's nothing async/threaded going on the system shouldn't swap things out under the user. The whole thing is happening inside the toolkit base widget's "draw" routine, so you need to do all the rendering immediately, but on the other hand the backends tend to be fairly careful about not changing global state while something is drawing. So I think we can be comfortable that the backend won't change the context out while the render method is happening.

Outside of that, if the user called out to native code to explicitly change the context, it could happen, but that would be something very intentional and backend-specific and I can't think of a situation where someone would want to do that during rendering (even rendering to an off-screen frame buffer or texture as part of the rendering call doesn't need to change the context).

Edit: I just ran across the following admonition in the iOS context docs: "Never change the context from inside your drawing method."

I included the widget as part of the renderer API very early on before I'd actually written any Renderers.
...
In any case, it may not be needed at all - if the PR ends up basically complete and nothing needs the widget, then we can probably remove it from the signature, and anything which needs the widget can pass it to the Renderer directly at set-up time.

Even if it isn't needed, it's likely desirable to retain it for consistency purposes. Plus I'm sure someone will want to tweak something or retrieve some property of the triggering widget.

Actually, thinking about it, the widget's style information might realistically be needed by some rendering code. So we keep it. I'll see if I can fold that into an example.

That's also fair. Clearly labelling something as beta does gives us coverage for removing it.

Yep, a small, beta widget is low-stakes if it doesn't bring in big dependencies.