Render Canvas to an arbitrarily-sized image

[johnzhou721]

What is the problem or limitation you are having?

I am building an app in which a preview of a document is displayed onscreen that is drawn using Canvas; however, I needed the ability to export that preview onto an arbitrarily-sized image.

Describe the solution you'd like

A size parameter on as_image of Canvas that lets you export a scaled view of the Canvas.

This will be implemented in most backends by scaling the Canvas before rendering onto an image; reset_transform must be handled appropriately. Cocoa and iOS allows for setting a different logical size for an image than the physical size, which can be useful to implement this

Describe alternatives you've considered

1. Status quo. One must manually resize the canvas, wait for layout, and then grab the image, and then reset image.
2. An canvas-like API on toga.Image that allows directly annotating an image.

Additional context

A desktop implementation (with some additional fixes) without reset_transform support lives in johnzhou721/togax-frost, my personal component library. It has been used in johnzhou721/tiler successfully.

[freakboy3742]

My initial thought was that this was out of scope for Toga - resizing an image should be the domain of image manipulation after initial creation. However, it occurred to me that if you have a very large canvas, and you only need a small thumbnail, that will involve using a lot of memory for a temporary image; allowing for rendering directly to an arbitrary image size would be an optimization that wouldn't be possible otherwise.

So - the question then becomes the specific API - especially when it comes to aspect ratio.

If I have a 100x100 canvas, and I request a 200x300 image... do I get a 200x300 image, even though the aspect ratio will be different? Or do I get a 200x200 or 300x300 image? Should I be able to request a "200 x None" or "None x 300" image to preserve aspect ratio? We're going to have to allow None as a size specifier for "give me the native image size", so maybe allowing None on a single axis makes sense as well?

[johnzhou721]

My initial thought was that this was out of scope for Toga - resizing an image should be the domain of image manipulation after initial creation. However, it occurred to me that if you have a very large canvas, and you only need a small thumbnail, that will involve using a lot of memory for a temporary image; allowing for rendering directly to an arbitrary image size would be an optimization that wouldn't be possible otherwise.

The use case was actually converse: Im drawing the output of my app onto a thumbnail / preview Canvas to be exported to a larger image. That said, a better option would’ve been to draw to a pillow image, and then use an ImageView…

None on a single axis makes sense and mirrors our ImageView API, with unset width and height meaning autoScale.

that will involve using a lot of memory for a temporary image;

Well—if we’re gonna cover these optimizations, then we should also do something else: allowing a portion of the canvas to be drawn directly to the image. My thought on this would be to add a type toga.Region for top left and bottom right points, and allow people to pass in a toga.Region for cropping.

But that cropping can and should be left for later in another ticket, do you agree?

[freakboy3742]

But that cropping can and should be left for later in another ticket, do you agree?

I'm less convinced that cropping would be in scope. The primary argument I can see for scaling is that image scaling is a direct mapping to a transformation (i.e., you can apply a transformation matrix to the canvas drawing instructions), so there's no need for a library like Pillow to perform the scaling. However, there's no transform that can be used for cropping - you will always need to render the entire image (as you can't easily evaluate if a line/curve will overlap the crop area); so you'll always need to render the entire image.

[johnzhou721]

I'm less convinced that cropping would be in scope. The primary argument I can see for scaling is that image scaling is a direct mapping to a transformation (i.e., you can apply a transformation matrix to the canvas drawing instructions), so there's no need for a library like Pillow to perform the scaling. However, there's no transform that can be used for cropping - you will always need to render the entire image (as you can't easily evaluate if a line/curve will overlap the crop area); so you'll always need to render the entire image.

Since you brought this up, I understand stand the fact that drawing everything still has a large cost, but I had some different arguments in mind that I suggest considering:

Drawing out-of-bounds on an image would be clipped AFAICT, so one can apply a translation and an appropriate scale to do this.

While having to render all elements is true for rendering, and does not give savings there, there’s a significant cost needed to open a large memory buffer for the large image, and pillow is really slow when loading images. So if we have a smaller offscreen buffer to start with, that would be useful.

Thoughts? If you think this is not worth the cost of an extra transform, I acknowledge that concern and would drop discussion of this part of the feature.

[freakboy3742]

Drawing out-of-bounds on an image would be clipped AFAICT, so one can apply a translation and an appropriate scale to do this.

That's not my understanding of how the image would be rendered; but I'll freely admit this is an area where I don't know the underlying APIs well, so it's entirely possible that I'm wrong.

If it's possible to specify a subarea of an image (defaulting to the whole image), and a size for the target image (defaulting to the size of the original); and APIs of that "flavour" are available on most platforms, then I could see it being in scope. That said - a working implementation with 1 feature is better than a theoretical design for an implementation with 5 features - and a 1 feature version will generally be easier to review as well.

[johnzhou721]

That's not my understanding of how the image would be rendered; but I'll freely admit this is an area where I don't know the underlying APIs well, so it's entirely possible that I'm wrong.

It’s entirely possible that I am wrong too. To be clear—even rendering to arbitrary size requires us to add a custom transform and track it appropriately on most platforms. But:

That said - a working implementation with 1 feature is better than a theoretical design for an implementation with 5 features - and a 1 feature version will generally be easier to review as well.

Sure—let’s limit discussion to rendering to arbitrary size here.

Is there anything else we want to discuss before we make this ready to be picked up? Is the fact that we need to manually add a scale transform before drawing to an image a blocker?

[freakboy3742]

Is there anything else we want to discuss before we make this ready to be picked up? Is the fact that we need to manually add a scale transform before drawing to an image a blocker?

I'm not sure I understand the question. What I was suggesting about a scaling transform is an implementation detail - that we can tell the canvas renderer to "apply a 2x scale", then apply the canvas drawing instructions, and you'll get an image a 2x size. The reason it's relevant is efficiency - you don't have to draw a 1x sized image, then scale that image to a 2x sized image - you can get the 2x image in a one shot render.

I can't think of anything else that needs to be discussed, though; as_image(format) becomes as_image(format, size=None) (and/or, if it falls out easily, as_image(format, size=None, crop=None)).

[johnzhou721]

Thanks. Apologies for the weirdly-worded question—I wanted to confirm that the implementation detail is acceptable.

I have no additional concerns by this point.