buffer empty on sprite hitTest when using compressed texture

[loudoweb]

Describe the bug
I have the following error when my mouse hits a sprite filled with beginBitmapFill using compressed texture (ATF).

caught TypeError: Cannot read properties of null (reading 'buffer')
    at lime__$internal_graphics_ImageCanvasUtil.convertToCanvas (ImageCanvasUtil.hx:32:16)
    at openfl_display__$internal_CanvasGraphics.createBitmapFill (CanvasGraphics.hx:95:3)
    at openfl_display__$internal_CanvasGraphics.playCommands (CanvasGraphics.hx:765:27)
    at openfl_display__$internal_CanvasGraphics.endFill (CanvasGraphics.hx:226:3)
    at openfl_display__$internal_CanvasGraphics.hitTest (CanvasGraphics.hx:325:7)
    at openfl_display_Graphics.__hitTest (Graphics.hx:1678:12)
    at openfl_display_Sprite.__hitTest (Sprite.hx:316:53)
    at openfl_display_Sprite.__hitTest (DisplayObjectContainer.hx:839:22)
    at openfl_display_Sprite.__hitTest (Sprite.hx:312:7)
    at DocumentClass.__hitTest (DisplayObjectContainer.hx:839:22)

To Reproduce
Steps to reproduce the behavior:

1. Use a ATF texture
2. create a sprite using drawRect or drawTriangles and beginBitmapFill
3. move your mouse onto the sprite
4. See error

Expected behavior
The buffer shouldn't be empty.

OpenFL Targets
all

Code Sample

var context3D = stage.context3D;
var atfData = Assets.getBytes ("assets/kenney-poly.atf");
var texture = context3D.createTexture (4096, 4096, Context3DTextureFormat.COMPRESSED_ALPHA, false);
texture.uploadCompressedTextureFromByteArray (atfData, 0);
var img1:BitmapData = BitmapData.fromTexture (texture);
graphics.beginBitmapFill(img1);
graphics.drawRect(0, 0, 100, 100);
graphics.endFill();

[joshtynjala]

Based on the current implementation, this appears to be the expected behavior.

As far as I can tell, BitmapData.fromTexture() is intended to be a way to make it possible to render the texture in an openfl.display.Bitmap only (and only when rendering OpenFL on the GPU). I would expect that calling most of the common methods on the BitmapData class will probably fail with a similar exception because the texture is never converted from a hardware format to a software format. (It's worth mentioning that BitmapData.fromTexture() does not exist in Flash, and that's probably for good reason because converting hardware textures to software bitmaps is very expensive, performance-wise).

BitmapData.fromTexture() has the following code:

var bitmapData = new BitmapData(texture.__width, texture.__height, true, 0);
bitmapData.readable = false;
bitmapData.__texture = texture;
bitmapData.__textureContext = texture.__textureContext;
bitmapData.image = null;

Notice, in particular, that the image private variable is set to null. That's where the raw software-renderable pixel data is supposed to be stored normally. Only the hardware texture is saved, and it isn't converted for software. However, a number of important operations happen in software. For instance, when you use beginBitmapFill(), it ends up calling ImageCanvasUtil.convertToCanvas(bitmap.image);. But image is null. With normal BitmapData it wouldn't be null.

Basically, in Flash, ATF textures weren't intended for use on the classic display list. They were meant for Stage 3D. OpenFL happens to render the classic display list using Stage 3D by default, which is pretty clever, but OpenFL has not provided a way to make ATF textures work on the classic display list as completely as PNG/JPG/etc. image formats can be.

[joshtynjala]

I added some details to the documentation for BitmapData.fromTexture() (including specifying that Graphics.beginBitmapFill() is one of the operations that does not work).

I also stopped the exception from being thrown when calling Graphics.beginBitmapFill() and bitmapData.readable == false. Instead, a hardware-only texture will render as plain black when passed to Graphics.beginBitmapFill(). So it won't render the texture, for the reasons specified, but it will prevent the app from crashing.

It also makes native targets more consistent with html5 because those previously failed silently without rendering anything.

[loudoweb]

Thanks for your answer.

It also makes native targets more consistent with html5 because those previously failed silently without rendering anything.

No. It renders actually. But crashes on mouse over. I don't know what the convertToCanvas is doing. But in case it's for delimit the bounds of the sprite, we could get/build that information when calling drawRect or drawTriangles. I don't get why it calls that from a hitTest... At least, it would be preferable to stop the exception from being thrown from there (in convertToCanvas instead of displaying a full back sprite. In that case, as an user, I could use the differ library to mimic hitTest.

As far as I can tell, BitmapData.fromTexture() is intended to be a way to make it possible to render the texture in an openfl.display.Bitmap only

No we can use it in Tilemap actually.

I came across this bug during some tests I made. Drawing 464 sprites with Tilemap on html5 takes 275ms. With my polygon library that uses drawTriangles, it takes 178ms and with ATF it takes 45ms. So I'm afraid to lose something, especially because Stage3D looks less convienent and uses Agal instead of GLSL if I'm correct. For the record, with that, I'm also beating Unity3D rendering on native.

[joshtynjala]

It also makes native targets more consistent

No. It renders actually.

I didn't see anything rendering on C++/Neko/HashLink when I tested the same beginBitmapFill() code there.

No we can use it in Tilemap actually.

Fair enough. I often forget about the existence of Tilemap, since it wasn't a Flash API. Regardless, that's a similar use-case to Bitmap in terms of rendering a hardware texture with no access to software pixel data.