Allow notify subtilte to upper app through event

[lcksk]

Expected behavior of the wanted feature

Currently, on the Android platform, if -vo=mediacodec_embed is used, the subtitles feature becomes completely unavailable. Is it possible to notify the app layer via events or callbacks with the already decoded subtitle (img or text) so that the Android app layer can easily receive each frame of subtitle data (timed text or img) and display them using separate TextView or TextureView ？

If subtitles could be sent to the client as an event or callback just before rendering, apps wouldn't have to worry about synchronizing with low-level primitives. They'd only have to focus on displaying the captioned data. This wouldn't just benefit mediacodec_embed, but all other video outputs (vo) that don't support subtitles.

Alternative behavior of the wanted feature

No response

Log File

No response

Sample Files

No response

[lcksk]

In a similar manner how Android MediaPlayer handles MEDIA_TIMED_TEXT, this approach involves the player framework taking care of demuxing and decoding each subtitle track, as well as syncing them with the video frames. It only notifies other display controls of the subtitle data and duration when they need to be rendered, rather than overlaying them directly within the player. If we want to implement this feature, can the current mpv architecture handle it relatively easily?

[ruihe774]

Why not

To use hardware decoding with --vo=gpu instead, use --hwdec=mediacodec or mediacodec-copy along with --gpu-context=android.

I think vo_gpu offers better feature supports and quality. Do you have to use vo_mediacodec_embed?

[lcksk]

Yes, compared to other opengl-based VO , mediacodec_embed has a superior performance and can easily achieve video rendering at 60fps,little cpu overhead. Moreover, it has an advantage that others lacks: on SurfaceView, hardware manufacturers usually use a dedicated Video Panel layer to play videos. This layer is specifically designed for video display and offers better picture quality optimization (PQ) and superior display performance, especially on large screens of TV devices, where the advantages are more pronounced.

[ruihe774]

hardware manufacturers usually use a dedicated Video Panel layer to play videos

I'm afraid they can also add some stupid "optimizations" that change the color, sharpen the picture, etc.

so that the Android app layer can easily receive each frame of subtitle data (timed text or img) and display them using separate TextView or TextureView ？

Rendering ASS is hard. I don't think you can use TextView to render ASS. Even if mpv provides you with image, synchronization is also difficult.

You can modify your mpv to expose such interfaces, but I'm afraid that the upstream won't add such interfaces for the benefit of this single VO. The extra maintenance effort is not worth it.

[lcksk]

I'm afraid they can also add some stupid "optimizations" that change the color, sharpen the picture, etc.

On MTK Android TV devices, it seems we're stuck with no other option if you want better PQ and MEMC,etc..,those can only work by VDP (that's video display panel), which means using SurfaceView.

in practice, SurfaceView really does perform better. It's got superior actual performance and delivers higher image quality (with richer, more vibrant colors).

Rendering ASS is hard. I don't think you can use TextView to render ASS. Even if mpv provides you with image, synchronization is also difficult.

My understanding is that we just pass video frame overlay part to the app to handle,
such as TextureView, handling it on the Java layer is really much easier,converting an image to a texture and then passing the texture to TextureView is much simpler compared to what's below.

and the synchronization part is done inside the player, right? of course, there might be some latency in event callbacks to the app due to asynchronous reasons, like tens of milliseconds or a hundred milliseconds, but it feels relatively acceptable.

For example, the following pseudo-code:

mpv_event_sub sub_event = {
        .type = pic or timed text,
        .start = xxxx(mp_time_ns),
        .end = xxxx + duration(mp_time_ns + duration),
        .size =xxx,
        .data = xxx,
        xxxx
        
    };

    mp_notify(mpctx, mpv_event_sub, &sub_event)

[lcksk]

I'm currently stuck on this and have no idea how to continue.