Skip to content

Always sort AdaptationSets by mimeType #2612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Always sort AdaptationSets by mimeType #2612

wants to merge 2 commits into from
+68 −0

Conversation

albertdaurell
Copy link

@albertdaurell albertdaurell commented Jul 16, 2025 •
edited
Loading

Version

Media3 main + 1.6.1

Details

When playing Live DASH Manifest with SSAI ads the SSAI service can send AdaptationSets randomly sorted:

  • First time:

    <Period id="1" >
  <AdaptationSet mimeType="video/mp4" >
     <Representation id="1" />
     <Representation id="2" />
     <Representation id="3" />
     <Representation id="4" />
  </AdaptationSet>
  <AdaptationSet mimeType="audio/mp4" >
     <Representation id="5" />
  </AdaptationSet>
</Period>

  • Second time:

    <Period id="2" >
  <AdaptationSet mimeType="audio/mp4" >
     <Representation id="5" />
  </AdaptationSet>
  <AdaptationSet mimeType="video/mp4" >
     <Representation id="1" />
     <Representation id="2" />
     <Representation id="3" />
     <Representation id="4" />
  </AdaptationSet>
</Period>

So, DefaultDashChunkSource::updateManifest receives the manifest update and because adaptationSetIndex changed there is a IndexOutOfBoundsException, BUT the main problem is that even if we "avoid" the exception... audio + video tracks will be mixed and a media error will be thrown!

Proposed solution

Always sort AdaptationSet by mimeType, so indexes will not change.

Logs

Logs without the fix, see java.lang.IndexOutOfBoundsException: Index 4 out of bounds for length 1 error:

2025-07-16 11:49:11.744  5223-5272  ExoPlayerImplInternal   androidx.media3.demo.main            E  Playback error
                                                                                                      androidx.media3.exoplayer.ExoPlaybackException: Source error
                                                                                                          at androidx.media3.exoplayer.ExoPlayerImplInternal.handleIoException(ExoPlayerImplInternal.java:933)
                                                                                                          at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:909)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:102)
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205)
                                                                                                          at android.os.Looper.loop(Looper.java:294)
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67)
                                                                                                      Caused by: androidx.media3.exoplayer.upstream.Loader$UnexpectedLoaderException: Unexpected IndexOutOfBoundsException: Index 4 out of bounds for length 1
                                                                                                          at androidx.media3.exoplayer.upstream.Loader$LoadTask.handleMessage(Loader.java:523)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205) 
                                                                                                          at android.os.Looper.loop(Looper.java:294) 
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67) 
                                                                                                      Caused by: java.lang.IndexOutOfBoundsException: Index 4 out of bounds for length 1
                                                                                                          at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
                                                                                                          at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
                                                                                                          at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
                                                                                                          at java.util.Objects.checkIndex(Objects.java:359)
                                                                                                          at java.util.ArrayList.get(ArrayList.java:434)
                                                                                                          at androidx.media3.exoplayer.dash.DefaultDashChunkSource.updateManifest(DefaultDashChunkSource.java:330)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaPeriod.updateManifest(DashMediaPeriod.java:187)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource.processManifest(DashMediaSource.java:938)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource.onManifestLoadCompleted(DashMediaSource.java:777)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1482)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1468)
                                                                                                          at androidx.media3.exoplayer.upstream.Loader$LoadTask.handleMessage(Loader.java:519)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:106) 
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205) 
                                                                                                          at android.os.Looper.loop(Looper.java:294) 
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67) 
2025-07-16 11:49:11.747  5223-5223  EventLogger             androidx.media3.demo.main            D  droppedFrames [eventTime=107.49, mediaPos=39.95, window=0, period=2, 2]
2025-07-16 11:49:11.754  5223-5365  PipelineWatcher         androidx.media3.demo.main            D  onWorkDone: frameIndex not found (18446744073709551615); ignored
2025-07-16 11:49:11.754  5223-5365  PipelineWatcher         androidx.media3.demo.main            D  onWorkDone: frameIndex not found (18446744073709551615); ignored
2025-07-16 11:49:11.754  5223-5365  PipelineWatcher         androidx.media3.demo.main            D  onWorkDone: frameIndex not found (18446744073709551615); ignored
2025-07-16 11:49:11.754  5223-5365  PipelineWatcher         androidx.media3.demo.main            D  onWorkDone: frameIndex not found (18446744073709551615); ignored
2025-07-16 11:49:11.754  5223-5272  MediaCodec              androidx.media3.demo.main            D  keep callback message for reclaim
2025-07-16 11:49:11.755  5223-5364  CCodecConfig            androidx.media3.demo.main            I  query failed after returning 19 values (BAD_INDEX)
2025-07-16 11:49:11.755  5223-5364  CCodecConfig            androidx.media3.demo.main            D  c2 config diff is   c2::u32 coded.vui.color.primaries = 1
                                                                                                      c2::u32 input.buffers.max-size.value = 3670016
                                                                                                      c2::u32 output.delay.value = 11
                                                                                                      c2::u32 raw.crop.height = 720
                                                                                                      c2::u32 raw.crop.width = 1280
                                                                                                      c2::u32 raw.max-size.height = 1088
2025-07-16 11:49:11.757  5223-5364  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 1342179345.
2025-07-16 11:49:11.757  5223-5364  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 2415921170.
2025-07-16 11:49:11.757  5223-5364  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 1610614798.
2025-07-16 11:49:11.760  5223-5223  EventLogger             androidx.media3.demo.main            D  videoDisabled [eventTime=107.50, mediaPos=39.95, window=0, period=2]
2025-07-16 11:49:11.764  5223-5223  EventLogger             androidx.media3.demo.main            D  videoSize [eventTime=107.50, mediaPos=39.95, window=0, period=2, w=0, h=0]
2025-07-16 11:49:11.767  5223-5223  EventLogger             androidx.media3.demo.main            D  rendererReady [eventTime=107.51, mediaPos=39.95, window=0, period=2, rendererIndex=0, video, false]
2025-07-16 11:49:11.769  5223-5272  MediaCodec              androidx.media3.demo.main            D  keep callback message for reclaim
2025-07-16 11:49:11.769  5223-5359  CCodecConfig            androidx.media3.demo.main            I  query failed after returning 20 values (BAD_INDEX)
2025-07-16 11:49:11.770  5223-5359  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 1342179345.
2025-07-16 11:49:11.770  5223-5359  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 2415921170.
2025-07-16 11:49:11.770  5223-5359  Codec2Client            androidx.media3.demo.main            W  query -- param skipped: index = 1610614798.
2025-07-16 11:49:11.773  5223-5223  EventLogger             androidx.media3.demo.main            D  audioDisabled [eventTime=107.51, mediaPos=39.95, window=0, period=2]
2025-07-16 11:49:11.775  5223-5364  SurfaceUtils            androidx.media3.demo.main            D  connecting to surface 0xecc2a828, reason connectToSurface
2025-07-16 11:49:11.775  5223-5364  MediaCodec              androidx.media3.demo.main            I  [c2.mtk.avc.decoder] setting surface generation to 5348358
2025-07-16 11:49:11.775  5223-5364  SurfaceUtils            androidx.media3.demo.main            D  disconnecting from surface 0xecc2a828, reason connectToSurface(reconnect)
2025-07-16 11:49:11.775  5223-5364  SurfaceUtils            androidx.media3.demo.main            D  connecting to surface 0xecc2a828, reason connectToSurface(reconnect)
2025-07-16 11:49:11.775  5223-5364  Codec2-Out...ufferQueue androidx.media3.demo.main            D  C2SurfaceSyncMemory created 20(20)
2025-07-16 11:49:11.775  5223-5364  Codec2-Out...ufferQueue androidx.media3.demo.main            D  remote graphic buffer migration 0/0
2025-07-16 11:49:11.776  5223-5364  Codec2Client            androidx.media3.demo.main            D  setOutputSurface -- generation=5348358 consumer usage=0x900 sync
2025-07-16 11:49:11.780  5223-5364  Codec2Client            androidx.media3.demo.main            D  Surface configure completed
2025-07-16 11:49:11.780  5223-5364  SurfaceUtils            androidx.media3.demo.main            D  disconnecting from surface 0xecc1c578, reason disconnectFromSurface
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.783  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.784  5223-5364  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.mtk.avc.decoder#757] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.785  5223-5359  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.android.aac.decoder#492] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.786  5223-5359  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.android.aac.decoder#492] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.786  5223-5359  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.android.aac.decoder#492] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.786  5223-5359  CCodecBufferChannel     androidx.media3.demo.main            D  [c2.android.aac.decoder#492] MediaCodec discarded an unknown buffer
2025-07-16 11:49:11.786  5223-5223  EventLogger             androidx.media3.demo.main            D  videoDecoderReleased [eventTime=107.53, mediaPos=39.95, window=0, period=2, c2.mtk.avc.decoder]
2025-07-16 11:49:11.787  5223-5359  hw-BpHwBinder           androidx.media3.demo.main            I  onLastStrongRef automatically unlinking death recipients
2025-07-16 11:49:11.789  5223-5223  EventLogger             androidx.media3.demo.main            D  audioDecoderReleased [eventTime=107.53, mediaPos=39.95, window=0, period=2, c2.android.aac.decoder]
2025-07-16 11:49:11.798  5223-5359  MediaCodec              androidx.media3.demo.main            D  flushMediametrics
2025-07-16 11:49:11.799  5223-5359  MediaCodec              androidx.media3.demo.main            D  flushMediametrics
2025-07-16 11:49:11.799  5223-5359  MediaCodec              androidx.media3.demo.main            W  no metrics handle found
2025-07-16 11:49:11.800  5223-5364  SurfaceUtils            androidx.media3.demo.main            D  disconnecting from surface 0xecc2a828, reason disconnectFromSurface
2025-07-16 11:49:11.803  5223-5385  hw-BpHwBinder           androidx.media3.demo.main            I  onLastStrongRef automatically unlinking death recipients
2025-07-16 11:49:11.808  5223-5364  MediaCodec              androidx.media3.demo.main            D  flushMediametrics
2025-07-16 11:49:11.808  5223-5364  MediaCodec              androidx.media3.demo.main            D  flushMediametrics
2025-07-16 11:49:11.809  5223-5364  MediaCodec              androidx.media3.demo.main            W  no metrics handle found
2025-07-16 11:49:11.813  5223-5223  EventLogger             androidx.media3.demo.main            D  timeline [eventTime=107.56, mediaPos=39.95, window=0, period=2, periodCount=3, windowCount=1, reason=SOURCE_UPDATE
2025-07-16 11:49:11.814  5223-5223  EventLogger             androidx.media3.demo.main            D    period [60.00]
2025-07-16 11:49:11.815  5223-5223  EventLogger             androidx.media3.demo.main            D    period [29.96]
2025-07-16 11:49:11.815  5223-5223  EventLogger             androidx.media3.demo.main            D    period [?]
2025-07-16 11:49:11.816  5223-5223  EventLogger             androidx.media3.demo.main            D    window [40.00, seekable=true, dynamic=true]
2025-07-16 11:49:11.816  5223-5223  EventLogger             androidx.media3.demo.main            D  ]
2025-07-16 11:49:11.828  5223-5223  EventLogger             androidx.media3.demo.main            E  playerFailed [eventTime=107.56, mediaPos=39.95, window=0, period=2, errorCode=ERROR_CODE_IO_UNSPECIFIED
                                                                                                      androidx.media3.exoplayer.ExoPlaybackException: Source error
                                                                                                          at androidx.media3.exoplayer.ExoPlayerImplInternal.handleIoException(ExoPlayerImplInternal.java:933)
                                                                                                          at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:909)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:102)
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205)
                                                                                                          at android.os.Looper.loop(Looper.java:294)
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67)
                                                                                                      Caused by: androidx.media3.exoplayer.upstream.Loader$UnexpectedLoaderException: Unexpected IndexOutOfBoundsException: Index 4 out of bounds for length 1
                                                                                                          at androidx.media3.exoplayer.upstream.Loader$LoadTask.handleMessage(Loader.java:523)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205) 
                                                                                                          at android.os.Looper.loop(Looper.java:294) 
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67) 
                                                                                                      Caused by: java.lang.IndexOutOfBoundsException: Index 4 out of bounds for length 1
                                                                                                          at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
                                                                                                          at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
                                                                                                          at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
                                                                                                          at java.util.Objects.checkIndex(Objects.java:359)
                                                                                                          at java.util.ArrayList.get(ArrayList.java:434)
                                                                                                          at androidx.media3.exoplayer.dash.DefaultDashChunkSource.updateManifest(DefaultDashChunkSource.java:330)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaPeriod.updateManifest(DashMediaPeriod.java:187)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource.processManifest(DashMediaSource.java:938)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource.onManifestLoadCompleted(DashMediaSource.java:777)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1482)
                                                                                                          at androidx.media3.exoplayer.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1468)
                                                                                                          at androidx.media3.exoplayer.upstream.Loader$LoadTask.handleMessage(Loader.java:519)
                                                                                                          at android.os.Handler.dispatchMessage(Handler.java:106) 
                                                                                                          at android.os.Looper.loopOnce(Looper.java:205) 
                                                                                                          at android.os.Looper.loop(Looper.java:294) 
                                                                                                          at android.os.HandlerThread.run(HandlerThread.java:67) 
                                                                                                    ]
2025-07-16 11:49:11.832  5223-5223  EventLogger             androidx.media3.demo.main            D  state [eventTime=107.57, mediaPos=39.95, window=0, period=2, IDLE]
2025-07-16 11:49:11.939  5223-5223  EventLogger             androidx.media3.demo.main            D  isPlaying [eventTime=107.68, mediaPos=39.95, window=0, period=2, false]
2025-07-16 11:49:11.955  5223-5223  EventLogger             androidx.media3.demo.main            D  audioTrackReleased [eventTime=107.70, mediaPos=39.95, window=0, period=2, 2,12,48000,false,false,144000]
2025-07-16 11:49:17.122  5223-5288  BufferPoolAccessor2.0   androidx.media3.demo.main            D  bufferpool2 0x993cbae8 : 0(0 size) total buffers - 0(0 size) used buffers - 1508/1517 (recycle/alloc) - 9/1503 (fetch/transfer)
2025-07-16 11:49:17.122  5223-5288  BufferPoolAccessor2.0   androidx.media3.demo.main            D  bufferpool2 0x993c6f88 : 0(0 size) total buffers - 0(0 size) used buffers - 820/829 (recycle/alloc) - 9/825 (fetch/transfer)
2025-07-16 11:49:17.122  5223-5288  BufferPoolAccessor2.0   androidx.media3.demo.main            D  evictor expired: 2, evicted: 2

@marcbaechinger marcbaechinger self-assigned this Jul 18, 2025
@marcbaechinger marcbaechinger force-pushed the sort-adaptationsets-by-mimetype branch from cb34246 to 59239b9 Compare July 18, 2025 09:37
@marcbaechinger
Copy link
Contributor

I'm going to send this for internal review now. You may see some more commits being added as I make changes in response to review feedback. Please refrain from pushing any more substantive changes as it will complicate the internal review - thanks!

@marcbaechinger marcbaechinger self-requested a review July 18, 2025 09:37
@tonihei
Copy link
Collaborator

tonihei commented Jul 18, 2025

After some further thought, we won't be able to merge the PR as it is. The reason is that the DASH spec has no indication that ordering adaptation sets by mime type is required or expected.

You are pointing out a valid issue though. When updating MPDs, the id tags should be used to match Period, AdaptationSet and Representation instances to each other. Your example interestingly also shows that the AdaptationsSet tags have no id, but this should still work. Citing the spec (ISO 23009-1 5th Edition, section 5.4.1):

the values of any AdaptationSet@id attributes shall be the same in the original and the updated MPD unless the containing Period element has been removed

any Representation with the same @id and within the same Period as a Representation appearing in the previous MPD shall provide functionally equivalent attributes and elements and shall provide functionally identical Segments with the same indices in the corresponding Representation in the new MPD

So what should really happen is that we match adaptation sets id values, and lacking that, representation id values, which both need to be unique within their Period. Luckily, id values are mandatory on Representation, so this should always work. The part about functionally identical may be a problem though, so we should be prepared to do a bit of fuzzy matching if ids are duplicated.

Although I can't find it explicitly for this case, the DASH spec generally uses the position in the MPD if this unique id is not provided (e.g. for patch updates). So matching by index in the manifest if no id match is possible is still a valid fallback I'd say.

Where does that actually go wrong though?

  • As far as I can see DashMediaSource makes no use of the adaptation sets and presentations, so the update is only relevant when updating existing DashMediaPeriod instances.
  • DashMediaPeriod uses adaptationSetIndices in its TrackGroupInfo class
  • This in turn is mostly used to create the DefaultDashChunkSource instances (which also store the adaptationSetIndices locally).

So I think the actual fix needs to identify the new adaptationSetIndices for each TrackGroupInfo, update that locally, and also update each existing DashChunkSource with its new adaptationSetIndices via the updateManifest method.

The DASH IOP guidelines also prescribe that adaptations and representations within the same Period should not be removed. So at least this logic doesn't need to handle cases where some of those indices suddenly disappear.

Would you be willing to look into an updated PR that does that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marcbaechinger marcbaechinger Awaiting requested review from marcbaechinger

Assignees

@marcbaechinger marcbaechinger

Labels
should merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@albertdaurell @marcbaechinger @tonihei