[Release 0.6] arm tests failing

[metascroy]

After merging #10078, the following tests are consistently failing (https://hud.pytorch.org/hud/pytorch/executorch/release%2F0.6/1?per_page=50):

• pull / unittest-arm / linux-job
• trunk / test-arm-reference-delegation / linux-job

[metascroy]

@SS-JIA can you have a look?

[SS-JIA]

#10114