Plex

What I would do if look at every point in the video and audio chain and validate they all support hardware decoding (or passthrough / direct stream) for your supported codec.

Based on what you told me. I know your chain might look something like this:

Server
  > Network
     > streamer
        > HDMI
           > TV
        > audio cable
           > receiver or amp
              > audio cable
                 > speaker

For your two speaker setup, is it hooked up with a digital cable or analogue?

If the stereo is analogue, does it support audio at a different sampling and bitrate than what the player puts out?

Or if it's digital, does it support AC3? In a quick search I saw complaints about EAC3 not working well despite being advertised as such. So I wouldn't rule it out.

When you stream from the Roku, you can look in your media servers settings to see if it's transcoding the video or audio. However for the stereo system, or whatever is powering the speakers, you would have to reference its supported formats.

I did find the Roku Developers page and something noteworthy is

Supported video codecs

Videos can be encoded using H.264, HEVC (H.265), VP9, or AV1 (DASH only) codecs.

Something else that may be a good lead, looking at their supported audio codecs it lists

16bit | 48 Khz | Passthrough

For AC3, this is a but ambiguous to me, if given 44.1hkz (much more popular), will it pass through 44.1khz? Will it be delayed? Will it refuse to play? They also don't specify if this is hardware accelerated or software decoding within the Roku.

If the Roku's cpu is being fully utilized decoding video in software, it may not have the cpu to decode the audio reliably. This is unlikely as most streamers are built with hardware decoding support to enable cheaper smaller lower power components to be used.

I did see on the forums people recommend AAC for general support but Roku themselves recommenf E-AC3 for 5.1 audio.

People often recommend forcing Direct Stream and disabling transcoding as a troubleshooting step to force out the container formats that won't work. It may also be worth a test and see if you lose audio.

As for where I look specifically, I tend to look for the most verbose documentation I can find, the Roku developer docs i linked would be what I would look for. I also did a lot of research on audio gear which helped as prior knowledge. Audio-science review forums taught me a lot over time from reading reviews.

As an aside I generally recommend Open Source formats where possible (AC3 and EAC3 are not) because they tend to have broader support in perpetuity. I also find Android players tend to have very broad codec support (piggybacking mobile phone components).

But I don't like to recommend new hardware in general because not everyone can buy a new device.

This is a bit all over the place at this point, but I hope it gives you some leads for determining the cause!