Science fiction presents a vision of the future - it is, I think, an effective mirror for the collective thoughts and beliefs about what is to come. For much of the 20th century people were strongly optimistic about the future - postwar and into the tech boom in the 80s and 90s it seemed like everything was only going to get better.

Nowadays though... we don't have that optimism anymore. We have climate change rapidly escalating, corporation's sucking us dry, states doing fuck all about it. This is reflected in those grim police robots and dark themes, just as the shiny space ships and friendly aliens of the past reflected the optimism of the time.

N.b. I do agree with the other commenter who said audience expectations of "realism" play a role - but I also think audiences have a pretty warped idea of what is realistic.