It took me a while to figure this out during my first playthrough. Once you realise it's not like harvest moon and there is no end to the game and you can just do what you want for as long as you want, the game became much more enjoyable for me

It's not about worrying that I won't be able to do stuff again, just the existence of the ever-looming timer is stressful. And you have to water your crops every morning, and harvest them when they're ready. You can only plant certain crops in certain seasons. There are requests on the bulletin board that have time limits. Shops are only open on certain days, and I know if I don't do that thing today I won't be able to tomorrow because it will be closed. If the timer didn't matter, then there wouldn't be a timer.

Don't get me wrong, I like Stardew Valley. But I stopped playing it, partly because I found it too stressful. Which isn't great cause people always advertise it as this really relaxing game, and I'm sure it is for them, but I guess my brain just works differently. And I really want to like Stardew Valley more, I want to be able to play it and relax.

I'm not even trying to minmax or anything, I try to take my time, but it doesn't really work. I can't just ignore the timer, cause then I'll pass out and wake up with hardly any stamina.

Edit: Also "you can just focus on one thing" sure, but then what do I do for the rest of the day?