It looks like it may be related to new comments, but they're not yellow every time. I just went on a post with "23 [Comments] (10 new)", but only one is yellow. I'm using the darkly theme.
I find it annoying because I can't easily see if something is a top-level comment or a second level comment.