"Filler" in terms of anime often refered to original content that was "filled" between the actual adapted content from the manga. This is from a time where some shounen series were running perpetually next to the manga and once caught up had to do something to fill the time. This lead to very loooong series like Dragonball Z where you had to fill an entire episode just with the content of a 20 page chapter and to filler arcs that were often scourned. The issue was, that nothing there could be of any lasting consequence as to not throw a wrench into the actual story developing from the manga.

Now, you use filler as in stuff that apprently fills time in the source, as stuff that doesn't really advance the story and characters in your view. I am not agree with that, but that's another matter.