You aren't imagining things. People got caught up in the weird details, the fact that the plane meant for the white house just happened to not reach its destination (even though George W. Bush who was president at the time was in Florida anyways), the supposed untrustworthiness of the US government (staging terrorist attacks to garner support for things wasn't even a new feature among American agencies, though all confirmed proposals had been rejected by the president), the fact this resembles something out of Nero Caesar's playbook (which would make the whole thing kind of well-established at this point), and the fact that Osama Bin Laden's response message to Americans was "released" just before the next election (almost like they were trying to then garner support for an election).
Seek out reasons to conspiracy-theorize though and you will find an Achilles Heel one out of ten times, and people here conjure them at a megafactory's pace. Raising an eyebrow towards the conspiracy theorists is the fact the circumstances from the Middle Eastern perspective that led to the attack though, as well as the fact there even was direct acknowledgement by Osama Bin Laden and later their hosts in Pakistan at all, make it so that, even if it had been American agents who carried it out, it still might as well have been carried out by Osama Bin Laden by some form of proxy/tribute (in other words, his nation made it impossible to say they hadn't looked forward to overseeing it, and from a war standpoint it would have been an act of war in a way either way, plus there are the witness accounts of the plane passengers, like we should ignore those), and it skews matters that both planes and buildings in New York City were not built to code (absolutely every liberty was taken even considering the more lenient building code at the time, for example the stairs were like motel stairs and the anti-fire system was inadequate), which throws a wrench into discussions of architectural physics (of note, I consider it odd people use physics to determine the suspects, that's more of something that merely makes one wonder the "how" about something we all know physically happened).
Rule of thumb, when people go about this, I would think one should think in terms of a court of law. You're a prosecutor making a case against or in favor or a suspect. Are you going to say "look at the physics of something that clearly happened, that doesn't look right" or "but Emperor Nero did it" or "the person I'm accusing has a track record" or "some things seem awfully convenient"? Maybe you would, but that's you, and testimony would become your nightmare. Also note that I'm sure nobody is saying agnosticism isn't completely possible, even though people would think "alright, either you think they did this or that person did it".