From Graphene's FAQ

Many other devices are supported by GrapheneOS at a source level, and it can be built for them without modifications to the existing GrapheneOS source tree. Device support repositories for the Android Open Source Project can simply be dropped into the source tree, with at most minor modifications within them to support GrapheneOS. In most cases, substantial work beyond that will be needed to bring the support up to the same standards. For most devices, the hardware and firmware will prevent providing a reasonably secure device, regardless of the work put into device support.

To get down to your actual reservations about privacy: when you flash a new Graphene ROM onto your phone, you're replacing all the software down to the low level stuff. The AOSP devs, google devs, XDA devs, and graphene devs refer to it at flashing the firmware. The only google code you're running is the Android bootloader, which goes for any smartphone.

Further, if you look into it, "Google" pixels aren't actually manufactured by Google. This means their hardware is about as trustworthy as any other phone's. As to why Graphene only officially supports Pixels, I do not fully understand their needs/reasoning, just that they have determined it is the best for them.

Basically my point boils down to: if you have issues with the hardware, the same should go for any smartphone. If you're bothered by google software, you needn't worry insofar as you trust the Graphene devs. If you consider the Pixels "tainted" by association to Google, then the same should go for Graphene and any other ROMs, since the kernel is based off of the AOSP—a google run project—and any android phone, for the same reason.

All that being said, CalyxOS supports a slightly wider variety of devices.