If they are on the same l2 network then your options are basically either to use nginx ACLs or a local firewall on the nginx host since the traffic wont traverse the firewall.
Something like iptables -I INPUT -s 1.2.3.0/24 -j DROP on the nginx host should work
I regret moving my phone number to Google Voice because at this point I fully expect Google to someday kill the service.