Unable to remove port group

I'm trying trying to remove a portgroup named test from vCenter 6.5 but I'm unable to because vCenter says the port group is in use: "The resource "test" is in use."

There aren't VM's connected to this port group. How do I remove the port group?

If I try it directly by logging into the ESXi host it also fails.

It's a production server so I can't put it in maintenance mode nor can I reboot.