Remove-Tagassignment fails to remove tag, WebUI removes tag fine. Add tag back, and Remove-Tagassignment works

I have a batch of VMs that are exhibiting a strange behavior.

If I run:

Get-TagAssignment -Entity $vm -category $categoryname| Remove-TagAssignment

I get prompted to remove the appropriate tag, to which I say 'Yes', and the command ends with no error.  The tag is not removed though

If I remove the tag manually from the WebUI, the tag is removed as expected.  If I then re-add the same tag, I am able to run the above command and it successfully removes the tag.

I checked the tagassignment tag ID/UID are the same across runs.  This happened for both PowerCLI 11.5 and 12.0.  I have not tested on other versions, and so far as I know only affects a handful of VMs that have moved folders.  Moving the VMs had no discernible effect (i.e. not a permissions issue, also the same administrator account was used for all operations).

Thoughts or suggestions?