Repeated use of Gemcutter's Prism bugged

PS5 Pro (I suspect this one is just "controller" but whatever)
The interaction technically works, as in the currency is still ready to be used, but after each use the cursor jumps away and the button now takes you to skill assignment window. You can move the cursor back but for some reason it takes multiple jumps all around inventory even though it's just a bit to the right initially.