Cannot click objects in Wolf

I have my talisman in weapon slot 1 and shred on left click. In the options I also have prioritize interact selected. I cannot click on any interactable objects unless I swap to weapon set 2 which puts me out of form. Do I really need to weapon swap each time I need to click an object?

It only works if I put my talisman in weapon slot 2 and leave weapon slot 1 empty. Then I can have shred on left click and still interact with objects while in form, but doing so messes with spirit reservations turning off.