PS5 triggers not working with mount sometimes

When playing with the PS5 controller and entering a new map the skills on the triggers L2, R2 are not working while riding a mount. Usually I have to unmount, use the triggers a few times and mounting again for them to start working again.

I think it has something to do with skill being bound to different weapon sets and fast switching between while having only one weapon but two different quivers equipped:

I use:
- 1 bow in both sets.
- 2 different quivers in each set.
- The skill on R2 that is often not working Lightning Rod with Rigwald's Ferocity gem.
- The Lightning Rod is set to use weapon set 2. All other skills use set 1.