Okay i'm officially a dumbass - looking at the pinout i can clearly see Arduino pins and normal GPIO pins are mixed up on the board I used (seeed xiao rp2040). Arduino strikes again!

Just checked - of course if works fine on proper pins