The functions you mentioned set the pull-ups that are integrated into the microcontroller and their value is about 50k.
It is a good idea to try with these pull-ups if you have not tried this yet.
In the previous post I was thinking about real pull-up resistors to 3V3 or 5V about 10k on each encoder interface.
Both ways should work but the lower value gives better immunity to interferences.
You can also try to check the signals by multimeter. If the motor is turning slowly enough (reaaaalllyyy slowly) you will be able to observe the signal levels for LOW and HIGH, and check if they are OK - below 0,5V and above 2,5V, respectively. In that way it will be possible to check the hardware.
These are my only ideas for this moment. What motors/encoders do you use? I think I know this from another topic but please confirm