You want the max possible signal voltage swing at the ptx/resistor junction as the encoder wheen spins, but it will never be zero nor will it be Vcc. If you're getting Vcc, check your connectiobs & ptx polarity. Then adjust your resistors. For a reflective sensor, you need lots of LED current, so a low LED resistor, and a high ptx resistor. 1-4 volts is pretty good signal swing, and the PicAxe's schmidt triggers do a good job of filtering:
http://www.youtube.com/watch?v=PfMnl...3&feature=plcp