The instructions for this switch are not written real well. First you must establish if you have a single pole (lights switched from one location) or true 3-way system (lights switched from two or more locations) It sounds to me like you actually have a single pole switch. The way the dimmer works is that in the "on" position, the black wire from the dimmer is touching one of the two red wires on from the dimmer, and in the "off" position the black wire from the dimmer is touching the other red wire from the dimmer. You will first need to extablish what the 3 wall box wires do, they should be as follows:
1) power in (constanly has voltage)
2) power out to bathroom and hall lights
3) power out to light fixture
Looking at the instructions and comparing to your description, it looks like the wire you have attached to the black wire from the dimmer is your "power in" and then the red wire that comes out with the green wire (dimmer) is the "power out to light fixture" wire from the wall box, and the red wire that comes out with the black wire from the dimmer is the "power out to bathroom and hall lights" wire from the wall box. Run a quick test though to ensure what each wire does.
Once that's established, connect the "power in" wire, the "power out to bathroom and hall lights" and the black wire from the dimmer all under one wire cap. Then connect the "power out to light fixture" wire to the red wire from the dimmer (the one that exists the same hole as the green ground wire) under a second wire cap. Connect the green wire from the dimmer to the bare wires in the wall box under a third wire cap. Be sure to put a fourth wire cap on the other red wire that exits the same hole of the dimmer as the black wire and should not connect to anything. (this wire will have voltage when the switch is "off").
Here's a few things, first, hopefully you took note of the wire that was on the black terminal of the old 3-way switch, if you didn't- it will be a bit more complicated.
The wire that was on the black terminal of your old 3-way switch will get attached to the black wire of the 6633-PL. The two wires that were on each of the other terminals (not including the ground) will each (individually) go the red wires of the 6633-PL.
Now if you didn't take note of the wires/terminal assignment of the old 3-way switch, you will need a volt meter to determine which wire is the hot wire, (basically supplies power to this circuit), once you've identified it, If it is found in the location where you plan to install the dimmer, then you would attach that wire to the black lead on the 6633-PL, the two traveler wires (the wires that "travel" between the 2 locations) will each (individually) go the red wires of the 6633-PL.