If you can get to a Service Manual there are detailed instruction for troubleshooting / testing the Starter Relay.
Assuming the Run/Stop Switch is in the RUN position, the key is ON and you have lights, and bike is in neutral when you press the starter button you should hear the relay CLICK. This also assumes your START button is good.
According to the Service Manual for the 2003 - 2007 S/R/C models there are 4 terminals on the relay.
When you apply 12V across them the relay should CLICK and there should be continuity between the other two terminals.
Hooking up 12V and hearing a CLICK isn't enough, you should check the terminals as well.
This procedure is outlined on page 18-14 of the 2003 - 2007 VTX 1300 S/R/C Service Manual
Note- Of course if you do not hear a click and the relay checks good, test for 12 Volts on the appropriate terminals on the connector to the relay - make sure you press the Start Button and all conditions for starting are good - Neutral / Run Switch ON.
I hope your trouble is the relay or starter switch. I've been researching ( studying ) the starter system, and man it's a b$#*!ch to work on that starter, especially if you have to get to the torque limiter or starter clutch! :shock: