Prague Castle, like most defensive works, is on a hill. If you have the stamina, you can just head across the Charles Bridge and walk up through the small alleyways to gain access to the castle and St Vitus Cathedral. There is, an easier, alternative.
Tram number 22 climbs the hill to the right hand side of the castle. There is even a stop dedicated to visiting the castle – Prazsky Hrad. You will find a large number of people leaving the tram at this stop.
However, if you prefer a more scenic, downhill stroll past such delights as the Junior Firefighter training camp, approaching the castle from above, take the tram two more stops to Pohorelec. From the tram stop, cross the road, turn left and head downhill, keeping left towards the castle.