New York City is the largest city in New York State. Generally, when travelers say they are from or going toward "New York", they are referring to this city, which is actually made up of 5 boroughs and 3 islands: Manhattan, Staten Island, and Long Island (which Brooklyn and Queens are a part of). The Bronx, the fifth borough, is connected with the mainland of New York State.
New York City is a major hub for trains: freight trains, Amtrak, Metro-North, MTA, LIRR, NJ Transit, and PATH trains all pass in and out of this city. Although riding trains for free (train-hopping) is illegal, it is one of the most viable ways to get out of the city for free. In most passenger trains (NJ Transit, Amtrak, etc.), the only way to ride for free is to hide in a bathroom or display a "Seat Check" ticket found on the ground or from a previous ride. Generally speaking, if you are discovered to be riding without paying the proper fare, you will be asked to pay it. If you can not pay it, then you will be asked to exit the train at the next station. As long as you main friendly and cooperative, there should not be a problem with legal issues, arrest, or fines.
Hopping freight trains is a common way to ride long distances for free. It is extremely practical to do if you have phone or internet access, because there are numbers and/or websites that will give you the direction that each freight is headed and its estimated time of arrival, based on that freight's identification number.
Don't get stuck in Albany.