You can walk directly to Granville Island from Samesun Vancouver. To get there you'll want to turn left out of the hostel and walk across Granville Bridge. Once you're almost across the bridge you'll see some stairs on your left which will take you directly to the island.
We recommend not using Google Maps as it'll say you cannot walk across the bridge, which you can.