Well to be honest He didn't. "Australia" was unknown then. The orders James Cook had were to sail west from Tahiti (after observing the transit of Venus) until he reached Terra Australis Incognita - "The Unknown Southern Land" - and to map it, record observations of it and, if he felt it was worthwhile, to claim the land, as long as it was unoccupied