I'm neither an astronomer nor an astrophotographer and I do this. The sun always rotates in the same direction, it's not exactly rocket science.
But if I were to ever visit the southern hemisphere, then I'm practically guaranteed to pick the worst possible sitting spot at least once.