When Is The Right Time To Pull The A.C. Out?

I go through this every year.  Fall is very obviously here yet I hesitate to take the air conditioning out of the windows just in case we get another warm day.  So when is the perfect time to take the air out?Do you wait until the snow is on the ground?  Is there a special date or holiday that you wait for?

The other question is, how long do you wait to turn the heat on?