This phrase dates back to colonial days and the upper class. When a suitor would arrive at his sweetheart’s home with a small courting mirror when he intended to propose marriage to her.
He would place the mirror on the table, facing upwards and suggest a union between the two of them. If her answer to his proposal was yes, she would look into the mirror and smile at her beau’s reflection.
If the answer was no, she would discretely turn the mirror face down hence the term “turned down.”
Historically Speaking 