This expression comes from the Czechoslovakian saying, “Potrefená Husa nejvíc kejhá” which, literally translated, is: “A shot goose gabbles the most!” The English equivalent is, “A guilty conscience needs no accuser.”
In 1744, Matthew Bishop used the English expression in his book, “The Life and Adventures of Matthew Bishop of Deddington in Oxfordshire.” However, there are earlier versions of this phrase including the Scottish proverb recorded in 1721 that states: “A guilty Conscience self accuses. A Man that has done ill shews his Guilt” and in 1597 in Elizabethan anthology, Politeuphuia in the passage that read: “A Guilty conscience is a worme that bites and neuer ceaseth. A guiltie conscience is neuer without feare.”
It goes back farther than that and a version of the expression is found in Chaucer’s The Canterbury Tales written in 1390 in the story, “The Canon’s Yeoman’s Prologue.” In this tale, Chaucer writes: “For Catoun seith that he that gilty is Demeth alle thyng be spoke of him.”
However, all of those are a rewording of a passage from the Bible from the book of Genesis that speaks of the situation between Joseph and his brothers: “And the men were afraid, because they were brought into Joseph’s house; and they said, Because of the money that was returned in our sacks at the first time are we brought in; that he may seek occasion against us, and fall upon us, and take us for bondmen, and our asses.” (Genesis 43:18)
So if a shot goose gabbles the most, then someone who speaks as if he or she is guilty is certainly going to look like “something Katie shot at and hit!”