OK, got it, its an insult
"Have you ever heard people talk insultingly about uneducated, uncultured country folks? They probably referred to them as yahoos.
If someone calls you a yahoo, it's definitely an insult. Yahoos are thought to have very little education and culture. People think of yahoos as folks who were raised in a barn — maybe literally. A yahoo can also be called a bumpkin, hayseed, hick, rube, yokel, or hillbilly. You can't really be a yahoo if you're from a big city. So if you want to insult someone from New York, you'll need to find a different word."