Yes, but you can listen to a song on the radio whilst your doing the washing and think, "that's good." Then when you sit down with your headphones and listen to the same song carefully you may get a fuller appreciation. Even more so if you read and understand the lyrics. By careful appreciation you may even think a bad song is actually good or vice versa. Of course there's no need to do this if you don't want to.
I understand what Ellington meant; just enjoy the songs if you want to but I'm sure that he would be flattered if you displayed a deeper understanding of his work. Mingus certainly would.