I can't believe I forgot! I just rewatched that episode a few days ago. How embarrassing. I thought I remembered Dean saying it, but I couldn't find it in any SPN quotes when I ran a quick search. I guess it just didn't show up as a result because the wording was slightly different.

Thanks so much for the help! It was really bugging me.