Yes, the mic is open all the time. Google will then suggest adverts based on your conversation.
It's one of those "opt out" things but you can't real remove the entire operating system. So they have you.
There is video on YouTube where a guy randomly talks about digs (he doesn't own any pets) he then opens Google and it shows dogs toys on every website.