The stock xfce4-panel Clock *calendar for some weird reason has the week number hard coded. So you can't hide it. I can't think of a single time in my life I thought to myself, "gee, I wonder what week out of 52 we are currently in". This is such a strange choice.