How do you remove rubber headlight bulb cover?

[detroiter]

My drivers side light went out and so tonight I picked up some nice Phillips xenon style lights. I went to install it in the drivers side and boy was it a tight fit! Worse off once pulling off the plug, it had a ton of white lithium grease on it. Then I reached the rubber sleeve. I've been able to slide off the circular housing part but I can't get it to come all the way off. I don't want to pull too hard because I don't want to rip it or anything. Can someone give me some hints here?