By Kathy Ehrich Dowd

05/14/2013 at 11:40 AM EDT

Bill Hader in character as Stefon

Dana Edelson/NBC

Arrivederci, Vinny Vedecci. So long, Stefon.

Bill Hader will be leaving Saturday Night Live after its season finale this Saturday – and he’s taking his arsenal of spot-on impersonations and wacky characters with him, reports The New York Times.

“It was a hard decision, but it has to happen at some point,” Hader told the Times. “It got to a point where I said, ‘Maybe it’s just time to go.’ “

And while he admits it was tough telling executive producer Lorne Michaels he planned to leave, it clearly could have been a lot worse.

“I’d heard stories that you get very emotional in those conversations,” he said. “And I’ve had other people tell me, ‘Oh, I cried.’ I didn’t, but I did think I was about to faint.”

Hader joined SNL in 2005 and was promoted to a full cast member a year later. He quickly earned a reputation as an impeccable impersonator, who could amusingly imitate everyone from James Carville to Charlie Sheen to Clint Eastwood.

The Oklahoma native told the Times his decision to leave was based in part by seeing friends and colleagues like Andy Samberg and Kristen Wiig depart last season to pursue new opportunities, as well as to move his family – including, daughters Hannah, 3, and Harper, 9 months – to Los Angeles both for his own career and for his wife’s, filmmaker Maggie Carey.

His departure comes on the heels of fellow castmate and SNL head writer Seth Meyers’ news that he will replace Jimmy Fallon as the host of Late Night next year.

Hader says the audience should not expect a sappy sendoff on Saturday, like last season when Mick Jagger serenaded Kristen Wiig on her final show.

“My joke was, we’ll start singing ‘Ruby Tuesday’ and just say bye to Kristen again,” Hader riffed on his SNL goodbye to the Times. “I can just do a little wave at goodnights, and that signifies it.”