David Letterman’s decision to retire next year has predictably set off a storm of speculation about successors to his “Late Show” throne.

Stephen Colbert (pictured) has had quite a reversal of fortune in the past seven days. Last week he was forced to defend his Comedy Central series from accusations of racial insensitivity stemming from a joke involving a reference to Asian-Americans.

But in the past 48 hours Colbert has been tabbed by showbiz insiders and the media as a top contender to move to CBS’ 11:35 p.m. slot.

CBS reps have steadfastly declined comment on possible Letterman successors. A source close to the situation said the network intends to move quickly to set its post-Letterman lineup, a process that is expected to take a few weeks but not months.

Colbert’s current contract with Comedy Central runs through 2014. He has hosted “The Colbert Report” in the persona of faux conservative commentator since 2005 after getting his start on Comedy Central’s “The Daily Show with Jon Stewart.”

Moving to CBS would give him the opportunity to move out from behind the mask of the fulminating Bill O’Reilly-esque character he’s played for nearly a decade. That could be a risk for CBS as viewers don’t have much of a sense of Colbert’s actual on-canera personality. However, he excels in the guest interview segments on “Colbert Report,” even when done in character.

Letterman, who turns 67 next week, surprised viewers on Thursday when he announced he would exit late night TV sometime in 2015.

“We don’t have the timetable for this precisely down,” he told the Ed Sullivan Theater audience. “I think it will be at least a year or so, but sometime in the not too distant future.”

Sources close to the situation tell Variety the decision was ultimately Letterman’s despite the fact that he is now going up against much younger competitors on NBC and ABC.

He joked about his impending departure (and old age) on Friday’s telecast, saying he knew it was time to hang it up when his makeup artist who turned to him recently and said “‘Dave, there’s really nothing more I can do.'”

The longer the speculation continues, the more difficult the situation becomes for Letterman’s latenight companion, Craig Ferguson.

Speculation is mounting that CBS does not see “The Late Late Show” host as a prime contender to replace Letterman. Ferguson, 51, is in the midst of contract talks with CBS as his current deal ends later this year.

Ferguson does, however, have a clause in his contract that calls for him to either succeed Letterman or command a form of payout from CBS. A report in the New York Daily News Friday said his payout could be as much as $8 million-$12 million.

Chelsea Handler, the host of E!’s “Chelsea Lately,” has emerged as a possible replacement for Ferguson, as the host is in the midst of her own contract wrangle with the NBCUniversal cabler.

Cynthia Littleton contributed to this report