But he still has sympathy for Mark Richt, the man Dooley hired 10 years ago to be Georgia's head coach. As Richt comes under increasing fire from critics and fans, Dooley defended him.
"Any coach is going to go through tough times, I don't care who you are," Dooley said Monday. "When he came here, he won a championship his second year. (Richt has been) consistently winning 10 football games. So he's going through a tough time. That's what happens. … A coach has to address it, analyze it, make some decisions and go forward. And I'm sure he'll do that."
Richt has a career record of 91-31 at Georgia, and has won two SEC championships. Percentage-wise, he is the winningest coach in school history.
But the Bulldogs are 1-4 this year, a year after going 8-5.
"Four or five years ago, Joe Paterno had three losing seasons in four years at a place called Penn State," Dooley said. "Bear Bryant went through his problems late 60, very early 70s. Bobby Bowden ended up, even though what an incredible coach he was, his last few years at Florida State weren't very pleasant."
Bowden was eventually forced out last year, while Paterno and Bryant survived to experience better times. Dooley seems to think that if Georgia sticks with Richt, the same will happen.
Dooley himself went through some down years at Georgia. After winning the SEC his third and fifth seasons, the Bulldogs slipped to five-win seasons in 1969 and 1970; after winning another SEC title in 1976, the Bulldogs won five games the next year and six in 1979.
Then they won the national title in 1980.
"You need to be able to realize that while you aspire for the best, you've gotta realize that that's not always possible," Dooley said. "And you have to hang tough through the tough times. What's the old saying, Tough times don't last, but tough people do."
Dooley stepped down as Georgia's athletics director in 2004 under controversial circumstances. School president Michael Adams wanted a change, even though Dooley wasn't ready to retire.
While Dooley no longer has an official role at Georgia, he drops in occasionally at the athletic complex that bears his name. He dropped by before the season to see Richt, but believes the best thing is to stay away, as he will when his son Derek coaches Tennessee in Athens.
The man who hired one coach and sired the other will be watching Saturday's game safely from the den of his Athens home.
"There is no question I have to pull for my son. It is family first," Dooley said on Monday. "It is a great dilemma of course. But nevertheless to not pull for Sanford Stadium in Georgia, there is no question I couldn't do that. So I am staying away."
Derek Dooley grew up in Athens, and played at Clarke Central High School.
"People forget that I left the state of Georgia when I was 18," Derek Dooley said on Monday "Since then, I've worn a lot of different colors."
His father says Ray Goff recommended the Bulldogs offer Derek a spot. But Vince, who called Derek a "possession receiver" and "a good football player, not a great football player," felt it was best for his son to go to Virginia, where he walked on.
"If he had been a Lynn Swann or something I wouldn't have hesitated," Dooley said, laughing.