(Richmond, Va.) — Gov. Ralph Northam spent much of the day Sunday in seclusion at his family home on the Eastern Shore as aides delivered conflicting advice about whether he should resign or work to clear his name. He held an impromptu meeting with staff just before the start of the Super Bowl to discuss his fate and whether he should continue in office.

Some of his strongest allies – former Gov. Terry McAuliffe, and U.S. Senators Tim Kaine and Mark Warner – joined a growing chorus of Democrats and Republicans calling for his resignation, saying that the discovery of a racist photo on his 1984 medical school yearbook page make it impossible for Northam to lead the state effectively.

The chairwoman of the Congressional Black Caucus says that Virginia Gov. Ralph Northam “still does not understand the seriousness of his actions.” There have been calls for Northam’s resignation over a medical school yearbook photo, which shows one person in blackface and another in a Klan robe.

Northman first said that’s not the man he is today, then said he was not in the photo, but says he did dress in blackface once while imitating Michael Jackson.