He started in Atlanta, but was moved to The Rock.
A bit about his time in Atlanta:
"Before arriving at Alcatraz, Capone had been a master at manipulating his environment at the Federal Penitentiary in Atlanta. Despite strict convictions from the courts, Capone was always able to persuade his keepers into procuring his every whim, and often dictated his own privileges. It was said that he had convinced many guards to work for him, and his cell boasted expensive furnishings which included personal bedding along with many other amenities not extended to other inmates serving lesser crimes. His cell was carpeted, and also had a radio around which many of the guards would sit with Al conversing and listening to their favorite radio serials. His friends and family maintained residence in a nearby hotel, and each day he was flooded with visitors."
http://www.alcatrazhistory.com/cap1.htm