Former foreign minister Alexander Downer says Prime Minister Anthony Albanese set the Royal Commission up in a way that made sure he and his ministers would never have to front it.