Lost star Josh Holloway says he was once offered the part of Gambit in X-Men Origins but was later let go because they wanted someone “10 years younger”