The castle used for filming the medieval scenes in Doomsday was Blackness Castle near Edinburgh. Blackness was not the first castle of choice for filming the movie as director Neil Marshall notes in the DVD/Blu-ray commentary. However it proved to be a better option due to it's rocky interior and the way it suited itself to form the ampitheatre for the combat scenes.

The photo below is Courtesy of Creative Scotland – Location Service.