Actually it was the Roman legions who built the wall, neither the Scottish or English borderers would have wanted it!
It took six years to build most of Hadrian's Wall with the work coming from three Roman legions – the Legio II Augusta, Legio VI Victrix, and Legio XX Valeria Victrix, totalling 15,000 soldiers, plus some members of the Roman fleet.
From Wiki;