You said in your OP he "suspects" emphysema. One could agree to that (concave curve, high VC, flow metrics low normal, especially FEF25-75) but its a rather weak suspicion (low normal is still normal). Did the doctor know that you are able to swim, run etc?
A follow-up in 3 months is certainly a good idea, whether at the NHS or one of the doctors Sheryl and Noodle recommended.
Also ask for a DLCO test.
And, yes, I would taper off the inhaler- especially as you said it had no effect at the spirometry.