'Downton Abbey fans still tell me off for jilting Edith'