This morning I checked and the emoji image had gone, but the emoji code was still being applied whenever certain characters were used, including "£".
image.png
image.png
I figured if it's doing a "live" replacement of those things then it must be something to do with the "Replace..." options in the plugin settings:
image.png
So I disabled them both and did a restart / rebuild. The issue went away.
I re-enabled them 1 by 1 to try and figure out which was causing the problem (rebuilding and restarting each time) and now, they're both re-enabled and the problem has not come back.
So I'm a bit mystified. Evidently it was a problem with one of those options, but setting them again seems to have somehow fixed it.