With both being async it's hard to follow what would happen if both run at the same
time, so just prevent it and serialize start/stopping instead.