When I tried this the first few times it did not work. This seems to be because I was doing it too fast. The following steps seem to work consistently (some may be unnecessary):

Hit instance lockout on your main running a specific dungeon.(I used the Motherlode)

Log out, and close your client.

Log in on another character on another realm and enter the same dungeon. You can leave immediately you enter. Once outside reset the instance.

Log out, and close your client again.

Wait at least two minutes.

Log-in on your main and re-enter the dungeon you've been locked out of. You should be able to enter and your instance lockout total should have been reset to 0 (ie you can run the dungeon ten times again).