Yeah, gets very confusing once you factor in page #s and such. I'd prefer the "posts per page" and "topics per page" options remain admin-only, while allowing users to toggle between pagination and infinite scrolling.
A "hybrid" pagination+infinite scrolling would be nice as well, so you can get the best of both worlds.