To just the author and mods, unfortunately we cannot support this yet, as we don't have a "post author" pseudo-group. That is a good idea, though.
We did accomplish this via nodebb-plugin-support-forum, you could try this and see if it still works?
@arasbm ACP > Manage Categories. Click Edit on the right of a category and click on permissions, search for the name of the user, and an options box will come up to the right of that user, click that, very bottom will have the option of "Moderator" click that. Do that for each mod in each category. It's not the fastest method, but does allow for different moderators for different categories.