I'm not really sure what you're trying to do here - if you want to copy a repository to a local folder you should use git clone and not git checkout - the latter is used for switching branches and generally moving to a different version of a repository.
However, got should not be used to install a plugin. NPM is the utility you're looking for. It supports two methods of authentication for GitHub private repositories - OAuth and SSH.
For OAuth you'll need to generate a personal access token (settings>developer settings>personal access tokens) with permission to read private repositories (just repository permissions if I remember correctly), and then go to the main NodeBB directory and use npm install git+https://<github_token>:x-oauth-basic@github.com/NodeBB/nodebb-plugin-xxx.git
If you prefer to use SSH instead, follow the instructions here to generate the keys and add to your account, and then run in the main NodeBB folder npm install git+ssh://git@github.com/NodeBB/nodebb-plugin-xxx.git
