1.11.x upgrade takes a long time and maybe it's horked?

Dave Jensen

The step "Changing dbsearch mongodb search schema to use _id..." is taking a long time to complete. So long that my ssh session disconnected and unfortunately, I wasn't running tmux.

I reconnected, started tmux and ran upgrade again. The problem is that I have no idea if it's okay. Soon after running I got this error:

4. Updating NodeBB data store schema...
Parsing upgrade scripts...
2018-11-30T14:43:57.305Z [24078] - warn: [upgrade/appendPluginScripts] Unable to read plugin.json for plugin `nodebb-plugin-import-smf-1`. Skipping.
OK | 1 script(s) found, 67 skipped
  → [2019/1/26] Changing dbsearch mongodb search schema to use _id...
2018-11-30T14:43:57.404Z [24078] - error: can't find index with key: { id: 1 }
MongoError: can't find index with key: { id: 1 }
    at /var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:581:63
    at authenticateStragglers (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:504:16)
    at Connection.messageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:540:5)
    at emitMessageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:310:10)
    at Socket.<anonymous> (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:453:17)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10) {"nIndexesWas":2,"ok":0,"errmsg":"can't find index with key: { id: 1 }","code":27,"codeName":"IndexNotFound","name":"MongoError"}
2018-11-30T14:43:57.409Z [24078] - error: can't find index with key: { id: 1 }
MongoError: can't find index with key: { id: 1 }
    at /var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:581:63
    at authenticateStragglers (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:504:16)
    at Connection.messageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:540:5)
    at emitMessageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:310:10)
    at Socket.<anonymous> (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:453:17)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10) {"nIndexesWas":2,"ok":0,"errmsg":"can't find index with key: { id: 1 }","code":27,"codeName":"IndexNotFound","name":"MongoError"}

Now, I have no idea what's happening. I think it's doing something but since there's no progress updates maybe it's spinning for no reason and I've been down for 8+ hours for no reason. Or maybe it is doing something. The error makes me worried that I have to revert back to my snapshot and start from scratch.

I kinda think it would be nice to outsmart the user a little bit an run this upgrade job in the background by default with instructions on how to check the progress in the logs or maybe just background the job and foreground log tailing or something.

Dave Jensen

It's actually super frustrating. Downtime this long after a big change like this could seriously kill a community. People get frustrated and stop coming, Google crawlers think the site is unreliable and stop sending traffic. I realized that updating an index is a long job but ... ugh. Now I'm doing it again and maybe it'll run for another 8 hours? Or more?

Dave Jensen

Is there any way to bypass the the index upgrade and just drop the index instead?

I don't mind if search isn't working for a while. I'd rather have no search, the forums running, and re-index later.

<baris>

It is probably the reindex of the search plugin, you can drop the search collections searchpost and searchtopic and remove these lines https://github.com/barisusakli/nodebb-plugin-dbsearch/blob/master/upgrades/dbsearch_change_mongodb_schema.js#L47-L51 and run the upgrade again, that should complete much quickly.

I might remove those 2 calls since reindex can be run from mongodb cli as well.

Dave Jensen

@baris said in 1.11.x upgrade takes a long time and maybe it's horked?:

search collections searchpost and searchto

Thank you, I'll try that now.

<baris>

Also you should be able to check if something is running in mongodb with these commands

>use <db_name>;
>db.currentOp();

Dave Jensen

This post is deleted!

Dave Jensen

@baris db.currentOp() worked

Those lines I'm supposed to remove were curly braces. That doesn't seem quite right.

Dave Jensen

I see the code lines were for v3.0.0

Dave Jensen

@baris

New error, hope it's okay:

4. Updating NodeBB data store schema...
Parsing upgrade scripts...
2018-11-30T15:21:24.425Z [24448] - warn: [upgrade/appendPluginScripts] Unable to read plugin.json for plugin `nodebb-plugin-import-smf-1`. Skipping.
OK | 1 script(s) found, 67 skipped
  → [2019/1/26] Changing dbsearch mongodb search schema to use _id...
2018-11-30T15:21:24.509Z [24448] - error: ns not found
MongoError: ns not found
    at /var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:581:63
    at authenticateStragglers (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:504:16)
    at Connection.messageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:540:5)
    at emitMessageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:310:10)
    at Socket.<anonymous> (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:453:17)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10) {"ok":0,"errmsg":"ns not found","code":26,"codeName":"NamespaceNotFound","name":"MongoError"}
2018-11-30T15:21:24.512Z [24448] - error: ns not found
MongoError: ns not found
    at /var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:581:63
    at authenticateStragglers (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:504:16)
    at Connection.messageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:540:5)
    at emitMessageHandler (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:310:10)
    at Socket.<anonymous> (/var/srv/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:453:17)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10) {"ok":0,"errmsg":"ns not found","code":26,"codeName":"NamespaceNotFound","name":"MongoError"}

Thanks for the help BTW.