[nodebb-plugin-rss] RSS Feed plugin

Silvina

[www-data@vultr forums-gameredge]$ ./nodebb log
at /srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:167:9
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at async.waterfall.content (/srv/forums-gameredge/src/topics/create.js:117:1 3)
at fn (/srv/forums-gameredge/node_modules/async/lib/async.js:641:34)
at Object._onImmediate (/srv/forums-gameredge/node_modules/async/lib/async.j s:557:34)
at processImmediate [as _immediateCallback] (timers.js:345:15)
info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
info: [app] Database connection closed.
info: [app] Web server closed to connections.
info: [app] Shutdown complete.

After upgrading to nodebb 0.5.1, i noticed that my nodebb instance been dying at random times.
Is this plugin causing the issue?

<baris>

Can you post the full log, should be at logs/output.log.

Silvina

Hi @baris
Running via ./nodebb dev

info: [notifications.prune] Removing expired notifications from the database.
warn: Route requested but not found: /forum/index.php?topic=256.90
error: [[error:too-many-posts, 10]]
error: TypeError: Cannot read property 'topicData' of undefined
at setTimestampToFeedPublishedDate (/srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:198:23)
at /srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:167:9
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at /srv/forums-gameredge/src/user.js:190:12
at /srv/forums-gameredge/node_modules/async/lib/async.js:592:17
at done (/srv/forums-gameredge/node_modules/async/lib/async.js:135:19)
at /srv/forums-gameredge/node_modules/async/lib/async.js:32:16
at /srv/forums-gameredge/node_modules/async/lib/async.js:589:21
at /srv/forums-gameredge/src/database/redis/sets.js:44:4
[cluster] Child Process (9827) has exited (code: 1)
[cluster] Spinning up another process...

From Output log

[32minfo[39m: Time: Wed Oct 15 2014 00:28:06 GMT+0000 (UTC)
[32minfo[39m: Initializing NodeBB v0.5.1
[32minfo[39m: * using configuration stored in: /srv/forums-gameredge/config.json
[32minfo[39m: * using redis store at 127.0.0.1:6379
[32minfo[39m: * using themes stored in: /srv/forums-gameredge/node_modules
[32minfo[39m: [user.startJobs] Registering User Jobs
[32minfo[39m: Enabling 'trust proxy'
[cluster] Child Process (8950) listening for connections.
[33mwarn[39m: [plugins/nodebb-plugin-spoilers] staticDir is deprecated, use staticDirs instead
[32minfo[39m: [hotswap] Router with id plugins replaced successfully
[32minfo[39m: [plugins] All plugins reloaded and rerouted
[32minfo[39m: [meta/css] Minifying LESS/CSS
[32minfo[39m: [hotswap] Router with id auth replaced successfully
[32minfo[39m: NodeBB attempting to listen on: 0.0.0.0:4567
[32minfo[39m: NodeBB is now listening on: 0.0.0.0:4567
[cluster] Child Process (8950) listening for connections.
[32minfo[39m: [sounds] Sounds OK
[32minfo[39m: [meta/templates] Compiling templates
[32minfo[39m: [themes] Successfully compiled templates.
[32minfo[39m: [meta/css] Done.
[32minfo[39m: [meta/css] Committing stylesheet (acpCache) to disk
[32minfo[39m: [meta/css] Stylesheet (acpCache) committed to disk.
[32minfo[39m: [meta/css] Done.
[32minfo[39m: NodeBB Ready
[32minfo[39m: [meta/css] Committing stylesheet (cache) to disk
[32minfo[39m: [meta/css] Stylesheet (cache) committed to disk.
[32minfo[39m: [meta/js] Successfully minified.
[32minfo[39m: [meta/js] Retrieved Mapping.
[32minfo[39m: [meta/js] Compilation complete
[32minfo[39m: [meta/js] Committing minfile to disk
[32minfo[39m: [meta/js] Client-side minfile and mapping committed to disk.
TypeError: Cannot read property 'topicData' of undefined
at setTimestampToFeedPublishedDate (/srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:198:23)
at /srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:167:9
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at async.waterfall.content (/srv/forums-gameredge/src/topics/create.js:117:13)
at fn (/srv/forums-gameredge/node_modules/async/lib/async.js:641:34)
at Object._onImmediate (/srv/forums-gameredge/node_modules/async/lib/async.js:557:34)
at processImmediate [as _immediateCallback] (timers.js:345:15)
[32minfo[39m: [app] Shutdown (SIGTERM/SIGINT) Initialised.
[32minfo[39m: [app] Database connection closed.
[32minfo[39m: [app] Web server closed to connections.
[32minfo[39m: [app] Shutdown complete.

Silvina

From Error.log

{"level":"error","message":"[meta/css] Syntax Error: .bg-variant is undefined - header.less on line 124","timestamp":"2014-09-24T15:16:06.561Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:23:05.307Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:41:00.746Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:42:00.466Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:43:00.691Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:44:00.683Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:45:00.394Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox connect ETIMEDOUT","timestamp":"2014-10-08T01:37:03.232Z"}
{"level":"error","message":"[[error:no-privileges]]","timestamp":"2014-10-08T02:00:00.602Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:56:40.082Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:57:40.087Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:58:40.088Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:59:40.096Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.theverge.com/rss/frontpage getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.073Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://isthereanydeal.com/rss/deals/us/ getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.073Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.polygon.com/rss/index.xml getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.074Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://robertsspaceindustries.com/comm-link/rss getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.074Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:01:20.105Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:01:40.102Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:02:40.108Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:03:40.112Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:04:40.118Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:05:40.122Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:06:40.127Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:07:40.132Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:08:40.136Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:09:40.141Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:10:40.146Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:11:40.150Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:12:40.151Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:13:40.158Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:14:40.162Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:15:40.165Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:16:40.170Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:17:40.169Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:18:40.173Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:19:40.182Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:20:40.186Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:21:40.187Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:22:40.186Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:23:40.195Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:24:40.199Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:25:40.202Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:26:40.207Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:27:40.209Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:28:31.073Z"}
{"level":"error","message":"[[error:too-many-posts, 10]]","timestamp":"2014-10-13T13:00:00.878Z"}

<baris>

Published 0.1.20 which fixes the crash.

error: You can only post once every 10 seconds - please wait before posting again

Looks like you regular user weren't able to post because of the post delay, I fixed that as well, let me know when you update.

{"level":"error","message":"error Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:28:31.073Z"}```

Not sure about those errors.

Silvina

Hi @baris
Installed the latest rss plugin and left it running over the night.

Some issues encountered. It is duplicating the posts.

<baris>

You can try with version 0.1.21, although if the publishedDate of the articles are changing then they will get reposted. The plugin checks the latest published date and posts articles that are newer. So if an articles publishedDate is getting updated to something more recent it gets reposted.

Tanner

I had to remove the plugin because of the duplicate post issue.

peter

Plugin looks good.

@baris is there a way to set each feed post to go to draft?

trevor

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

charles

@trevor said:

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

+1 for duplicate check

Silvina

@charles said:

@trevor said:

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

+1 for duplicate check

+2 for duplicate checks

trevor

For a bonus, duplicate Title checks as well.

Chris

This working in the latest release? Seems to stop after one post for me.

Also seems to be parsing markdown or something improperly but I think that's another issue altogether.

Chris

Ah yes, duplicate post issue here as well. +1 Duplicate post.

Chris

$30 paypal bounty to the first person that can fix the duplicate posting issue

<baris>

@Pyrax sent a PR for the duplicate post issue, merged and published 0.2.1

https://github.com/barisusakli/nodebb-plugin-rss/pull/6

Tanner

Thanks, @Pyrax !

julian

Ooh, does @pyrax get $30 now? XD

Does NodeBB get a finders fee for facilitating this transaction? XD

Normando

Hello.

It is posible to retrieve a youtube channel videos and post every new video uploaded?

The sames as for blogs, but for youtube. I am looking at the google youtube api, but I can't figured how to do it.

Great plugin!