[nodebb-plugin-rss] RSS Feed plugin

Silvina

Hi @baris
Running via ./nodebb dev

info: [notifications.prune] Removing expired notifications from the database.
warn: Route requested but not found: /forum/index.php?topic=256.90
error: [[error:too-many-posts, 10]]
error: TypeError: Cannot read property 'topicData' of undefined
at setTimestampToFeedPublishedDate (/srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:198:23)
at /srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:167:9
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at /srv/forums-gameredge/src/user.js:190:12
at /srv/forums-gameredge/node_modules/async/lib/async.js:592:17
at done (/srv/forums-gameredge/node_modules/async/lib/async.js:135:19)
at /srv/forums-gameredge/node_modules/async/lib/async.js:32:16
at /srv/forums-gameredge/node_modules/async/lib/async.js:589:21
at /srv/forums-gameredge/src/database/redis/sets.js:44:4
[cluster] Child Process (9827) has exited (code: 1)
[cluster] Spinning up another process...

From Output log

[32minfo[39m: Time: Wed Oct 15 2014 00:28:06 GMT+0000 (UTC)
[32minfo[39m: Initializing NodeBB v0.5.1
[32minfo[39m: * using configuration stored in: /srv/forums-gameredge/config.json
[32minfo[39m: * using redis store at 127.0.0.1:6379
[32minfo[39m: * using themes stored in: /srv/forums-gameredge/node_modules
[32minfo[39m: [user.startJobs] Registering User Jobs
[32minfo[39m: Enabling 'trust proxy'
[cluster] Child Process (8950) listening for connections.
[33mwarn[39m: [plugins/nodebb-plugin-spoilers] staticDir is deprecated, use staticDirs instead
[32minfo[39m: [hotswap] Router with id plugins replaced successfully
[32minfo[39m: [plugins] All plugins reloaded and rerouted
[32minfo[39m: [meta/css] Minifying LESS/CSS
[32minfo[39m: [hotswap] Router with id auth replaced successfully
[32minfo[39m: NodeBB attempting to listen on: 0.0.0.0:4567
[32minfo[39m: NodeBB is now listening on: 0.0.0.0:4567
[cluster] Child Process (8950) listening for connections.
[32minfo[39m: [sounds] Sounds OK
[32minfo[39m: [meta/templates] Compiling templates
[32minfo[39m: [themes] Successfully compiled templates.
[32minfo[39m: [meta/css] Done.
[32minfo[39m: [meta/css] Committing stylesheet (acpCache) to disk
[32minfo[39m: [meta/css] Stylesheet (acpCache) committed to disk.
[32minfo[39m: [meta/css] Done.
[32minfo[39m: NodeBB Ready
[32minfo[39m: [meta/css] Committing stylesheet (cache) to disk
[32minfo[39m: [meta/css] Stylesheet (cache) committed to disk.
[32minfo[39m: [meta/js] Successfully minified.
[32minfo[39m: [meta/js] Retrieved Mapping.
[32minfo[39m: [meta/js] Compilation complete
[32minfo[39m: [meta/js] Committing minfile to disk
[32minfo[39m: [meta/js] Client-side minfile and mapping committed to disk.
TypeError: Cannot read property 'topicData' of undefined
at setTimestampToFeedPublishedDate (/srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:198:23)
at /srv/forums-gameredge/node_modules/nodebb-plugin-rss/index.js:167:9
at /srv/forums-gameredge/node_modules/async/lib/async.js:544:30
at async.waterfall.content (/srv/forums-gameredge/src/topics/create.js:117:13)
at fn (/srv/forums-gameredge/node_modules/async/lib/async.js:641:34)
at Object._onImmediate (/srv/forums-gameredge/node_modules/async/lib/async.js:557:34)
at processImmediate [as _immediateCallback] (timers.js:345:15)
[32minfo[39m: [app] Shutdown (SIGTERM/SIGINT) Initialised.
[32minfo[39m: [app] Database connection closed.
[32minfo[39m: [app] Web server closed to connections.
[32minfo[39m: [app] Shutdown complete.

Silvina

From Error.log

{"level":"error","message":"[meta/css] Syntax Error: .bg-variant is undefined - header.less on line 124","timestamp":"2014-09-24T15:16:06.561Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:23:05.307Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:41:00.746Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:42:00.466Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:43:00.691Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:44:00.683Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.rssitfor.me/getrss?name=%40AmazonAppADay Cannot read property 'feed' of null","timestamp":"2014-09-24T15:45:00.394Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox connect ETIMEDOUT","timestamp":"2014-10-08T01:37:03.232Z"}
{"level":"error","message":"[[error:no-privileges]]","timestamp":"2014-10-08T02:00:00.602Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:56:40.082Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:57:40.087Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:58:40.088Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T10:59:40.096Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.theverge.com/rss/frontpage getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.073Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://isthereanydeal.com/rss/deals/us/ getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.073Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed http://www.polygon.com/rss/index.xml getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.074Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://robertsspaceindustries.com/comm-link/rss getaddrinfo ESRCH","timestamp":"2014-10-13T11:00:40.074Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:01:20.105Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:01:40.102Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:02:40.108Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:03:40.112Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:04:40.118Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:05:40.122Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:06:40.127Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:07:40.132Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:08:40.136Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:09:40.141Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:10:40.146Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:11:40.150Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:12:40.151Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:13:40.158Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:14:40.162Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:15:40.165Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:16:40.170Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:17:40.169Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:18:40.173Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:19:40.182Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:20:40.186Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:21:40.187Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:22:40.186Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:23:40.195Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:24:40.199Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:25:40.202Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:26:40.207Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:27:40.209Z"}
{"level":"error","message":"[[nodebb-plugin-rss:error]] Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:28:31.073Z"}
{"level":"error","message":"[[error:too-many-posts, 10]]","timestamp":"2014-10-13T13:00:00.878Z"}

<baris>

Published 0.1.20 which fixes the crash.

error: You can only post once every 10 seconds - please wait before posting again

Looks like you regular user weren't able to post because of the post delay, I fixed that as well, let me know when you update.

{"level":"error","message":"error Error pulling feed https://rssfeeds.s3.amazonaws.com/goldbox getaddrinfo ESRCH","timestamp":"2014-10-13T11:28:31.073Z"}```

Not sure about those errors.

Silvina

Hi @baris
Installed the latest rss plugin and left it running over the night.

Some issues encountered. It is duplicating the posts.

<baris>

You can try with version 0.1.21, although if the publishedDate of the articles are changing then they will get reposted. The plugin checks the latest published date and posts articles that are newer. So if an articles publishedDate is getting updated to something more recent it gets reposted.

Tanner

I had to remove the plugin because of the duplicate post issue.

peter

Plugin looks good.

@baris is there a way to set each feed post to go to draft?

trevor

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

charles

@trevor said:

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

+1 for duplicate check

Silvina

@charles said:

@trevor said:

@Tanner said:

I had to remove the plugin because of the duplicate post issue.

Just thought of a good idea for core integration; Duplicate post checking.

+1 for duplicate check

+2 for duplicate checks

trevor

For a bonus, duplicate Title checks as well.

Chris

This working in the latest release? Seems to stop after one post for me.

Also seems to be parsing markdown or something improperly but I think that's another issue altogether.

Chris

Ah yes, duplicate post issue here as well. +1 Duplicate post.

Chris

$30 paypal bounty to the first person that can fix the duplicate posting issue

<baris>

@Pyrax sent a PR for the duplicate post issue, merged and published 0.2.1

https://github.com/barisusakli/nodebb-plugin-rss/pull/6

Tanner

Thanks, @Pyrax !

julian

Ooh, does @pyrax get $30 now? XD

Does NodeBB get a finders fee for facilitating this transaction? XD

Normando

Hello.

It is posible to retrieve a youtube channel videos and post every new video uploaded?

The sames as for blogs, but for youtube. I am looking at the google youtube api, but I can't figured how to do it.

Great plugin!

a_5mith

@Normando Hey buddy, sort of. Your mileage may vary with this one. It depends how Baris' plugin works.

Youtube has an RSS feed you can use, but I'm not sure quite how it would work with this plugin. Add this to your RSS feed input box, then where you want it to go, then test it I guess.

https://gdata.youtube.com/feeds/api/users/USER_ID/uploads

Replace USER_ID with the usernameor channel ID of the youtube channel. (You can enter this URL into your browser and it will return the feed. For example: This Link would show you all the videos from Tom Scott in RSS feed format.

EDIT: You can also use this link to fetch the images as well if you want slightly more information.

https://gdata.youtube.com/feeds/base/users/USER_ID/uploads

Chris

@julian Haha, he would if I could test it.

Doesn't seem to be working for me. I add the RSS feed and it seems to do an initial pull (I have it set for 1 minute) but doesn't continue looking (or at the very least, nothing new gets added).

Very strange how I delete the RSS feed, add it. It pulls all items and they show up fine. Duplicate appear to have stop showing up, but nothing new shows up either for me, it seems. Unfortuantely there's no verbose output when running nodebb dev so I can't see it trying to do anything either.

I'll keep messing around with it and see if it's something on my end I guess.