get all posts

Accalia Elementia

@lewismcmahon what do you want to do with them? is there some kind of processing you need to do on them?

do you want them from the client side or the server?

do you need them once or over time?

louisemcmahon

@accalia i need to do processing on them server side as part of the s3 uploads plugin

Accalia Elementia

@lewismcmahon ah. you're operating deeper than i thought you would be. the way i was going to suggest wont help you..... sorry.

best of luck, i regret taht you are operating beyond my sphere of knowledge.

<baris>

You can use the batch class

batch.processSortedSet('posts:pid', function(pids, next) {
    // do something with pids
    // call next to process next batch
    next();			
}, {}, function(err) {
	if (err) {
		return callback(err);
	}
	console.log('we are done');
});

Keep in mind this would take a while depending on how many posts you have. If you want to spread the operation over a period of time just call next with a timeout like this

setTimeout(next, 1000);

louisemcmahon

@accalia dont worry im operating beyond mine aswell

@baris thanks ill try that

sic2

Hi!

I have a similar question to @louisemcmahon

I can get the most recent posts from the API /api/recent/posts (API route code

I can see that there is no way of getting all posts (at least I do not see it). Do you know if there is a workaround or there is a plan to add this feature?

Thanks!

julian

From outside of NodeBB, you'll probably want to parse sitemap.xml (which is XML, ew), or more easily, iterate through /recent?page=X where X is an increasing number...

Pretty sure we flat out don't have a route that returns all topics, because if you hit it on a large forum, you're likely going to lock up the NodeBB process as it tries to compile that list of topics for you

Accalia Elementia

@sic2 said in get all posts:

Hi!

I have a similar question to @louisemcmahon

I can get the most recent posts from the API /api/recent/posts (API route code

I can see that there is no way of getting all posts (at least I do not see it). Do you know if there is a workaround or there is a plan to add this feature?

Thanks!

it sounds like you want the contents of a topic client side to process.

if you have a browser that is ES6 compliant (chrome confirmed, firefox is probably good, not sure about safari, IE is out) then this GIST will work fr downloading a single topic as CSV.

view the gist
https://gist.github.com/AccaliaDeElementia/a74ea51ae05433171f17bd4d397b287d

to select the topic to download you edit the final line of that script to include the topic ID you wish to save, replacing the example topic id (in this case 20937) with the one you desire.

downloadTopic(20937).then(result=>console.log(result))

then execute the script in your browser console while being on the target forum.

if it complains about window.socket not existing on line 84 of the script, simply refresh the forum with the console open and try again. I've not figured out why, but sometimes chrome can't find the websocket object used by the forums if the console is opened after the page loads.

sic2

@julian thanks, you answer was very helpful.

It looks like I can get a next pointer to the next page. This pointer, however, is not visible on all forums. I will investigate whether this is something that the admins have to configure or an issue with the nodebb version run.

Thanks again!

sumerkipl

@louisemcmahon ,
You can try the filters unreplied for getting all topics which don't have any reply till.
https://community.nodebb.org/recent?filter=unreplied