How to increase Number of URLs in sitemap.xml file?

[ankitpr89]

Currently I see there are only 63 latest urls that appear in /sitemap.xml file
How can i increase this number?

[julian]

You know, this is a good question.

NodeBB used to display ALL topics on sitemap.xml, except we were finding that it would take many seconds to load, effectively locking the Node process (if you only had 1 process).

We restricted it to load recent topics only, although I can imagine that this isn't good for SEO.

Does anybody know of a standard for paginated sitemaps?

[baris]

https://github.com/NodeBB/NodeBB/blob/master/src/sitemap.js#L52 you can increase the value 49.

[hek]

Does it take several seconds to build it? Maybe caching the sitemap in memory would help? Or simply write all new topics to a static file and rebuild it at startup.

[julian]

The sitemap is cached, although the process to build it takes a couple seconds (if you have many thousand topics).

Might not be too bad if we optimized it a bit...

[psychobunny]

Optimize sitemap? Inb4: https://github.com/NodeBB/NodeBB/commit/006322f3863f5dabbc9b4051cac6b0f466e37916 @baris

0 barisusakli committed to NodeBB/NodeBB

much faster sitemap

no need to load all the data, when we only use slug and lastposttime

[ankitpr89]

Guys thank you so much for your reply.
Since I mostly look into Online Marketing Stuffs, that's why asked this question.
From our team @vatsal is taking care of tech., he would be happy to give feedback.

Yes, according to me maintaining a list of sitemap record is quite important for search engines.

Have a look at sitemap samples from here: http://www.amazon.com/robots.txt ; at the end of file sitemap location is given.
For large websites and forums .xml.gz format can be followed.

[julian]

There's some info here about sitemap indexes, which we might put into action: http://www.sitemaps.org/protocol.html#index

[hek]

Yep, this is quite important.

Did some test googling... and most of our forum thread content has fallen out of the google index. There are a wealth of information in those threads with lots of technical mumbo jumbo keywords which would drive more sensor-nerds to us if they were searchable.

OT: NodeBB gets appreciation from the members: http://forum.mysensors.org/topic/472/push-notifications
Thanks for all the effort you put into this!

[ankitpr89]

@julian @psychobunny hope to see some important update on this.
As this is quite important for drawing traffic from search engines.

[ankitpr89]

@hek my temporary work around is to generate sitemap manually and put on server at different url address; then submit it to search engines.
This can be done manually.

Used this to generate sitemap: http://www.web-site-map.com/xml_sitemap.php

[julian]

Good workaround @ankitpr89. We'll try to change up the sitemap, let me issue it now

gh#2254

julianlam created this issue in NodeBB/NodeBB

closed Sitemap refactoring #2254

[ankitpr89]

@julian Thank you so much Julian!

[ExperienceOz]

Hi guys, sorry to bump such an old thread but couldn't find anything more relevant.

Has there been any change in the way NodeBB generates sitemaps since back when this was originally posted? We are not seeing a full list of URLs in our sitemap either (using Lavender theme if it makes any difference) and a lot of should-be-popular threads aren't being indexed as a result.

Thanks!

[joes]

@julian @psychobunny Did this get fixed? Does the current sitemap file work with Web master tools etc?

[Vamsi Krishna]

@julian @psychobunny hey guys, any updates on this? We have been looking for something similar too!

[julian]

Yes, the current sitemap should list all publicly available topics/categories.

[Vamsi Krishna]

hey @julian thanks for your quick reply. I checked the sitemap for our forum, but it shown only the categories & topics headers. Is this how it's meant to be shown or am I missing something?

[PitaJ]

@Vamsi-Krishna what you see there are links to other sitemap files. They are essentially imported when the main sitemap is parsed by Google et al.