Disqus importer

sanatisharif

Hi , We have recently imigrated to nodeBB. Our previous comment system was Disqus. So now despite the fact that we are really enjoying this new experience we want to transmit our users' comments from our old Disqus to our new nodeBB . Here is the export format that Disqus offers:
https://help.disqus.com/customer/portal/articles/472149-comments-export

An other problem is that we want to submit these past comments with the attached usernames(not something like an admin username) and now some of these usernames do not exist!

We were wondering if you could build a pluging for this Disqus transfer issue.
Thanks alot

bentael

@sanatisharif

I don't see why not. I'd like to get a sample dump if you have one - as for the usernames that don't have an account, we can use the post.handle (link), which the import plugin already supports, as _post._guest

sanatisharif

@bentael output of Disqus->Admin->setup->Export comments is:

https://media.disqus.com/uploads/exports/2780/230/sanatisharif-2016-01-09T15:15:44.220647-all.xml.gz

This file have utf8 issue, Like nodebb-Vbulltine-importer

Best Regards

bentael

Ok, looking at the schema, it seems pretty straight forward, we can deal with the anonymous users, and for the non-anonymous, we can create an account for them.

However, there is 1 problem, each disqus thread doesn't really have a content, it just links to another page somewhere.

for example

<thread dsq:id="2122903199">
	<id />
	<forum>sanatisharif</forum>
	<category dsq:id="2822236" />


         // there is ONLY a link, no content, because that content is on that page.
	<link>http://sanatisharif.ir/Sanati-Sharif-Video/6/4/164</link>   

	<title>تماشا دبیرستان دانشگاه صنعتی شریف</title>
	<message />
		<createdAt>2014-01-14T22:58:41Z</createdAt>
	<author>
		<email>[email protected]</email>
		<name>دبیرستان غیردولتی پسرانه دانشگ</name>
		<isAnonymous>false</isAnonymous>
		<username>sanatisharif</username>
	</author>
	<ipAddress>5.239.228.205</ipAddress>
	<isClosed>false</isClosed>
	<isDeleted>false</isDeleted>
</thread>

You have 2 options:

either just paste a link in the topic.content to the original thread page
or somehow, integrate with this plugin https://github.com/psychobunny/nodebb-plugin-blog-comments which @psychobunny wrote.

I couldn't figure out which CMS you're using on http://sanatisharif.ir to see if it's Ghost or WordPress or something else

sanatisharif

please excuse me if my English is poor,

@bentael said:

Ok, looking at the schema, it seems pretty straight forward, we can deal with the anonymous users, and for the non-anonymous, we can create an account for them.

This is a good idea.

@bentael said:

However, there is 1 problem, each disqus thread doesn't really have a content, it just links to another page somewhere.

disqus threads have a link to a website page that loads Comments on it like article in nodebb-plugin-blog-comments .
so every thread have some posts as a child! ( actually post = comment, each thread have multiple posts or nothing ( dead thread ) )
for example:

<post dsq:id="1201879787">
    <id />
    <message>
         <![CDATA[<p>کی جزوه خود اقای رمضانی در مورد نوسان و خازن را می گذارید</p>]]>
    </message>
    <createdAt>2014-01-15T14:24:05Z</createdAt>
    <isDeleted>false</isDeleted>
    <isSpam>false</isSpam>
    <author>
        <email>[email protected]</email>
        <name>علی</name>
        <isAnonymous>true</isAnonymous>
    </author>
    <ipAddress>5.22.92.163</ipAddress>
    <thread dsq:id="2940898899" />
</post>

in last line of this post point to a thread with 2940898899 id.
Also maybe each post has a parent ( for example reply to existing post )
Like this:

<post dsq:id="1202644249">
    <id />
     <message><![CDATA[<p>در صورتی که تمایل دارید به مطالب مورد نظر دسترسی راحت تری داشته باشید،</p><p>هر مطلب را در قسمت مربوط به آن مطرح نمایید</p><p>به طور مثال از این پس سوالات خود را در تالار دبیرستان مطرح نمایید.</p><p>اما پاسخ سوال شما :</p><p><a href="http://www.forum.sanatisharif.ir/b153/t79/msg1391/#msg1391" rel="nofollow">http://www.forum.sanatisharif....</a></p>]]>
    </message>
    <createdAt>2014-01-15T21:45:41Z</createdAt>
    <isDeleted>false</isDeleted>
    <isSpam>false</isSpam>
    <author>
        <email>[email protected]</email>
        <name>دبیرستان غیردولتی پسرانه دانشگ</name>
        <isAnonymous>false</isAnonymous>
        <username>sanatisharif</username>
    </author>
    <ipAddress>2.190.119.94</ipAddress>
    <thread dsq:id="2940898899" />
    <parent dsq:id="1201879787" />
</post>

these two post that I post them here belong to a thread with 2940898899 id :

<thread dsq:id="2940898899">
    <id />
    <forum>sanatisharif</forum>
         <category dsq:id="2822236" />
    <link>http://sanatisharif.ir/Sanati-Sharif-Lesson/11/4/</link>
    <title>جزوات درس فیزیک</title>
    <message />
    <createdAt>2014-08-19T16:13:31Z</createdAt>
    <author>
    <email>[email protected]</email>
         <name>دبیرستان غیردولتی پسرانه دانشگ</name>
         <isAnonymous>false</isAnonymous>
         <username>sanatisharif</username>
    </author>
    <ipAddress>199.30.24.167</ipAddress>
    <isClosed>false</isClosed>
    <isDeleted>false</isDeleted>
</thread>

@bentael said:

You have 2 options:

either just paste a link in the topic.content to the original thread page

or somehow, integrate with this plugin https://github.com/psychobunny/nodebb-plugin-blog-comments which @psychobunny wrote.

I want to merge disqus thread ( thread posts ) to topic that publish it using this plugin https://github.com/psychobunny/nodebb-plugin-blog-comments which @psychobunny wrote.

I think merging with existing article related to this issues:
https://github.com/NodeBB/NodeBB/issues/3951
https://github.com/NodeBB/NodeBB/issues/3106
please tell me, if i'm wrong!

@bentael said:

I couldn't figure out which CMS you're using on http://sanatisharif.ir to see if it's Ghost or WordPress or something else

we don't have any cms, just a php website!

Thanks a lot,

bentael

@sanatisharif, i am not sure you answered my question,

What I am asking you is basically what to put in each Topic's content, like the main-post, or topic.content

Also,

maybe each post has a parent ( for example reply to existing post )

Yea, that's fine, that's easy.

sanatisharif

@bentael sorry about that,
We actually have a topic for each Disqus thread in NodeBB right now. We want to add those Disqus posts to the existing topics we have. Does it answer your question?

sanatisharif

@bentael
First of all I think it is a good idea to mention that we are using NodeBB as our forum and also comment system.I mean by using comment plugin we have published our webpages on our forum so we have a forum topic for each page.
Here is my suggestion for this plugin:
The plugin could have a section for uploading Disqus entries file . After uploading the file the plugin will recognize Disqus threads and will show me a table like below:

0_1452513696214_screenshot-www.tablesgenerator.com 2016-01-11 15-31-26.png

As you see this table contains my Disqus threads' title and link and a blank column for putting the suitable NodeBB Topic ID of the thread. So after filling in this table and pressing on a submit , Disqus posts will be inserted to desirable topics. Forgive me for pointing out two issues here:
a. Two or more threads can have same topics
b. Similar to NodeBB posts , Disqus posts have insertion time, so during the merging process the time issue should be taken care of .

So do you think it can be done?
Thanks a lot

bentael

@sanatisharif said:

We actually have a topic for each Disqus thread in NodeBB right now. We want to add those Disqus posts to the existing topics we have. Does it answer your question?

yes, that makes more sense

So after filling in this table and pressing on a submit

Ok, I understand, but are you saying you want to manually map each disqus_thread_id to an existing nodebb_topic_id?
just do it in an excel sheet, and export it as another CSV file, then the importer-plugin can use that, I am not gonna build an interface to enter these data, use excel - unless you think this can be done automatically, then I will help you with that

sanatisharif

@bentael

I don't get this idea of importing a CSV file. I mean the main point in my idea is that the plugin is going to fetch all of the threads out of the Disqus export for me and then I enter the correspondent NodeBB topic for each one .In this new idea of yours how is this gonna work?

sanatisharif

@bentael
I've been thinking about all of this and I think perhaps I get you now. Is your plugin going to do the insertion part for us? I mean are you saying that we give you that table I talked about (as a CSV or something) and then the plugin is going to add the Disqus posts according to that? In the other words this plugin is not going to fetch any Disqus posts from anythning!

bentael

@sanatisharif
actually you don't even need the CSV. All you have to do is:

export the XML file from Disqus
open that file in an editor, then "manually" add an field to each <thread> xml node, for example, i just added nbb:id="123456789" right next to the dsq:id

<thread dsq:id="2940898899" nbb:id="123456789">
    <id />
    <forum>sanatisharif</forum>
         <category dsq:id="2822236" />
    <link>http://sanatisharif.ir/Sanati-Sharif-Lesson/11/4/</link>
    <title>جزوات درس فیزیک</title>
    <message />
    <createdAt>2014-08-19T16:13:31Z</createdAt>
    <author>
    <email>[email protected]</email>
         <name>دبیرستان غیردولتی پسرانه دانشگ</name>
         <isAnonymous>false</isAnonymous>
         <username>sanatisharif</username>
    </author>
    <ipAddress>199.30.24.167</ipAddress>
    <isClosed>false</isClosed>
    <isDeleted>false</isDeleted>
</thread>

sanatisharif

@bentael

Hi again, We have built the XML file and here is the link to file :
http://dl1.sanatisharif.ir/disqus_remove_with_comment_tid.xml

Thanks for every thing.

bentael

@sanatisharif few questions.

ok, seems like all you want to do is import the disqus comments as NodeBB reply-posts to the relevant topics, minding the timestamps of the comments.
How come I don't see disqus enabled on your site? did you disable it?
some of the disqus threads are using the same nbb:id, is that correct? for example: dsq:id="2278882614" and dsq:id="2278946707" they are both using nbb:tid="1356"
I am gonna need a snapshot of your NodeBB dump to test against, dont share it publicly here as it may contain sensitive info, email it to [email protected]

This seems like one-time-use script, I can do it, but it's custom, it's not like a feature I can easily add to the -import plugin. The other thing is that I am gonna be every busy in the next 2 weeks, so that's gonna have to wait till early February

sanatisharif

@bentael

Thanks a lot . We appreciate your efforts so much.
Here are the answers to your questions:

Yes, that's exactly the case.
As you know we have migrated to NodeBB and we don't use Disqus anymore. Do you need Disqus to be enabled ? Why?
Yes , that's true . we have some Disqus threads that goes to the same NodeBB topic.
With all due respect we can't share this in any way. But an idea for doing this could be creating some dump posts in a local NodeBB and adding some Disqus threads to those posts to see whether the plugin works or not.

Finally about this being a one-time-script, I have to say that I disagree. I think this could be a plugin that helps anyone who wants to migrate from Disqus to NodeBB. Because in any case you have some Disqus threads that must go to a particular NodeBB topic. In the other words they gotta export their Disqus threads (which includes pots as well) and decide which thread goes to which NodeBB topic. Then they will be looking for a plugin to do the task of adding posts to NodeBB. Is that right?

sanatisharif

@bentael

we can add some input box to admin page of plugin, for customization blocks!

function setTopicId(json,thread,callback){
  /**
  * some customization.
  *
  **/
  callback(json,thread,tid);
}
function getNodebbTopicId(link){
 var id = -1 ;
 /**
 *each website has their own implementation.
 *
 **/
 return id;
}
function getCommentID(json, callback){

        var threads =  json.disqus.thread ;
        var itemsProcessed = 0;

        for (i = 0; i < threads.length; i++) {
            try{

                var link = parseInt(threads[i].link) ;
                var id = getNodebbTopicId(link);
                threads[i].$['nbb:tid'] = "";
                setTopicId(json,id,threads[i],function(json,thread,tid){
                    itemsProcessed++;
                    thread.$['nbb:tid'] = tid+"";
                    if(itemsProcessed === threads.length) {
                        callback(json);
                    }
                });
            }catch(ex){
                itemsProcessed++;
                console.error(ex);
            }

        }
};
var json = // Convert disqus xml file to json Object using some library, for example xml2js!

getCommentID(json, function(){
});

bentael

@sanatisharif said:

Do you need Disqus to be enabled ? Why?

No, i just wanted to able to verify the correctness by matching them manually.

With all due respect we can't share this in any way. But an idea for doing this could be creating some dump posts in a local NodeBB and adding some Disqus threads to those posts to see whether the plugin works or not.

That would work i guess.

helps anyone who wants to migrate from Disqus to NodeBB.

Not everyone has a nodebb-topic matching each disqus-thread, most people just use disqus as a comments system on a blogs, and migrating blogs to NodeBB isn't easy. The other issue that it involves manually adding that nbb;id thing, you had to do it manually.

Anyways, like i mentioned, I am out of town of the next 2 weeks, i'll look at this when I get back, feel free to remind me please.

sanatisharif

@bentael

The other issue that it involves manually adding that nbb;id thing, you had to do it manually.

We have implemented a piece of code for that and we were going to publish it . But even if he couldn't use the code , at least there is a plugin that does half of the way.

We can't thank you enough for being so much supportive and patient. See you in couple of weeks . Good luck.

sanatisharif

@bentael

Hi I'm back again , I hope you are fine and in the mode for doing this. So how do you feel? I say let's get this done .

FDX

I recently migrated to NodeBB from Disqus as well, it would be great to see some easy solution for this

By the way, @sanatisharif , since you are in the same situation you may be interested in reply/quote feature for nodebb-plugin-blog-comments. You have an example here (I think I need to change the font-size of that page ).