Plugin development 2021 updated - environment and dev workflow - request

[Radu Ionescu]

@radu-ionescu

Example 1 modification to the Persona theme of the category/topic component is a combination of the cards at the top (that show the latest activity - i.e. nodebb-plugin-recent-cards - and present a snippet of the original post) with the Teaser Post where you show the latest comment snippet.

Instead of:

(User Avatar) Title - Votes|Posts|Views || (Teaser Post Snippet)

You would show:

(Original Post Snippet as in latest activity cards) (other info, but compacted)
~~ (Larger Teaser Post Snippet) (other info, but compacted)

And similarly as the nodebb-plugin-recent-cards, the topics cross category boundaries. So now instead of the latest globally, you show the latest from a category. And you place this cards in the category topics lists instead of the normal infinity scroll list/paginated list of topics. You would not need to interlace them with the category topics by time since the topics from the category and subcategories include already also the respective category topics.

[PitaJ]

note: I use the terms "folder" and "directory" interchangeably, both refer to the filesystem.

I did not manage to trigger grunt to recompile with my small changes.

What file exactly were you editing?

My problem was that even now I do not understand where to create this themes folder

When you git clone a repository, like so:

git clone https://github.com/NodeBB/nodebb-theme-quickstart

It will create a folder, nodebb-theme-quickstart in the current working directory. This is your theme directory.

from where to where to create symlinks ("link your theme into the node_modules" cited from your text)

You need to enter the theme directory and run that command

cd nodebb-theme-quickstart
npm link

npm link creates symlinks for you. When you run that command without an argument, it creates a symlink in a central location and will tell you something like

linked /path/to/npm/something/nodebb-theme-quickstart -> /path/to/nodebb-theme-quickstart

Then when you run npm link <module> in the nodebb directory like so

npm link nodebb-theme-quickstart

It will create a link in node_modules that points to the first link and will tell you something like

linked node_modules/nodebb-theme-quickstart -> /path/to/npm/something/nodebb-theme-quickstart -> /path/to/nodebb-theme-quickstart

I don't know why I'm explaining how those commands works, that's something you could have found on your own.

Anyways, the point of this is that the module you're working on is made available to nodebb by being present in node_modules. Is doesn't matter where you initially clone the quickstart repo, as long as it's linked.

that I should just place it the root of an even fake node module like node_modules/my_fake_symlinked_even_module/themes.

Forget anything about a themes/ folder. That's not how it works now. I don't know if it ever worked like that.

---

A quick correction:

Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

You can probably use a union DB operation to get a list of recent topics from multiple categories at once. That would be the optimal way of doing what you want.

---

Example 1:
...
I want to break this topic into smaller topics, i.e. a topic would be a drone video of a bridge that is built on 2nd March, followed by a topic with the results of the public auction for the last segment, a topic about how the plans did not include an overpass at X location, etc. All of this I called anonymous topics since the title is not important as they are in fact part of the same larger topic, but the focus is the content.

Example 2:
...
The same forum where you have a category of Highways and subcategories Highway B1, ... Highway B100. The category Highway should not even contain topics, but you would like to show all the updates from all subcategories (i.e. individual highways) made. Most people would like to hear about all the updates (anyway, true and impactful updates are 1-2 every day across all of the highways). In case you want to dive deeper in a particular highway you go into the respective subcategory. On top of this you would have another category named Road infrastructure that contains Highways and National Roads, and the same argument is valid that you would want to see topics (updates) here from both subsections.

Okay so there are two requests here:

1. Show a topic list for all subcategories
2. Organize categories where some can't have their own topics

I think #2 is covered by "category sections". In the ACP you can edit a category and mark it as a section. If you do so, I think this prevents people from creating topics directly under that category.

As for #1, I think it's certainly possible to add this functionality within your theme. As noted above, there are actually DBAL functions for fetching from multiple sets at once. But I'd recommend you start with just using the node APIs already provided for categories and topics, and then optimize from there once you have something working.

One more thing: the read API and write API are for external services or clients. They're not for plugins to interface with. Unfortunately the API that plugins interface with is the internal undocumented API. You'll see plugins doing things like this:

// import the NodeBB DBAL module
const db = require.main.require('./src/database');

The only way to find things in the internal API is by searching through files in src/. Creating a public documented API here is something that's been on the back burner for a long time. If you have any questions about how to accomplish things there, please ask.

[Radu Ionescu]

@pitaj said in Plugin development 2021 updated - environment and dev workflow - request:

What file exactly were you editing?

I was modifying the build/public/templates/categories.tpl file. It has this template look that I am very familiar with, even from the old days of JSP. But actually comes very close to the Angular feel of template ( {componentData} -> {{componentData}}, <!-- IF -> *ngIf , <!-- BEGIN -> *ngFor).

And when I see similar code like components.get('category/topic', 'index', topicIndex);, my mind goes and recognizes the familiar angular way of defining component properties. What differs is that there is no a Component class and everything is spread in lots of files and coming from a Typescript world, this ES5 style looks a lot like compiled code (i.e. instead of a constructor, inheritance and so on, I have define('topicSelect', ['components'], function (components) {, I need to define by hand a class methods the old-fashioned way module.exports = function (Posts) { Posts.shouldQueue = async function (uid, data))

Afterwards, I understand that NodeBB hooks are a mix of property/event binding of Angular with lifecycle events (onInit,afterViewInit,onDestroy - when I read in the docs or forum that widgets will be called only when rendered for example)

I really have no issue with compile time since I am used from Angular to wait for Typescript compilation (although starting with Angular9+ and Ivy this issue does not exist anymore). Here, if I remember correctly from what I read, everything is JS and ES5.

On top of this, there are these splits of client code between backend and frontend. I actually worked with SSR in Angular and I can understand that even there it is hard to keep this frontier clean. But here, I am simply lost because of the ES5 style combined with the plugin/widget mindset. There are also some architectural design choices that are not obvious when reading the code (and not at all documented to add). Even the fact that you have a .tpl file accompanied by a .js file with the "compiled" code for the render engine is a design choice that is easily explained and should be among the first 3-4 paragraphs in the dev docs.

A critical piece of information is the code organization, as in the layout of directories and some information on files/folder contents. Frankly, one liners like this would be more than enough: "Contains compiled code during build - DO NOT TOUCH - use source files (see X directory/file)", "In this file you find the tpl of componets X,Y,Z (inspect HTML code to see placement in page)", "This file contains the Class definition of X", "Client code for SSR related to". And some of them do not require per file definitions, if from the folder and file name you can clearly see that all of them are variations of the same directory definition.

Like you said, building a forum is hard work, and I appreciate deeply the effort and time invested in an open source forum project. I am just saying that even an advanced user can not dive in and it is very restrictive the entry barrier. With minimal effort, you would have an increase in devs in the community. I am willing and prepared to contribute, but I just can't seem to start.

I don't know why I'm explaining how those commands works, that's something you could have found on your own.

I thank you for taking the time to explain this to me. However, saying that I need to build a node module (cloned from base theme repo to integrate it a plugin for the theme) and use node resolution would have been enough. The problem is that the documentation and forum topics lead to another idea.

Npm link is a way, but I could achieve this with subprojects or use even the foolish node resolution which looks at every node_module folder all the way to your root folder (something that is not very well known about node resolution of modules), place it in the "core" of the project and it will still be used, a local npm repo instead of the npm.js official repo.

If I want to develop this using grunt, I will have to add to the watch this node module to have grunt take care of rebuilding on file changes I make in the child theme repo.

---

A quick correction:

Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

You can probably use a union DB operation to get a list of recent topics from multiple categories at once. That would be the optimal way of doing what you want.

---

Example 1:
...
I want to break this topic into smaller topics, i.e. a topic would be a drone video of a bridge that is built on 2nd March, followed by a topic with the results of the public auction for the last segment, a topic about how the plans did not include an overpass at X location, etc. All of this I called anonymous topics since the title is not important as they are in fact part of the same larger topic, but the focus is the content.

Example 2:
...
The same forum where you have a category of Highways and subcategories Highway B1, ... Highway B100. The category Highway should not even contain topics, but you would like to show all the updates from all subcategories (i.e. individual highways) made. Most people would like to hear about all the updates (anyway, true and impactful updates are 1-2 every day across all of the highways). In case you want to dive deeper in a particular highway you go into the respective subcategory. On top of this you would have another category named Road infrastructure that contains Highways and National Roads, and the same argument is valid that you would want to see topics (updates) here from both subsections.

Okay so there are two requests here:

1. Show a topic list for all subcategories
2. Organize categories where some can't have their own topics

I think #2 is covered by "category sections". In the ACP you can edit a category and mark it as a section. If you do so, I think this prevents people from creating topics directly under that category.

As for #1, I think it's certainly possible to add this functionality within your theme. As noted above, there are actually DBAL functions for fetching from multiple sets at once. But I'd recommend you start with just using the node APIs already provided for categories and topics, and then optimize from there once you have something working.

One more thing: the read API and write API are for external services or clients. They're not for plugins to interface with. Unfortunately the API that plugins interface with is the internal undocumented API. You'll see plugins doing things like this:

// import the NodeBB DBAL module
const db = require.main.require('./src/database');

The only way to find things in the internal API is by searching through files in src/. Creating a public documented API here is something that's been on the back burner for a long time. If you have any questions about how to accomplish things there, please ask.

This is very helpful information on how to achieve this.

I will try out first the child theme cloning later today and come back with feedback. I will focus on just simple CSS/HTML changes and see how it goes.

I should have more time during the weekend to dive in even deeper in data retrieval.

[PitaJ]

I was modifying the build/public/templates/categories.tpl file.

Don't modify any files in build. They're all the results of the build process. Even files that look like source files (like the tpl files) are not, and are already processed to some degree. Any changes you make will be overwritten when building.

No offense intended, but why did you think touching files in build/ was a good idea after I said source files were compiled into build/?

It has this template look that I am very familiar with, even from the old days of JSP. But actually comes very close to the Angular feel of template ( {componentData} -> {{componentData}}, <!-- IF -> *ngIf , <!-- BEGIN -> *ngFor).

Yeah you'll find source tpl files in themes, plugins, and src/views that are like that. They're benchpress templates.

And when I see similar code like components.get('category/topic', 'index', topicIndex);, my mind goes and recognizes the familiar angular way of defining component properties. What differs is that there is no a Component class and everything is spread in lots of files and coming from a Typescript world, this ES5 style looks a lot like compiled code (i.e. instead of a constructor, inheritance and so on, I have define('topicSelect', ['components'], function (components) {, I need to define by hand a class methods the old-fashioned way module.exports = function (Posts) { Posts.shouldQueue = async function (uid, data))

NodeBB was started before component frameworks were widespread, and doesn't use anything like that. The client and server code are split up in a kind of component way, but all of the DOM interaction is extremely manual.

Afterwards, I understand that NodeBB hooks are a mix of property/event binding of Angular with lifecycle events (onInit,afterViewInit,onDestroy - when I read in the docs or forum that widgets will be called only when rendered for example)

There's no data binding. Everything is just rendered to HTML and placed on the page, then all interaction is handled with manually assigned events and such. components is just a helper for selecting elements with a component attribute.

I really have no issue with compile time since I am used from Angular to wait for Typescript compilation (although starting with Angular9+ and Ivy this issue does not exist anymore). Here, if I remember correctly from what I read, everything is JS and ES5.

On top of this, there are these splits of client code between backend and frontend. I actually worked with SSR in Angular and I can understand that even there it is hard to keep this frontier clean. But here, I am simply lost because of the ES5 style combined with the plugin/widget mindset. There are also some architectural design choices that are not obvious when reading the code (and not at all documented to add). Even the fact that you have a .tpl file accompanied by a .js file with the "compiled" code for the render engine is a design choice that is easily explained and should be among the first 3-4 paragraphs in the dev docs.

There's not really any client code on the backend, unless you're talking about the templates. There's only a small amount of data processing before passing that data into the template.

A critical piece of information is the code organization, as in the layout of directories and some information on files/folder contents. Frankly, one liners like this would be more than enough: "Contains compiled code during build - DO NOT TOUCH - use source files (see X directory/file)", "In this file you find the tpl of componets X,Y,Z (inspect HTML code to see placement in page)", "This file contains the Class definition of X", "Client code for SSR related to". And some of them do not require per file definitions, if from the folder and file name you can clearly see that all of them are variations of the same directory definition.

Given my previous explanation of the directory structure, what was still confusing?

I thank you for taking the time to explain this to me. However, saying that I need to build a node module (cloned from base theme repo to integrate it a plugin for the theme) and use node resolution would have been enough. The problem is that the documentation and forum topics lead to another idea.

Npm link is a way, but I could achieve this with subprojects or use even the foolish node resolution which looks at every node_module folder all the way to your root folder (something that is not very well known about node resolution of modules), place it in the "core" of the project and it will still be used, a local npm repo instead of the npm.js official repo.

We only support placing modules in node_modules. A lot of our code depends on this being the case. We recommend linking because this works best for us, and we don't want people to lose their work developing directly in node_modules/nodebb-plugin-yours when npm decides to clean up.

If I want to develop this using grunt, I will have to add to the watch this node module to have grunt take care of rebuilding on file changes I make in the child theme repo.

No, because our grunt is set up to already watch for changes in nodebb-plugin-*, nodebb-theme-*, etc modules in node_modules. It's not any more complicated than just running grunt, making changes in the source files, and waiting for it to automatically rebuild and restart.

I will try out first the child theme cloning later today and come back with feedback. I will focus on just simple CSS/HTML changes and see how it goes.

I should have more time during the weekend to dive in even deeper in data retrieval.

Good luck!

[Radu Ionescu]

I managed to make the simple theme change work as per your guidance. Just a follow up on my experience

No, because our grunt is set up to already watch for changes in nodebb-plugin-*, nodebb-theme-*, etc modules in node_modules. It's not any more complicated than just running grunt, making changes in the source files, and waiting for it to automatically rebuild and restart.

I added manually my child theme module in the watch list of grunt. But, the issue was coming from the fact that my changes in the *.tpl files in the child theme were not triggering, just the changes to the *.js files. Now it seems to work fine for all file types and I simply will leave these to have been caused by some messy caching IntelliJ was doing with the FS write buffer.

Given my previous explanation of the directory structure, what was still confusing?

It will still be confusing for me for a long time until I master it a bit, but your explanations were very helpful and I understood what you were saying.

My comment was related to asking a Q&A question for this and maybe you could place it in the Developers FAQ section, that I understood also serves as complementary documentation from answers in the forum. It was a suggestion nonetheless. My typical hand-over discussion or introduction of a new team member to the code base would start at presenting the code organization (strictly referring to the technical discussion, not the business related one about product scope and features)

Good luck!

The easy part is done. Now I will have to find the inspiration in the nodebb-plugin-recent-cards to see how I can pull in the /api/category/{cid}/{slug} response also the original post snippet. I could only find the teaser post field for the topic.

[PitaJ]

I added manually my child theme module in the watch list of grunt. But, the issue was coming from the fact that my changes in the *.tpl files in the child theme were not triggering, just the changes to the *.js files.

Did it not work before doing that? The gruntfile should without any changes detect any tpl changes as long as your theme is active. If it doesn't, that's a bug that we should fix.

Again: you shouldn't need to change to gruntfile at all. It should just work, and does in our experience.

[Radu Ionescu]

@pitaj I just removed it and it works without that explicit addition of the module. My particular setup could be the culprit to blame and some OS caching of writes to disk, for sure. All my tools are installed in an Ubuntu WSL environment and I am editing in IntelliJ.

I started grunt and installed the theme in the panel. There was a short restart for the install to take effect. At this point the watch was not triggering on my module.

... warn: [plugins/load] The following plugins may not be compatible ...
  * nodebb-plugin-emoji

I restarted grunt and at this point the triggering was okay, but I could also see the plugin being picked up in grunt output.

... warn: [plugins/load] The following plugins may not be compatible ...
  * nodebb-plugin-emoji
  * nodebb-theme-persona-news-forum

...

>> File "node_modules/nodebb-theme-persona-news-forum/templates/topics_list.tpl" changed

Anyway, I could not see the template changes until I made this refactoring in the nodebb-theme-quickstart\lib\theme.js

var Theme = module.exports;
var Theme = {};
...
module.exports = Theme;

JS has so many crazy particularities that I do not know if my change actually means something (it was a bug in the code initially or not). Working for the past year mostly with Typescript, and working with JS ES6+ format almost exclusively, I honestly do not know if what I did needed to be done.

[PitaJ]

@radu-ionescu that JS change you made makes no difference. I suspect that the restart from changing it led to the template cache to be flushed either on the server side or client side.

[Bejan]

@radu-ionescu said in Plugin development 2021 updated - environment and dev workflow - request:

Just started reading this thread, it has come a long way..

Nonetheless, the occasions multiply when I feel the urge to chime in. I'm in a similar position as you seem to be: sw dev w/ xp but being overwhelmed with the view the whole nodebb package. So far I dug my way through, having many questions, most of them cleared as of now, thanks to some awesome devs. But Me Too I suffer from the incomplete and outdated doc syndrome.

The docs says that you can be more efficient if you limit scope like ./nodebb build adminjs admincss tpl. But I did not find the definition of those scopes and I would be clueless on what scope to use.

I started with selective builds with more or less educated guesses which scope is/might be affected. More often than not I noticed, those guesses where just those - guesses. Sometimes right, sometimes not and often partially right as other scopes were inflicted, too. So my solution is the big hammer called "just-do-it-all". PITA. Go-around roughly 60 secs for build/restart/nginx reconnecting with nodebb instances. Not using grunt any more, not compatible with multiple nodebb instances, yet(?). My workflow ought to be as close to production as possible, so I'm running multiple instances for dev. PITA, again.

A very good point of yours is the need for an architectural overview of nodebb. What's were, how and when of sorts.

I'm using yalc and the compilation step (script) is nudged by

yalc push && ~/nodebb build && sudo systemctl restart nodebb

I'd set up a page to briefly sketch how the cogs work together if there's interest. Create child theme, register with yalc etc. All of it can be found in different places, I just plugged some of those ideas together in what works for me (tm).

Edit: I'm so glad @PitaJ having me pointed to this thread as I seem to have missed it completely so far.

[Bejan]

@gotwf said in Plugin development 2021 updated - environment and dev workflow - request:

Like all powerful tooling, Emacs doe have its learning curve. But it is free and pretty much does any and everything.

.. and even coffee.

As does vim with a couple of bundles. This is what I'm used to and also working with on nodebb.

It's just that you have to pick one and try it or learn how to do it.