Plugin development 2021 updated - environment and dev workflow - request

[Radu Ionescu]

Hello everybody! I just started using NodeBB and as a developer I want to dive in and modify it and I have some ideas for some widgets/layouts.

My problem is that for the past days I have been trying to find the courage to write some code, but I simply do not know how to start. I can only find topics from 5-6 years ago, mentions of plugins on GH with code not updated for years.

Could someone publish an updated guide which covers a recent version of the forum (1.16.x), please? Or at least some tips in this topic.

I have been working with Angular for frontend and I would like to setup something similar for developing in NodeBB like hot/live reload in development - or something with a fast update during development.

How can I write a module/component from scratch or modify an in-place clone to see the result. I could modify directly the code, but my dev mind would literally freak-out. I want to publish my code as a plugin/widget when complete and maintain it with following updates.

Before being judged for being lazy, I simply can say that what brought me to NodeBB, the flexibility and modularity of the code, has evolved now into the nightmare complexity in which a person trying to join the dev community like me is seeing.

I have included here some examples of what I was aiming to develop and what are my issues with this. They could be used as examples to spark the conversation.

1. Theme or widget? Core or plugin?

In the Persona theme I want to modify the category/topic component to show a snippet of the post (like it is displayed in the Teaser Post when using the option Show the latest post, including the original post, if no replies) . I want to remove the title, as in my case the topics are just simple breakdowns of the category or lets just call them updates (anonymous, hence no title). I wanted to copy the compact display using the small icons and numbers (like in the post/header component) instead of the 3 large text columns of Votes, Posts, Views. And a more advanced feature to show the last 3 replies in the topics underneath the original post snippet (or simply the last as it is shown in the Teaser Post when there is a reply)

Easy peasy, just moving some code around and some HTML,CSS, I said. But, boy I was wrong. First thing I do not get is should this code go into a widget or a theme.

If it is a theme, do I need to copy the whole Persona theme just to have this small change. I could learn the ins and outs of the extend panel and overwrite somehow there, but that's not development, not maintainable, not sharable. I read that I can place a tpl file in the /themes directory, but I do not understand how can I share my code with others (or where to add correctly and in modular fashion the js code to go along)? How does a partial theme based on Persona integrate with other themes?

So somehow it is a widget, I guess. How do I write/develop my own widget? How can I make it overwrite a theme element? How can I make it overwrite a core functionality (the Teaser Post is a general setting)

2. Where is what?

I want the topics from subcategories to show up in the category. I said that using the nodebb-widget-essentials (I found it mentioned to be a good reference point for newbies developers in a topic here) I would find a way since I can reuse the 'Recent', 'Popular' as inspiration, together with some core functions in the Read API. Again I hit a wall looking at this section in the forum and the docs. The docs kind of address how you publish a well defined widget (define widgetid, name, description) and the OpenAPI interface does not provide good usage examples, just syntax.

In the end, I would like a solution that is optimized, using the DBAL efficiently (if access to mongo is not possible), without huge amounts of data to be sent across the wire and all the heavy-lifting of the DB to be done in the browser. But I do admit that this requires some effort, but at this point the basic getAll functions seems out of reach

As a side note, I also have encountered the phrase 'just use VS and it will do it for you' when talking about environment setup. I have grown comfortable with IntelliJ and until now I have not been disappointed with developing anything in python, Java, Docker, Angular, Spring, etc. I would like to think that I can use it for NodeBB as well and if someone has some tips on this, I would thank him immensely.

[PitaJ]

I have been working with Angular for frontend and I would like to setup something similar for developing in NodeBB like hot/live reload in development - or something with a fast update during development.

You won't get hot/live reload, but you can use grunt to automatically rebuild and restart NodeBB when you make a change.

First thing I do not get is should this code go into a widget or a theme.

I don't think you want a widget here.

Widgets are reusable, customizable UI elements that you can place into specific areas in templates to add stuff to footers/headers/sidebars. These widgets areas are defined by the theme.

If you want to modify how an element of the forum looks but keep the same basic style of another theme (like Persona), then you probably want to create a child theme. This way you do not need to copy the whole Persona theme, though you certainly could do that instead.

I want the topics from subcategories to show up in the category.

This is another customization to the theme, so not a widget.

This is a pretty big change from how NodeBB usually works, and it won't be easy to implement.

As a side note, I also have encountered the phrase 'just use VS and it will do it for you' when talking about environment setup.

I don't know the context of that, but you can use any IDE or editor capable of working with JS. What environment setup are you looking for?

[yariplus]

IntelliJ page says only the Ultimate version supports node.

Bummer.

[Radu Ionescu]

@yariplus What I wanted to say is that my research on the forum lead me to find this phrase quite a lot: "Just use VS to develop NodeBB". And it was kind of written as the solution to setting the development environment. I must say that I am happy also developing in a text editor node and web files, only that the learning curve on a new project (especially complex like this one) is much easier with an IDE like IntelliJ as you benefit quite a lot from its indexing of files (search,autocomplete,jump to definition,etc). Plus it does a smart indexing and not just blind text indexing.

It is not very expensive to have Ultimate for personal use. Plus it has this perpetual fallback license where you own for a lifetime the version you bought, which makes the IDE usable for 2-3 years minimum (updates are not major with new versions when it comes to pure IDE functions).

[Radu Ionescu]

You won't get hot/live reload, but you can use grunt to automatically rebuild and restart NodeBB when you make a change.

I run it with the grunt command and looked in the watch task to understand what files are synced. I understand the limitations of not getting a full browser sync and webpack-dev-server experience. However this are some problems I am facing:

1. Grunt watches at compiled code (mostly the public folder). It is a nightmare to work in minified code, although obfuscation is not used. Even if you accomplish the task of change, you will need to go back and figure out where in the source to reflect the change. If you want to adhere to the DRY, SOLID principles, you would want to makes this changes into modules (plugins, widgets, child theme), which is a painful process at the moment. Just to make it clear, I am talking even about a small change like changing the color of a button, adding a span. Nothing even remotely advanced like I said above. Those were my target, but I wanted to start small, but I realize there is no easy way even to just learn by experimenting.

2. The docs is missing in my opinion a fundamental page that should be the very first thing a developer should find: Code organization. Here is some minimal information I could not find (although I could figure out some high level details, I will play the pure dumb card to make the point)
   2.1. What goes where and why ? What is the difference between the /src files and the /build/public files. Why is the /public folder not empty as it should normally hold only compiled code from the sources (the build process should be able to delete it and recreate it fully), are the /public/less stylesheets fundamental and that is where I make changes? What is server side code and what is client side code? From the docs, I understand that rendering can be done (and it is done) both on the server and on the client. It is a great feature, but a pain in the ass since now we are doing MVC throughout client and server. I can not simply grasp the complexity of changing a view or the choices of why a server rendering was used instead of a client one (or linking the counter parts together).
   2.2 The docs are not compatible with an open source solution and look more like the docs I write at work for my team to just have a checkbox from my supervisor (nobody want to do it, but it has to be done just to say it was done). Please consider this as constructive criticism and do not consider this as rude comment for all the developers in the community or the confounders, for which I have a great amount of respect . It is simply frustrating that this amazing forum build is shooting itself in the leg with this. A simple example of this is the fact that I can not understand from the docs even a simple operation "Create nodebb-theme-mytheme/templates ". Where to create this folder? I am pretty sure it is not in the root of the project. Again this has the same problem "your theme's /templates directory, .... You may have to create this directory". Is it /templates, public/templates, build/public/templates or all of my guesses are wrong.

3. With grunt you have to manually restart the build process as it is looking like I said before at compiled code. The docs says that you can be more efficient if you limit scope like ./nodebb build adminjs admincss tpl. But I did not find the definition of those scopes and I would be clueless on what scope to use. I could bet that they are less than 10 people that can use this feature optimally (not to mention it requires a lot of effort to plan, that just running the full build every time would save you time at the end). Again the problem also hints at 2.1, what files are covered by which build scope. What is the full list of build scopes and the architectural design choices behind them.

I don't think you want a widget here.

Widgets are reusable, customizable UI elements that you can place into specific areas in templates to add stuff to footers/headers/sidebars. These widgets areas are defined by the theme.

If you want to modify how an element of the forum looks but keep the same basic style of another theme (like Persona), then you probably want to create a child theme. This way you do not need to copy the whole Persona theme, though you certainly could do that instead.

I could have used the wrong term and widget was meant to comprise the plugin definition in the forum community. I would simply call them modules/packages in terms of programming culture. I understood that widgets are in this community referring to the Android vision of widgets that you glide on the screen in an anchor position, but still you would distribute them as nodebb-plugin-... npm packages.

The fact that you can pinpoint it to a theme change or limit it to that speaks about the fact that there is something missing in the architecture design. It is not just me who has problem in recognizing the design patterns. I can tell you that it is not just a newbie problem. There is such a high barrier at the beginning that it is impossible to begin and most people will just give up. My interest was to use a forum and I was leaning towards NodeBB. I do not wish to become a Jedi master in NodeBB just to be able to change a button from left to right, as I feel it is the case at the moment.

I am still balancing in my head the options of building from scratch a forum in Angular and writing a backend (plain development of my use case and needs without even aiming for making it a public project) or spending the same amount of effort to understand and build the themes/plugins/widgets for NodeBB. I wanted option 2 as it would benefit an open source community of users, but I simply can not find the motivation to begin (or the means). For full disclosure, I am not building this forum for a business or in any way is this related to my work. This is why I am a bit careful on how I spend my time and I kind of inclined to go for option 1, as I will at least gain some valuable experience.

This is another customization to the theme, so not a widget.

This is a pretty big change from how NodeBB usually works, and it won't be easy to implement.

I guess if I want to use the NodeBB forum, I will have to go with already available plugins and just kind of profit from the work of other people. I would have wanted to contribute, but I can not see how I would be able to start to write any code for the forum.

I don't know the context of that, but you can use any IDE or editor capable of working with JS. What environment setup are you looking for?

I replied to this in a comment to @yariplus below

[PitaJ]

@radu-ionescu

Grunt watches at compiled code (mostly the public folder). It is a nightmare to work in minified code, although obfuscation is not used.

I'm not sure what you're talking about. First of all, public contains almost entirely unminified source code (the only exception is public/uploads). build is where the build artifacts are placed. Secondly, grunt is set to track changes to source files, and ignores files in build.

Even if you accomplish the task of change, you will need to go back and figure out where in the source to reflect the change.

Can you give an example?

What goes where and why ? What is the difference between the /src files and the /build/public files. Why is the /public folder not empty as it should normally hold only compiled code from the sources (the build process should be able to delete it and recreate it fully), are the /public/less stylesheets fundamental and that is where I make changes? What is server side code and what is client side code?

src/ contains server-side code. public contains client-side code and a few modules that are shared with the server side. public/less contains mostly just the styles for the ACP. The actual styles and templates of the forum client are contained within whatever theme you're using. All of these client source files are built into the build directory.

The docs are not compatible with an open source solution and look more like the docs I write at work for my team to just have a checkbox from my supervisor

We don't have good docs for developing themes are plugins. This is why we recommend looking at other themes and plugins for examples.

Create nodebb-theme-mytheme/templates

git clone the quickstart theme repo into a directory of your choosing. Then you can rename it to whatever you want and add a templates directory for templates to override. You can then npm link in the theme directory and npm link <your theme in the nodebb directory to link your theme into the node_modules for your nodebb. Then it will show up as installed and you can activate it.

With grunt you have to manually restart the build process as it is looking like I said before at compiled code.

The whole purpose of grunt is to watch for changes in source files and rebuild and restart when a change is made, building only the assets affected by the change.

The docs says that you can be more efficient if you limit scope like ./nodebb build adminjs admincss tpl.

What did you try? ./nodebb help will give an overview and ./nodebb help build will tell you what is available for that command.

I could have used the wrong term and widget was meant to comprise the plugin definition in the forum community. I would simply call them modules/packages in terms of programming culture. I understood that widgets are in this community referring to the Android vision of widgets that you glide on the screen in an anchor position, but still you would distribute them as nodebb-plugin-... npm packages.

Well, here's the nomenclature we use here:

• plugin: a module which plugs into NodeBB and alters or adds behavior
• theme: a kind of plugin meant to provide a customized UI to the NodeBB client, only one theme can be active at a time
• widget: as described previously

I am still balancing in my head the options of building from scratch a forum in Angular and writing a backend (plain development of my use case and needs without even aiming for making it a public project)

Building a forum frontend and backend is not an easy task. If you feel like it as a learning experience you could build a forum front-end in Angular, and hook into the NodeBB backend through the read/write API.

I guess if I want to use the NodeBB forum, I will have to go with already available plugins and just kind of profit from the work of other people. I would have wanted to contribute, but I can not see how I would be able to start to write any code for the forum.

Well hopefully now you have some idea of how to get started. It's definitely not impossible to by any means to implement what you want here through various hooks.

I'm not exactly sure what you want, but it sounds like you want to show a list of the latest N topics from the set of subcategories C. Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

What I wanted to say is that my research on the forum lead me to find this phrase quite a lot: "Just use VS to develop NodeBB". And it was kind of written as the solution to setting the development environment.

I'm thinking you saw "VS Code". Visual Studio Code has a nice built-in JS engine that provides many of the features you want. No setup is required. I see no reason why IntelliJ couldn't do the same for the same codebase.

[gotwf]

@yariplus said in Plugin development 2021 updated - environment and dev workflow - request:

IntelliJ page says only the Ultimate version supports node.

Bummer.

fwiw; Emacs has a nodejs mode. The advent of Clojure several years back resulted in lots of Windows folks installing Emacs to get access to all its development goodies. Hence, there's lot of Emacs on Windows for dev/ide stuff on e.g. Stack Exchange and such. Commercial stuff has probably since caught up. And likely surpassed, if that's what you're used to. Like all powerful tooling, Emacs doe have its learning curve. But it is free and pretty much does any and everything.

[Radu Ionescu]

I'm not sure what you're talking about. First of all, public contains almost entirely unminified source code (the only exception is public/uploads). build is where the build artifacts are placed. Secondly, grunt is set to track changes to source files, and ignores files in build.

...
src/ contains server-side code. public contains client-side code and a few modules that are shared with the server side. public/less contains mostly just the styles for the ACP. The actual styles and templates of the forum client are contained within whatever theme you're using. All of these client source files are built into the build directory.

I would be tempted to open a new topic (maybe Q&A) about the folder structure of the source code (including the files that appear after the ./nodebb setup) to go into the details of this folders and what pieces of code/styles/templates could each of them contain. A good aanswer would be a good candidate for the Developer FAQ section in my opinion and very helpful. Your comment is helpful and I appreciate providing the info, but it still feels like a large puzzle with details spread throughout the docs and forum topics (some of them old and possibly talking about even pre 1.0 versions-it is hard to tell only by the date of the post)

Even if you accomplish the task of change, you will need to go back and figure out where in the source to reflect the change.

Can you give an example?
...
The whole purpose of grunt is to watch for changes in source files and rebuild and restart when a change is made, building only the assets affected by the change.
...
What did you try? ./nodebb help will give an overview and ./nodebb help build will tell you what is available for that command.

I was changing the category.tpl file to try to customize the look of a post in the category/topic component and working towards the Example 1 in my original post. And then came across the issue in not finding easily the css classes referenced in the template file. And finally, the nail in the coffin came from realizing that I find everywhere this component

// modules/components.js
...
'category/topic': function (name, value) {
			return $('[component="category/topic"][data-' + name + '="' + value + '"]');
		},
...

// src/client/category.js
...
if (!config.usePagination) {
			navigator.init('[component="category/topic"]', ajaxify.data.topic_count, Category.toTop, Category.toBottom, Category.navigatorCallback);
		} else {
			navigator.disable();
		}
...

And that my changes require a full compilation since the templates is cached in compiled classes (minified as in not human-readable generated code , but not obfuscated)

// templates/account/topics.js

 return "\r\n\t<li component=\"category/topic\" class=\"row clearfix category-item " + 
          __escape(helper(context, helpers, 'generateTopicClass', [guard(value)])) + 
          "\" data-tid=\"" + 
          __escape(guard(context && context['topics'] && context['topics'][key0] && context['topics'][key0]['tid'])) + 
          "\" data-index=\"" + 
          

I did not manage to trigger grunt to recompile with my small changes. I am having a hard time distinguishing between source files and generated ones (even when studying the watch task in Gruntfile.js to look at folders and files being watched)

Create nodebb-theme-mytheme/templates

git clone the quickstart theme repo into a directory of your choosing. Then you can rename it to whatever you want and add a templates directory for templates to override. You can then npm link in the theme directory and npm link <your theme in the nodebb directory to link your theme into the node_modules for your nodebb. Then it will show up as installed and you can activate it.

I lost the original post that lead me to this link deploy nodebb custom theme, but it talks about the same npm link solution (only that it uses yarn instead).

My problem was that even now I do not understand where to create this themes folder ("theme directory" cited from your text) and from where to where to create symlinks ("link your theme into the node_modules" cited from your text). It is the same issue I found in the docs when I said is reflects the documentation I am writing for my work projects. It tells the story at the end when you are done with the implementation and it comes from a person that spent a lot of time with it. For you, and maybe for other people, this themes directory is something that you are so used to that it is hard to understand even why I am asking about such an insignificant detail. But I still am uncapable of following your example in cloning a theme simply because I do not know where to put the code that I pulled via a git clone of the base theme repo or what symlinks to deploy - since I have the impression that your answer is that is not important where I put it, but that I symlink to the correct place in the end and the node resolution mechanism will find my code if it symlink-ed somewhere in the node_modules folder - and that I should just place it the root of an even fake node module like node_modules/my_fake_symlinked_even_module/themes.

I'm not exactly sure what you want, but it sounds like you want to show a list of the latest N topics from the set of subcategories C. Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

Regarding use case of Example 1:

It is a forum about developments of infrastructure projects for people interested in following or working in the field. Saying that one topic would be Highway B11 with 500km planned and broken into 10 smaller segments with a timeline stretched over 6-7 years. You end up with thousands of small comments all related to the same topic. It resembles an news website, only that the community is the one building the news and there is not a single source. On the plus side you can comment like you would on any news website and you can exchange ideas in the community (unlike a typical blog page/news website where everyone can comment and you do not have community, but rather followers )

I want to break this topic into smaller topics, i.e. a topic would be a drone video of a bridge that is built on 2nd March, followed by a topic with the results of the public auction for the last segment, a topic about how the plans did not include an overpass at X location, etc. All of this I called anonymous topics since the title is not important as they are in fact part of the same larger topic, but the focus is the content.

In a full linear (timeline comments) forum, this conversations become an issue if you keep a single topic for all of them (imagine all 3 updates coming at the same time - people will respond to each of them and the comments get interlaced and the conversation becomes hard to follow)

Regarding use case of Example 2.
The same forum where you have a category of Highways and subcategories Highway B1, ... Highway B100. The category Highway should not even contain topics, but you would like to show all the updates from all subcategories (i.e. individual highways) made. Most people would like to hear about all the updates (anyway, true and impactful updates are 1-2 every day across all of the highways). In case you want to dive deeper in a particular highway you go into the respective subcategory. On top of this you would have another category named Road infrastructure that contains Highways and National Roads, and the same argument is valid that you would want to see topics (updates) here from both subsections.

At the moment the only solution is to have a category Road infrastructure that contains subcategories Highways and National Roads. Highways contains 100 topics named Highway B1, ... Highway B100, each of them 1000+ comments long and with interlaced and messy conversations (actually it does not even encourage commenting and makes it act like a news website with many authors, but no comment section ). Here is an example forum that it is aimed at this and faces this issues forum.peundemerg.ro (it is not English based but I chose a place to get a feel of the issue - you might notice the 1611 page of the topic)

[Radu Ionescu]

@radu-ionescu

Example 1 modification to the Persona theme of the category/topic component is a combination of the cards at the top (that show the latest activity - i.e. nodebb-plugin-recent-cards - and present a snippet of the original post) with the Teaser Post where you show the latest comment snippet.

Instead of:

(User Avatar) Title - Votes|Posts|Views || (Teaser Post Snippet)

You would show:

(Original Post Snippet as in latest activity cards) (other info, but compacted)
~~ (Larger Teaser Post Snippet) (other info, but compacted)

And similarly as the nodebb-plugin-recent-cards, the topics cross category boundaries. So now instead of the latest globally, you show the latest from a category. And you place this cards in the category topics lists instead of the normal infinity scroll list/paginated list of topics. You would not need to interlace them with the category topics by time since the topics from the category and subcategories include already also the respective category topics.

[PitaJ]

note: I use the terms "folder" and "directory" interchangeably, both refer to the filesystem.

I did not manage to trigger grunt to recompile with my small changes.

What file exactly were you editing?

My problem was that even now I do not understand where to create this themes folder

When you git clone a repository, like so:

git clone https://github.com/NodeBB/nodebb-theme-quickstart

It will create a folder, nodebb-theme-quickstart in the current working directory. This is your theme directory.

from where to where to create symlinks ("link your theme into the node_modules" cited from your text)

You need to enter the theme directory and run that command

cd nodebb-theme-quickstart
npm link

npm link creates symlinks for you. When you run that command without an argument, it creates a symlink in a central location and will tell you something like

linked /path/to/npm/something/nodebb-theme-quickstart -> /path/to/nodebb-theme-quickstart

Then when you run npm link <module> in the nodebb directory like so

npm link nodebb-theme-quickstart

It will create a link in node_modules that points to the first link and will tell you something like

linked node_modules/nodebb-theme-quickstart -> /path/to/npm/something/nodebb-theme-quickstart -> /path/to/nodebb-theme-quickstart

I don't know why I'm explaining how those commands works, that's something you could have found on your own.

Anyways, the point of this is that the module you're working on is made available to nodebb by being present in node_modules. Is doesn't matter where you initially clone the quickstart repo, as long as it's linked.

that I should just place it the root of an even fake node module like node_modules/my_fake_symlinked_even_module/themes.

Forget anything about a themes/ folder. That's not how it works now. I don't know if it ever worked like that.

---

A quick correction:

Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

You can probably use a union DB operation to get a list of recent topics from multiple categories at once. That would be the optimal way of doing what you want.

---

Example 1:
...
I want to break this topic into smaller topics, i.e. a topic would be a drone video of a bridge that is built on 2nd March, followed by a topic with the results of the public auction for the last segment, a topic about how the plans did not include an overpass at X location, etc. All of this I called anonymous topics since the title is not important as they are in fact part of the same larger topic, but the focus is the content.

Example 2:
...
The same forum where you have a category of Highways and subcategories Highway B1, ... Highway B100. The category Highway should not even contain topics, but you would like to show all the updates from all subcategories (i.e. individual highways) made. Most people would like to hear about all the updates (anyway, true and impactful updates are 1-2 every day across all of the highways). In case you want to dive deeper in a particular highway you go into the respective subcategory. On top of this you would have another category named Road infrastructure that contains Highways and National Roads, and the same argument is valid that you would want to see topics (updates) here from both subsections.

Okay so there are two requests here:

1. Show a topic list for all subcategories
2. Organize categories where some can't have their own topics

I think #2 is covered by "category sections". In the ACP you can edit a category and mark it as a section. If you do so, I think this prevents people from creating topics directly under that category.

As for #1, I think it's certainly possible to add this functionality within your theme. As noted above, there are actually DBAL functions for fetching from multiple sets at once. But I'd recommend you start with just using the node APIs already provided for categories and topics, and then optimize from there once you have something working.

One more thing: the read API and write API are for external services or clients. They're not for plugins to interface with. Unfortunately the API that plugins interface with is the internal undocumented API. You'll see plugins doing things like this:

// import the NodeBB DBAL module
const db = require.main.require('./src/database');

The only way to find things in the internal API is by searching through files in src/. Creating a public documented API here is something that's been on the back burner for a long time. If you have any questions about how to accomplish things there, please ask.

[Radu Ionescu]

@pitaj said in Plugin development 2021 updated - environment and dev workflow - request:

What file exactly were you editing?

I was modifying the build/public/templates/categories.tpl file. It has this template look that I am very familiar with, even from the old days of JSP. But actually comes very close to the Angular feel of template ( {componentData} -> {{componentData}}, <!-- IF -> *ngIf , <!-- BEGIN -> *ngFor).

And when I see similar code like components.get('category/topic', 'index', topicIndex);, my mind goes and recognizes the familiar angular way of defining component properties. What differs is that there is no a Component class and everything is spread in lots of files and coming from a Typescript world, this ES5 style looks a lot like compiled code (i.e. instead of a constructor, inheritance and so on, I have define('topicSelect', ['components'], function (components) {, I need to define by hand a class methods the old-fashioned way module.exports = function (Posts) { Posts.shouldQueue = async function (uid, data))

Afterwards, I understand that NodeBB hooks are a mix of property/event binding of Angular with lifecycle events (onInit,afterViewInit,onDestroy - when I read in the docs or forum that widgets will be called only when rendered for example)

I really have no issue with compile time since I am used from Angular to wait for Typescript compilation (although starting with Angular9+ and Ivy this issue does not exist anymore). Here, if I remember correctly from what I read, everything is JS and ES5.

On top of this, there are these splits of client code between backend and frontend. I actually worked with SSR in Angular and I can understand that even there it is hard to keep this frontier clean. But here, I am simply lost because of the ES5 style combined with the plugin/widget mindset. There are also some architectural design choices that are not obvious when reading the code (and not at all documented to add). Even the fact that you have a .tpl file accompanied by a .js file with the "compiled" code for the render engine is a design choice that is easily explained and should be among the first 3-4 paragraphs in the dev docs.

A critical piece of information is the code organization, as in the layout of directories and some information on files/folder contents. Frankly, one liners like this would be more than enough: "Contains compiled code during build - DO NOT TOUCH - use source files (see X directory/file)", "In this file you find the tpl of componets X,Y,Z (inspect HTML code to see placement in page)", "This file contains the Class definition of X", "Client code for SSR related to". And some of them do not require per file definitions, if from the folder and file name you can clearly see that all of them are variations of the same directory definition.

Like you said, building a forum is hard work, and I appreciate deeply the effort and time invested in an open source forum project. I am just saying that even an advanced user can not dive in and it is very restrictive the entry barrier. With minimal effort, you would have an increase in devs in the community. I am willing and prepared to contribute, but I just can't seem to start.

I don't know why I'm explaining how those commands works, that's something you could have found on your own.

I thank you for taking the time to explain this to me. However, saying that I need to build a node module (cloned from base theme repo to integrate it a plugin for the theme) and use node resolution would have been enough. The problem is that the documentation and forum topics lead to another idea.

Npm link is a way, but I could achieve this with subprojects or use even the foolish node resolution which looks at every node_module folder all the way to your root folder (something that is not very well known about node resolution of modules), place it in the "core" of the project and it will still be used, a local npm repo instead of the npm.js official repo.

If I want to develop this using grunt, I will have to add to the watch this node module to have grunt take care of rebuilding on file changes I make in the child theme repo.

---

A quick correction:

Since a topic is only associated with one category, not with a category hierarchy, you'd need to grab the latest N topics from each of the subcategories, then sort them by timestamp, then grab the top N. So it may not be very scalable, just because of how they're stored in the DB. There are several ways around this but they'd require varying degrees of difficulty and compromise.

You can probably use a union DB operation to get a list of recent topics from multiple categories at once. That would be the optimal way of doing what you want.

---

Example 1:
...
I want to break this topic into smaller topics, i.e. a topic would be a drone video of a bridge that is built on 2nd March, followed by a topic with the results of the public auction for the last segment, a topic about how the plans did not include an overpass at X location, etc. All of this I called anonymous topics since the title is not important as they are in fact part of the same larger topic, but the focus is the content.

Example 2:
...
The same forum where you have a category of Highways and subcategories Highway B1, ... Highway B100. The category Highway should not even contain topics, but you would like to show all the updates from all subcategories (i.e. individual highways) made. Most people would like to hear about all the updates (anyway, true and impactful updates are 1-2 every day across all of the highways). In case you want to dive deeper in a particular highway you go into the respective subcategory. On top of this you would have another category named Road infrastructure that contains Highways and National Roads, and the same argument is valid that you would want to see topics (updates) here from both subsections.

Okay so there are two requests here:

1. Show a topic list for all subcategories
2. Organize categories where some can't have their own topics

I think #2 is covered by "category sections". In the ACP you can edit a category and mark it as a section. If you do so, I think this prevents people from creating topics directly under that category.

As for #1, I think it's certainly possible to add this functionality within your theme. As noted above, there are actually DBAL functions for fetching from multiple sets at once. But I'd recommend you start with just using the node APIs already provided for categories and topics, and then optimize from there once you have something working.

One more thing: the read API and write API are for external services or clients. They're not for plugins to interface with. Unfortunately the API that plugins interface with is the internal undocumented API. You'll see plugins doing things like this:

// import the NodeBB DBAL module
const db = require.main.require('./src/database');

The only way to find things in the internal API is by searching through files in src/. Creating a public documented API here is something that's been on the back burner for a long time. If you have any questions about how to accomplish things there, please ask.

This is very helpful information on how to achieve this.

I will try out first the child theme cloning later today and come back with feedback. I will focus on just simple CSS/HTML changes and see how it goes.

I should have more time during the weekend to dive in even deeper in data retrieval.

[PitaJ]

I was modifying the build/public/templates/categories.tpl file.

Don't modify any files in build. They're all the results of the build process. Even files that look like source files (like the tpl files) are not, and are already processed to some degree. Any changes you make will be overwritten when building.

No offense intended, but why did you think touching files in build/ was a good idea after I said source files were compiled into build/?

It has this template look that I am very familiar with, even from the old days of JSP. But actually comes very close to the Angular feel of template ( {componentData} -> {{componentData}}, <!-- IF -> *ngIf , <!-- BEGIN -> *ngFor).

Yeah you'll find source tpl files in themes, plugins, and src/views that are like that. They're benchpress templates.

And when I see similar code like components.get('category/topic', 'index', topicIndex);, my mind goes and recognizes the familiar angular way of defining component properties. What differs is that there is no a Component class and everything is spread in lots of files and coming from a Typescript world, this ES5 style looks a lot like compiled code (i.e. instead of a constructor, inheritance and so on, I have define('topicSelect', ['components'], function (components) {, I need to define by hand a class methods the old-fashioned way module.exports = function (Posts) { Posts.shouldQueue = async function (uid, data))

NodeBB was started before component frameworks were widespread, and doesn't use anything like that. The client and server code are split up in a kind of component way, but all of the DOM interaction is extremely manual.

Afterwards, I understand that NodeBB hooks are a mix of property/event binding of Angular with lifecycle events (onInit,afterViewInit,onDestroy - when I read in the docs or forum that widgets will be called only when rendered for example)

There's no data binding. Everything is just rendered to HTML and placed on the page, then all interaction is handled with manually assigned events and such. components is just a helper for selecting elements with a component attribute.

I really have no issue with compile time since I am used from Angular to wait for Typescript compilation (although starting with Angular9+ and Ivy this issue does not exist anymore). Here, if I remember correctly from what I read, everything is JS and ES5.

On top of this, there are these splits of client code between backend and frontend. I actually worked with SSR in Angular and I can understand that even there it is hard to keep this frontier clean. But here, I am simply lost because of the ES5 style combined with the plugin/widget mindset. There are also some architectural design choices that are not obvious when reading the code (and not at all documented to add). Even the fact that you have a .tpl file accompanied by a .js file with the "compiled" code for the render engine is a design choice that is easily explained and should be among the first 3-4 paragraphs in the dev docs.

There's not really any client code on the backend, unless you're talking about the templates. There's only a small amount of data processing before passing that data into the template.

A critical piece of information is the code organization, as in the layout of directories and some information on files/folder contents. Frankly, one liners like this would be more than enough: "Contains compiled code during build - DO NOT TOUCH - use source files (see X directory/file)", "In this file you find the tpl of componets X,Y,Z (inspect HTML code to see placement in page)", "This file contains the Class definition of X", "Client code for SSR related to". And some of them do not require per file definitions, if from the folder and file name you can clearly see that all of them are variations of the same directory definition.

Given my previous explanation of the directory structure, what was still confusing?

I thank you for taking the time to explain this to me. However, saying that I need to build a node module (cloned from base theme repo to integrate it a plugin for the theme) and use node resolution would have been enough. The problem is that the documentation and forum topics lead to another idea.

Npm link is a way, but I could achieve this with subprojects or use even the foolish node resolution which looks at every node_module folder all the way to your root folder (something that is not very well known about node resolution of modules), place it in the "core" of the project and it will still be used, a local npm repo instead of the npm.js official repo.

We only support placing modules in node_modules. A lot of our code depends on this being the case. We recommend linking because this works best for us, and we don't want people to lose their work developing directly in node_modules/nodebb-plugin-yours when npm decides to clean up.

If I want to develop this using grunt, I will have to add to the watch this node module to have grunt take care of rebuilding on file changes I make in the child theme repo.

No, because our grunt is set up to already watch for changes in nodebb-plugin-*, nodebb-theme-*, etc modules in node_modules. It's not any more complicated than just running grunt, making changes in the source files, and waiting for it to automatically rebuild and restart.

I will try out first the child theme cloning later today and come back with feedback. I will focus on just simple CSS/HTML changes and see how it goes.

I should have more time during the weekend to dive in even deeper in data retrieval.

Good luck!

[Radu Ionescu]

I managed to make the simple theme change work as per your guidance. Just a follow up on my experience

No, because our grunt is set up to already watch for changes in nodebb-plugin-*, nodebb-theme-*, etc modules in node_modules. It's not any more complicated than just running grunt, making changes in the source files, and waiting for it to automatically rebuild and restart.

I added manually my child theme module in the watch list of grunt. But, the issue was coming from the fact that my changes in the *.tpl files in the child theme were not triggering, just the changes to the *.js files. Now it seems to work fine for all file types and I simply will leave these to have been caused by some messy caching IntelliJ was doing with the FS write buffer.

Given my previous explanation of the directory structure, what was still confusing?

It will still be confusing for me for a long time until I master it a bit, but your explanations were very helpful and I understood what you were saying.

My comment was related to asking a Q&A question for this and maybe you could place it in the Developers FAQ section, that I understood also serves as complementary documentation from answers in the forum. It was a suggestion nonetheless. My typical hand-over discussion or introduction of a new team member to the code base would start at presenting the code organization (strictly referring to the technical discussion, not the business related one about product scope and features)

Good luck!

The easy part is done. Now I will have to find the inspiration in the nodebb-plugin-recent-cards to see how I can pull in the /api/category/{cid}/{slug} response also the original post snippet. I could only find the teaser post field for the topic.

[PitaJ]

I added manually my child theme module in the watch list of grunt. But, the issue was coming from the fact that my changes in the *.tpl files in the child theme were not triggering, just the changes to the *.js files.

Did it not work before doing that? The gruntfile should without any changes detect any tpl changes as long as your theme is active. If it doesn't, that's a bug that we should fix.

Again: you shouldn't need to change to gruntfile at all. It should just work, and does in our experience.

[Radu Ionescu]

@pitaj I just removed it and it works without that explicit addition of the module. My particular setup could be the culprit to blame and some OS caching of writes to disk, for sure. All my tools are installed in an Ubuntu WSL environment and I am editing in IntelliJ.

I started grunt and installed the theme in the panel. There was a short restart for the install to take effect. At this point the watch was not triggering on my module.

... warn: [plugins/load] The following plugins may not be compatible ...
  * nodebb-plugin-emoji

I restarted grunt and at this point the triggering was okay, but I could also see the plugin being picked up in grunt output.

... warn: [plugins/load] The following plugins may not be compatible ...
  * nodebb-plugin-emoji
  * nodebb-theme-persona-news-forum

...

>> File "node_modules/nodebb-theme-persona-news-forum/templates/topics_list.tpl" changed

Anyway, I could not see the template changes until I made this refactoring in the nodebb-theme-quickstart\lib\theme.js

var Theme = module.exports;
var Theme = {};
...
module.exports = Theme;

JS has so many crazy particularities that I do not know if my change actually means something (it was a bug in the code initially or not). Working for the past year mostly with Typescript, and working with JS ES6+ format almost exclusively, I honestly do not know if what I did needed to be done.

[PitaJ]

@radu-ionescu that JS change you made makes no difference. I suspect that the restart from changing it led to the template cache to be flushed either on the server side or client side.

[Bejan]

@radu-ionescu said in Plugin development 2021 updated - environment and dev workflow - request:

Just started reading this thread, it has come a long way..

Nonetheless, the occasions multiply when I feel the urge to chime in. I'm in a similar position as you seem to be: sw dev w/ xp but being overwhelmed with the view the whole nodebb package. So far I dug my way through, having many questions, most of them cleared as of now, thanks to some awesome devs. But Me Too I suffer from the incomplete and outdated doc syndrome.

The docs says that you can be more efficient if you limit scope like ./nodebb build adminjs admincss tpl. But I did not find the definition of those scopes and I would be clueless on what scope to use.

I started with selective builds with more or less educated guesses which scope is/might be affected. More often than not I noticed, those guesses where just those - guesses. Sometimes right, sometimes not and often partially right as other scopes were inflicted, too. So my solution is the big hammer called "just-do-it-all". PITA. Go-around roughly 60 secs for build/restart/nginx reconnecting with nodebb instances. Not using grunt any more, not compatible with multiple nodebb instances, yet(?). My workflow ought to be as close to production as possible, so I'm running multiple instances for dev. PITA, again.

A very good point of yours is the need for an architectural overview of nodebb. What's were, how and when of sorts.

I'm using yalc and the compilation step (script) is nudged by

yalc push && ~/nodebb build && sudo systemctl restart nodebb

I'd set up a page to briefly sketch how the cogs work together if there's interest. Create child theme, register with yalc etc. All of it can be found in different places, I just plugged some of those ideas together in what works for me (tm).

Edit: I'm so glad @PitaJ having me pointed to this thread as I seem to have missed it completely so far.

[Bejan]

@gotwf said in Plugin development 2021 updated - environment and dev workflow - request:

Like all powerful tooling, Emacs doe have its learning curve. But it is free and pretty much does any and everything.

.. and even coffee.

As does vim with a couple of bundles. This is what I'm used to and also working with on nodebb.

It's just that you have to pick one and try it or learn how to do it.