Embedded Metadata for Posts and other Objects

seh

Wondering what is the best way to store semantic metadata in posts. This would include things like:

key=value pairs for different types of values: numbers, strings, boolean, etc
space locations (geographic latitude,longitude)
time locations (unixtime integer is sufficient)
etc..

One way is to embed some JSON, like:

{ x: 12, y: 'b' }

Another way is to use a markdown macro:
[[x=12 y="b"]]

Some or all of these data fields can be cached in the database when a post is updated.

They can also be analyzed to create widgets for viewing and editing. For example, if

[[length value="12" units="km"]]

can create a text input box when the post is edited.

We can connect this to the tagging system to support schemas/ontologies such as:
http://schema.org
http://dbpedia.org
http://www.wikidata.org
https://github.com/automenta/netjs/blob/master/server/general.js
https://github.com/automenta/netjs/tree/master/plugin/schema_org

Giorgio Chiodi

@seh Yes please

psychobunny

I feel like storing stringified JSON would be simpler for you to work with. Just from the plugin's point of view, no idea about the software that actually reads the data in the end

seh

@psychobunny we can read it back in on the server-side using a filter, and store in the database for fast access and indexing. but i'm wondering if this is something that should be a global feature, allowing any plugin to read and write arbitrary metadata (keys/values) per post. i think there was something in the NodeBB server code about 'Post Fields' - is this similar somehow?

@psychobunny i'm not sure yet if what i'm planning is antithetical to NodeBB architecture or not. if you have some time i can guide you through Netention's features so that you might suggest how you would implement them in NodeBB. http://geekery.biz is a live demo (anonymous login works) but it may not be obvious how to reach all the features.

also, what's the best way to specify a GUID for a topic/post (or equivalent type of field) so that we can later replace it with a new copy? for example, in RSS, each post already has a GUID. i'd like to re-use that as the topic ID so that when the RSS is reloaded, i want to be able to replace the content most easily and efficiently, with minimal # of DB queries. this will also support some of the other interfaces I want, such as P2P synchronization between servers.

thanks!

psychobunny

Have a look at this thread too, there was a discussion similar to your above question

i think there was something in the NodeBB server code about 'Post Fields' - is this similar somehow

Not really, I think you're talking about filter:post.getFields right? That's more for embedding data at the bottom of your post (where the up/down vote buttons are in this current theme).

also, what's the best way to specify a GUID for a topic/post (or equivalent type of field) so that we can later replace it with a new copy?

A post ID (aka PID) is available in the topic api call, if that's what you're looking for?

seh

@psychobunny said:

Have a look at this thread too, there was a discussion similar to your above question

right. i like the JSON option best

also, what's the best way to specify a GUID for a topic/post (or equivalent type of field) so that we can later replace it with a new copy?

A post ID (aka PID) is available in the topic api call, if that's what you're looking for?

for example, in:

https://github.com/automenta/nodebb-plugin-webin/blob/master/index.js#L152

i would like to specify a topic TID (coming from the RSS item's GUID) so that when i encounter it again on the next RSS reload, i can replace it in one database query. right now a topic's new TID is auto-incremented, not specified by a parameter:

https://github.com/NodeBB/NodeBB/blob/master/src/topics/create.js#L23

maybe there's some other way to do this without changing the current TID design. i could store a mapping of GUIDs to TIDs in the RSS plugin but that doesn't seem like a good idea.

i see there's a few ways to find topics (by TID, by UID, etc..) but not by some other piece of metadata.

https://github.com/NodeBB/NodeBB/blob/master/src/topics.js#L171

seh

btw, here is the netention web server code:

https://github.com/automenta/netjs/blob/master/server/web.js

sorry it's a bit of a mess combining DB (mongodb), express, socket.io setup, and all the API calls in one file. here's a list of the key functions:

loads (at startup) and saves (periodically) system metadata in the database

function loadState,
function saveState

for the server processes new data, coming from client or another server; usually results in add to database

function notice(o, whenFinished, fromSocket) {
function noticeAll(objList) -

delete functions

function deleteObject(objectID, whenFinished, contentAddedToDeletedObject, byClientID) {
function deleteObjects(objs, whenFinished) {
function deleteObjectsWithTag(tag, callback) {

query

function getObjectByID(uri, whenFinished) {
function getObjectsByAuthor(a, withObjects) {
function getObjectsByTag(t, withObject, whenFinished) {
function getLatestObjects(n, withObjects, withError) {
function getExpiredObjects(withObjects) {

users can have multiple profiles/personas:

function addClientSelf(req, uid) {

function getCurrentClientID(req) {

function getClientSelves(req) {

        socket.on('become', function(target, _onResult) {  when client first login or switches identity

        socket.on('disconnect', function() { client disconnect

multi-level scope system for deciding who can access what object: public, trusted, private

function objCanSendTo(o, cid) { implements the

sends new or updated object to all clients and plugins

function broadcast(socket, o, whenFinished) {

when clients publish via websocket

        socket.on('pub', function(message, err, success) {

when client attempts to delete an object (by ID)

        socket.on('delete', function(objectID, whenFinished) {

current ontology for a client to load on startup

express.get('/ontology/:format', function(req, res) {

wikipedia proxy for selecting tags on live wikipages

express.get('/wiki/search/:query', compression, function(req, rres) {
express.get('/wiki/:tag/html', compression, function(req, rres) {
express.get('/wiki/:tag1/:tag2/html', compression, function(req, rres) {

webrtc roster

function updateUserConnection(oldID, nextID, socket) {

function setUserWebRTC(userID, webrtcID, enabled) {

        socket.on('webRTC', function(id, enabled) {

server chat post in specific channel

        socket.on('channelSend', function(channel, message) {

Giorgio Chiodi

@seh said:

global feature, allowing any plugin to read and write arbitrary metadata (keys/values) per post

I also think this would be ideal - we could also use to develop our own plugins for nodebb this way.

@julian @psychobunny

seh

@Giorgio-Chiodi thanks

For now @psychobunny suggested to store JSON data in the post content. This will work if the post data is filtered for client display, and read for updates when sent to the server. This can be used to maintain a separate index of objects described in posts. For example, geolocations can be indexed in its own database for optimized queries. A post might have multiple associated geolocations, or several GeoJSON "shapes".

{ hash: { x: 'y', z: 3}, array: [0.5,"b"], number: 0.2 }

Can Categories be considered a special kind of Tag? The main difference I see is that Topics can only appear in one Category though.