[nodebb-plugin-iframely] IFramely Embeds
-
I hope I'm not posting this in the wrong location. I'm trying to run the self hosted iframely to use with my forum. It seems to work fine when I go to the end point of my server in a browser however, when the call is made from Nodebb using the plugin, I see this error in bash. The link is displayed as a normal url in the forum. I'm sure it's just me doing something dumb and not seeing the obvious.
uri.match is not a function
TypeError: res.sendJsonCached is not a function
at respondWithError (/home/user/misc/iframely/app.js:97:9)
at Object.errorHandler [as handle] (/home/user/misc/iframely/app.js:128:7)
at next (/home/user/misc/iframely/node_modules/connect/lib/proto.js:185:17)
at Object.logErrors [as handle] (/home/user/misc/iframely/app.js:58:3)
at next (/home/user/misc/iframely/node_modules/connect/lib/proto.js:185:17)
at next (/home/user/misc/iframely/node_modules/connect/lib/proto.js:187:11)
at next (/home/user/misc/iframely/node_modules/connect/lib/proto.js:195:7)
at Object.handle (/home/user/misc/iframely/app.js:37:3)
at next (/home/user/misc/iframely/node_modules/connect/lib/proto.js:190:15)
at multipart (/home/user/misc/iframely/node_modules/connect/lib/middleware/multipart.js:64:37)If this post belongs elsewhere feel free to shoo me along to the right spot.. Thanks for the help!
-
@teh_g said in [nodebb-plugin-iframely] IFramely Embeds:
Something I've noticed in Iframely is that the number of API hits has been steadily increasing over time. I reached out to Iframely, and they noted that NodeBB appears to be making the majority of the API calls.
I've taken a look at your hits, and as I see the more from API calls than from iFrames. It means your NodeBB is making the most hits.
The way NodeBB folks have it implemented is that they keep the cache for 1 day in server's memory. If you restart the servers, for example, - the cache will be reset. Also, if your RAM is limited, some of the records can be evicted from cache prematurely.
We did helped NodeBB team develop the plugin - but from the point where the link is already extracted and till the point before it needs to be stored. So, basically - connecting to API and UI is our part. However, if you have a pull-request for longer cache times or eliminating the older links - my team should be able to review, merge and push it to NPM.I am far from a developer, so I'll ask if you guys can take a look at the plugin and try and add some method of lowering API hits. Perhaps if a post is older than x days, don't attempt to cache the Iframely hit until someone loads that post. Maybe going a step further and removing the embed if the post is over a certain age.
Any way to get traction on this?
I actually go to a point where I tried (and failed terribly) to setup a camo proxy and iframely on a Digital Ocean droplet. I'd love to not have to stumble through trying that again and keep using the service from iFramely.
-
@teh_g Hmm... we didn't actually write the IFramely plugin. Most of the ongoing maintenance is handled by the iframely team themselves, so I'd raise the issue with them to see what the problem is.
It could just be that as you have more content, you have more embeds, and those get loaded by your visitors?
-
@julian said in [nodebb-plugin-iframely] IFramely Embeds:
@teh_g Hmm... we didn't actually write the IFramely plugin. Most of the ongoing maintenance is handled by the iframely team themselves, so I'd raise the issue with them to see what the problem is.
When I asked them about it, they said that most of the hits were from API calls than from iFrames being loaded. This indicated to them that NodeBB is making most of the hits. Here is what they said as well:
The way NodeBB folks have it implemented is that they keep the cache for 1 day in server's memory. If you restart the servers, for example, - the cache will be reset. Also, if your RAM is limited, some of the records can be evicted from cache prematurely.
@julian said in [nodebb-plugin-iframely] IFramely Embeds:
It could just be that as you have more content, you have more embeds, and those get loaded by your visitors?
That could be a part of it, but that would be a big reason for wanting to age out older embeds. The more we can control for that, the better too. Beyond just age based, maybe add additional parameters, like "any age for first post", "only age out certain content", etc.
-
https://github.com/NodeBB/nodebb-plugin-iframely/blob/master/library.js#L21-L23
The cache is indeed one day, though it can be changed to be a week, perhaps, if that is better in your opinion.
However, if indeed your NodeBB is crashing often, then the cache is cleared, and it won't matter how far in the future the cache expiry is, since everything in it will be flushed and content pulled anew.
So I have to ask, is your forum crashing often?
-
@julian said in [nodebb-plugin-iframely] IFramely Embeds:
https://github.com/NodeBB/nodebb-plugin-iframely/blob/master/library.js#L21-L23
The cache is indeed one day, though it can be changed to be a week, perhaps, if that is better in your opinion.
However, if indeed your NodeBB is crashing often, then the cache is cleared, and it won't matter how far in the future the cache expiry is, since everything in it will be flushed and content pulled anew.
So I have to ask, is your forum crashing often?
Luckily my forum doesn't crash
. The only time it gets restarted is if there is a plugin update that needs a restart, or for new NodeBB versions.I'd love to have the option to change the cache in the admin panel. Expanding it to a week would cut down on a lot of API calls. It would also be interesting to see the impact on not caching the expansions at all...
A bigger change / enhancement would be to be able to configure post age for expansion. Basically say if a post is older than x days, or hasn't been updated in x days, don't expand the link at all (even when caching).
Edit: And at least for now, I might manually change that plugin to cache for a week. Would that cause problems when I update the plugin again some day, or will the change just be overwritten?
-
@teh_g said in [nodebb-plugin-iframely] IFramely Embeds:
Edit: And at least for now, I might manually change that plugin to cache for a week. Would that cause problems when I update the plugin again some day, or will the change just be overwritten?
Go ahead, when the change is (eventually) made, it'll conflict, but I presume you know enough about git to realise what's going on and fix it
Alternatively, just
git checkout .before pulling.... oh, but if you're not using git, but installing via npm, then installing a new version of the plugin will just clobber your changes, so that may be what you want anyway.
-
@julian said in [nodebb-plugin-iframely] IFramely Embeds:
Go ahead, when the change is (eventually) made, it'll conflict, but I presume you know enough about git to realise what's going on and fix it
Alternatively, just
git checkout .before pulling.... oh, but if you're not using git, but installing via npm, then installing a new version of the plugin will just clobber your changes, so that may be what you want anyway.
I do use npm. Assuming the change I want goes in, I am totally fine with it being clobbered
I am going to work with the Iframely team to see if the cache change does lower API hits. Going from one day to seven could significantly lower my API hits, which would be amazing.
-
-
@torn2 said in [nodebb-plugin-iframely] IFramely Embeds:
@teh_g We self host though...
If Iframely is self-hosted, I suspect it would be something in the config file of Iframely. You would want to set it to force smart frames.
You might be able to open a support ticket with the Iframely folk, they may know better.
-
Would someone mind posting their iframely config file as an example? I'm trying to self-host iframely on my server and I've been having difficulty configuring the file, and perhaps putting it in the correct location.
My NodeBB is in my home folder and I've got it set up with a typical Nginx proxy.
-
No, IFramely works on remote links, so if it's a local link, it might not work... or at the very least, IFramely wouldn't know how to turn that into a rich embed.
I think at present uploads just show up as a link for download or opening, right?
-
Testing this out for us now.
-
@Hawthorn said in [nodebb-plugin-iframely] IFramely Embeds:
Would someone mind posting their iframely config file as an example? I'm trying to self-host iframely on my server and I've been having difficulty configuring the file, and perhaps putting it in the correct location.
My NodeBB is in my home folder and I've got it set up with a typical Nginx proxy.
This is ours:
(function() { var config = { // Specify a path for custom plugins. Custom plugins will override core plugins. // CUSTOM_PLUGINS_PATH: __dirname + '/yourcustom-plugin-folder', DEBUG: false, RICH_LOG_ENABLED: false, // For embeds that require render, baseAppUrl will be used as the host. //@crazycells Enter host or IP here; baseAppUrl: "HOST OR IP", relativeStaticUrl: "/r", // Or just skip built-in renders altogether SKIP_IFRAMELY_RENDERS: true, // For legacy reasons the response format of Iframely open-source is // different by default as it does not group the links array by rel. // In order to get the same grouped response as in Cloud API, // add `&group=true` to your request to change response per request // or set `GROUP_LINKS` in your config to `true` for a global change. GROUP_LINKS: true, // Number of maximum redirects to follow before aborting the page // request with `redirect loop` error. MAX_REDIRECTS: 4, SKIP_OEMBED_RE_LIST: [ // /^https?:\/\/yourdomain\.com\//, ], /* // Used to pass parameters to the generate functions when creating HTML elements // disableSizeWrapper: Don't wrap element (iframe, video, etc) in a positioned div GENERATE_LINK_PARAMS: { disableSizeWrapper: true }, */ port: 8061, //can be overridden by PORT env var host: '0.0.0.0', // Dockers beware. See https://github.com/itteco/iframely/issues/132#issuecomment-242991246 //can be overridden by HOST env var // Optional SSL cert, if you serve under HTTPS. /* ssl: { key: require('fs').readFileSync(__dirname + '/key.pem'), cert: require('fs').readFileSync(__dirname + '/cert.pem'), port: 443 }, */ /* Supported cache engines: - no-cache - no caching will be used. - node-cache - good for debug, node memory will be used (https://github.com/tcs-de/nodecache). - redis - https://github.com/mranney/node_redis. - memcached - https://github.com/3rd-Eden/node-memcached */ CACHE_ENGINE: 'node-cache', CACHE_TTL: 0, // In milliseconds. // 0 = 'never expire' for memcached & node-cache to let cache engine decide itself when to evict the record // 0 = 'no cache' for redis. Use high enough (e.g. 365*24*60*60*1000) ttl for similar 'never expire' approach instead /* // Redis cache options. REDIS_OPTIONS: { host: '127.0.0.1', port: 6379 }, */ /* // Memcached options. See https://github.com/3rd-Eden/node-memcached#server-locations MEMCACHED_OPTIONS: { locations: "127.0.0.1:11211" } */ /* // Access-Control-Allow-Origin list. allowedOrigins: [ "*", "http://another_domain.com" ], */ /* // Uncomment to enable plugin testing framework. tests: { mongodb: 'mongodb://localhost:27017/iframely-tests', single_test_timeout: 10 * 1000, plugin_test_period: 2 * 60 * 60 * 1000, relaunch_script_period: 5 * 60 * 1000 }, */ // If there's no response from remote server, the timeout will occur after RESPONSE_TIMEOUT: 5 * 1000, //ms // Customize API calls to oembed endpoints. ADD_OEMBED_PARAMS: [{ // Endpoint url regexp array. re: [/^http:\/\/api\.instagram\.com\/oembed/], // Custom get params object. params: { hidecaption: true } }, { re: [/^https:\/\/www\.facebook\.com\/plugins\/page\/oembed\.json/i], params: { show_posts: 0, show_facepile: 0, maxwidth: 600 } }, { // match i=user or i=moment or i=timeline to configure these types invidually // see params spec at https://dev.twitter.com/web/embedded-timelines/oembed re: [/^https?:\/\/publish\.twitter\.com\/oembed\?i=user/i], params: { limit: 1, maxwidth: 600 } /* }, { // Facebook https://developers.facebook.com/docs/plugins/oembed-endpoints re: [/^https:\/\/www\.facebook\.com\/plugins\/\w+\/oembed\.json/i], params: { // Skip script tag and fb-root div. omitscript: true } */ }], /* // Configure use of HTTP proxies as needed PROXY: [{ re: [/^https?:\/\/www\.domain\.com/], proxy_server: 'http://1.2.3.4:8080', user_agent: 'CHANGE YOUR AGENT', headers: { // HTTP headers // Overrides previous params if overlapped. }, request_options: { // Refer to: https://github.com/request/request // Overrides previous params if overlapped. } }], */ // Customize API calls to 3rd parties. At the very least - configure required keys. providerOptions: { locale: "en_US", // ISO 639-1 two-letter language code, e.g. en_CA or fr_CH. // Will be added as highest priotity in accept-language header with each request. // Plus is used in FB, YouTube and perhaps other plugins "twitter": { "max-width": 550, "min-width": 250, hide_media: false, hide_thread: false, omit_script: false, center: false, // dnt: true, cache_ttl: 100 * 365 * 24 * 3600 // 100 Years. }, readability: { enabled: false // allowPTagDescription: true // to enable description fallback to first paragraph }, images: { loadSize: false, // if true, will try an load first bytes of all images to get/confirm the sizes checkFavicon: false // if true, will verify all favicons }, tumblr: { consumer_key: "INSERT YOUR VALUE" // media_only: true // disables status embeds for images and videos - will return plain media }, google: { // https://developers.google.com/maps/documentation/embed/guide#api_key maps_key: "INSERT YOUR VALUE" }, /* // Optional Camo Proxy to wrap all images: https://github.com/atmos/camo camoProxy: { camo_proxy_key: "INSERT YOUR VALUE", camo_proxy_host: "INSERT YOUR VALUE" // ssl_only: true // will only proxy non-ssl images }, */ // List of query parameters to add to YouTube and Vimeo frames // Start it with leading "?". Or omit alltogether for default values // API key is optional, youtube will work without it too. // It is probably the same API key you use for Google Maps. youtube: { // api_key: "INSERT YOUR VALUE", get_params: "?rel=0&showinfo=1" // https://developers.google.com/youtube/player_parameters }, vimeo: { get_params: "?byline=0&badge=0" // https://developer.vimeo.com/player/embedding }, /* soundcloud: { old_player: true // enables classic player }, giphy: { media_only: true // disables branded player for gifs and returns just the image } */ /* bandcamp: { get_params: '/size=large/bgcol=333333/linkcol=ffffff/artwork=small/transparent=true/', media: { album: { height: 472, 'max-width': 700 }, track: { height: 120, 'max-width': 700 } } } */ }, // WHITELIST_WILDCARD, if present, will be added to whitelist as record for top level domain: "*" // with it, you can define what parsers do when they run accross unknown publisher. // If absent or empty, all generic media parsers will be disabled except for known domains // More about format: https://iframely.com/docs/qa-format /* WHITELIST_WILDCARD: { "twitter": { "player": "allow", "photo": "deny" }, "oembed": { "video": "allow", "photo": "allow", "rich": "deny", "link": "deny" }, "og": { "video": ["allow", "ssl", "responsive"] }, "iframely": { "survey": "allow", "reader": "allow", "player": "allow", "image": "allow" }, "html-meta": { "video": ["allow", "responsive"], "promo": "allow" } } */ }; module.exports = config; })();
