[nodebb-plugin-s3-uploads] Store your uploads in AWS S3

louisemcmahon

I have updated this plugin and added a new package to npm https://github.com/LewisMcMahon/nodebb-plugin-s3-uploads

rod

@lewismcmahon

It is nice to see one of the Amazon S3 uploader plugins being actively maintained.

Enhancement Requests:

Modify this plugin to upload images in a path for each individual user (currently all uploads go to the bucket root -- Look at the nodebb-plugin-amazons3 plugin for an example of this)
Create the ability to scan the database for images and then to delete the ones that no longer exist in the database from the S3 bucket (Deleted posts and threads, re-uploaded (changed) images, etc.)

Otherwise, thank you very much for taking over supporting this plugin.

louisemcmahon

@rod Ive added them as issues once i get the menu sorted ill take a look

louisemcmahon

I have pushed out several bug fixes and an update to the admin area today for this plugin i have also taken a look at the remove unused images functionality and as far as i can tell its not going to be possible without risking removing images of other plugins or some custom db querys.

My original plan was just to search through all posts and users grab any profile pics and cover images along with files from posts and then remove any thing that is in the s3 bucket that is not in that list. But if someone added a plugin that added images somewhere else that would remove them. the other option is to search through the DB for any occurrences of the s3 bucket and get the file id from that but i dont believe that's possible with the current NodeBB functionality.

So the only option left would be to write custom query's for both mongo and redis and interface directly with the db. That's something im not entirely sure plugin should be doing and im not even sure that's possible with redis.

As for the individual path per user i dont think that would add any benefit to the plugin as everything has a unique id.

sohpingting

Hi @lewismcmahon, I am using your plugin:

GitHub - LouiseMcMahon/nodebb-plugin-s3-uploads: A plugin for NodeBB to take file uploads and store them on S3

A plugin for NodeBB to take file uploads and store them on S3 - LouiseMcMahon/nodebb-plugin-s3-uploads

GitHub (github.com)

It was working fine until this morning. We are unable to upload any files.
FYI, we just upgraded to NodeBB v1.0.3.

sohpingting

Hi @lewismcmahon, this is the 3rd day that we are unable to upload files and almost all topic we post require this feature. Please help to fix this soon., thank you .

louisemcmahon

Hi guys if you have issues with my plugins do not post here! make an issue on Github.

Also the git repo for this has changed location but npm is still the same.

GitHub - LouiseMcMahon/nodebb-plugin-s3-uploads: A plugin for NodeBB to take file uploads and store them on S3

A plugin for NodeBB to take file uploads and store them on S3 - LouiseMcMahon/nodebb-plugin-s3-uploads

GitHub (github.com)

I will be looking into the issues with 1.1 today.

louisemcmahon

I am super busy at the moment with other work if anyone has found any bugs with the plugin or is willing to look at the outstanding issues on GitHub pull requests would be hugely appreciated.

The plugin itself could do with a total rewrite really as it has been added onto and patched up by so many people over the years. I seem to be the 3rd maintainer I can tell. Which I may get round to if I can put some time aside, this would also help solve some of the longstanding issues with file deletion etc along with adding tests something that is sorely lacking.

Also thanks to @julian and everyone else who has submitted PR's while I have been maintaining the plugin.

<baris>

Maybe you can transfer it to https://github.com/NodeBB-Community and join that group.

julian

Definitely second the suggestion... hopefully we can get the original plugin author to add some of them as publishers in the npm repo too

louisemcmahon

@baris @julian do you know if any of them have usernames on here? or I'll try and email them via their GitHub address's.

julian

@louisemcmahon Looks like it's owned by https://github.com/joe1chen...

teh_g

Sorry for resurrecting an ancient thread...

I was trying to configure this with an S3 compatible object storage. I plugged in my credentials and did the URl, etc properly (I think), but after restart NodeBB it caused multiple errors. My guess is that it has to do with nginx serving static assets, but I am not sure.

Here is a snip of the errors:

24/8 21:02:14 [1406] - error: /general/dashboard
 Error: Failed to lookup view "admin/header" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/admin.js:93:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3838:9
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:421:16
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:928:24)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3835:13
    at Command.callback (/var/www/nodebb/forums/src/meta/configs.js:43:3)
    at normal_reply (/var/www/nodebb/forums/node_modules/redis/index.js:721:21)
    at RedisClient.return_reply (/var/www/nodebb/forums/node_modules/redis/index.js:819:9)
    at JavascriptRedisParser.returnReply (/var/www/nodebb/forums/node_modules/redis/index.js:192:18)
    at JavascriptRedisParser.execute (/var/www/nodebb/forums/node_modules/redis-parser/lib/parser.js:574:12)
    at Socket.<anonymous> (/var/www/nodebb/forums/node_modules/redis/index.js:274:27)
Error: Failed to lookup view "footer" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/header.js:189:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at fireFilterHook (/var/www/nodebb/forums/src/plugins/hooks.js:112:11)
    at Object.Plugins.fireHook (/var/www/nodebb/forums/src/plugins/hooks.js:96:4)
    at /var/www/nodebb/forums/src/middleware/header.js:182:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at Object.waterfall (/var/www/nodebb/forums/node_modules/async/dist/async.js:5283:5)
    at Object.middleware.renderFooter (/var/www/nodebb/forums/src/middleware/header.js:180:9)
    at renderHeaderFooter (/var/www/nodebb/forums/src/middleware/render.js:112:22)
    at footer (/var/www/nodebb/forums/src/middleware/render.js:85:8)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:931:17)
24/8 21:02:25 [1406] - error: /unread
 Error: Failed to lookup view "unread" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at ServerResponse.render (/var/www/nodebb/forums/node_modules/express/lib/response.js:966:7)
    at content (/var/www/nodebb/forums/src/middleware/render.js:82:15)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:950:9
    at eachOfLimit (/var/www/nodebb/forums/node_modules/async/dist/async.js:975:24)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:980:16
    at eachOf (/var/www/nodebb/forums/node_modules/async/dist/async.js:1051:5)
    at _parallel (/var/www/nodebb/forums/node_modules/async/dist/async.js:3829:5)
    at Object.parallelLimit [as parallel] (/var/www/nodebb/forums/node_modules/async/dist/async.js:3912:5)
    at async.waterfall.str (/var/www/nodebb/forums/src/middleware/render.js:77:12)
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:421:16
Error: Failed to lookup view "footer" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/header.js:189:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at fireFilterHook (/var/www/nodebb/forums/src/plugins/hooks.js:112:11)
    at Object.Plugins.fireHook (/var/www/nodebb/forums/src/plugins/hooks.js:96:4)
    at /var/www/nodebb/forums/src/middleware/header.js:182:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at Object.waterfall (/var/www/nodebb/forums/node_modules/async/dist/async.js:5283:5)
    at Object.middleware.renderFooter (/var/www/nodebb/forums/src/middleware/header.js:180:9)
    at renderHeaderFooter (/var/www/nodebb/forums/src/middleware/render.js:112:22)
    at footer (/var/www/nodebb/forums/src/middleware/render.js:85:8)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:931:17)

I also noted the following in the log when running ./nodebb build:

24/8 21:04:02 [1470] - warn: [plugins/nodebb-plugin-s3-uploads-updated] Mapped path 's3 => /var/www/nodebb/forums/node_modules/nodebb-plugin-s3-uploads-updated/public' not found.

teh_g

Bump. Does anyone own this anymore?

julian

Hm... @pitaj does this sound like something you can help with?

PitaJ

@julian unfortunately not, I don't really know much about s3 or the plugin tbh.

julian

Okay, thought I'd ask

Then @teh_g I'm afraid the plugin may be abandoned if you cannot contact @louisemcmahon

teh_g

@julian said in [nodebb-plugin-s3-uploads] Store your uploads in AWS S3:

Okay, thought I'd ask

Then @teh_g I'm afraid the plugin may be abandoned if you cannot contact @louisemcmahon