[nodebb-plugin-s3-uploads] Store your uploads in AWS S3

miksago

@julian yeah, if I fork it, I'd have to republish as a different package on NPM, which isn't something that's ideal.

chenneil

Hello

I have NodeBB V0.5.0 installed, S3 bucket set up and tried to install and activate the nodebb-s3-uploads from the admin panel->Plugins. Below is the error I have got whenever trying to upload a picture:
'''
Error

Upload Error : nodebb-plugin-s3-uploads :: The XML you provided was not well-formed or did not validate against our published schema
'''

Futher more, when trying to config the settings for nodebb-s3-uploads from NodeBB ACP --> Plugins section:
'''
Not Found

You seem to have stumbled upon a page that does not exist. Return to the home page.
'''

Though it installs and activates without any errors, what else can be wrong? Thanks in advance.

a_5mith

Seems Amazon may have changed something perhaps. The plugin is also very out of date. It's only compatible with 0.3.2, however 0.4.0 introduced breaking changes. I'm not sure the original author of this plugin is around any more. If I get the chance, I will have a look and see what the issue could be.

chenneil

All right!! Thank you @a_5mith.

chenneil

Hello @a_5mith have you had a chance to take a look at this? Thanks

wladiston

Is there someone looking for solutions?

<baris>

I think this plugin is maintained here now https://github.com/joe1chen/nodebb-plugin-s3-uploads

louisemcmahon

I have updated this plugin and added a new package to npm https://github.com/LewisMcMahon/nodebb-plugin-s3-uploads

rod

@lewismcmahon

It is nice to see one of the Amazon S3 uploader plugins being actively maintained.

Enhancement Requests:

Modify this plugin to upload images in a path for each individual user (currently all uploads go to the bucket root -- Look at the nodebb-plugin-amazons3 plugin for an example of this)
Create the ability to scan the database for images and then to delete the ones that no longer exist in the database from the S3 bucket (Deleted posts and threads, re-uploaded (changed) images, etc.)

Otherwise, thank you very much for taking over supporting this plugin.

louisemcmahon

@rod Ive added them as issues once i get the menu sorted ill take a look

louisemcmahon

I have pushed out several bug fixes and an update to the admin area today for this plugin i have also taken a look at the remove unused images functionality and as far as i can tell its not going to be possible without risking removing images of other plugins or some custom db querys.

My original plan was just to search through all posts and users grab any profile pics and cover images along with files from posts and then remove any thing that is in the s3 bucket that is not in that list. But if someone added a plugin that added images somewhere else that would remove them. the other option is to search through the DB for any occurrences of the s3 bucket and get the file id from that but i dont believe that's possible with the current NodeBB functionality.

So the only option left would be to write custom query's for both mongo and redis and interface directly with the db. That's something im not entirely sure plugin should be doing and im not even sure that's possible with redis.

As for the individual path per user i dont think that would add any benefit to the plugin as everything has a unique id.

sohpingting

Hi @lewismcmahon, I am using your plugin:

GitHub - LouiseMcMahon/nodebb-plugin-s3-uploads: A plugin for NodeBB to take file uploads and store them on S3

A plugin for NodeBB to take file uploads and store them on S3 - LouiseMcMahon/nodebb-plugin-s3-uploads

GitHub (github.com)

It was working fine until this morning. We are unable to upload any files.
FYI, we just upgraded to NodeBB v1.0.3.

sohpingting

Hi @lewismcmahon, this is the 3rd day that we are unable to upload files and almost all topic we post require this feature. Please help to fix this soon., thank you .

louisemcmahon

Hi guys if you have issues with my plugins do not post here! make an issue on Github.

Also the git repo for this has changed location but npm is still the same.

GitHub - LouiseMcMahon/nodebb-plugin-s3-uploads: A plugin for NodeBB to take file uploads and store them on S3

A plugin for NodeBB to take file uploads and store them on S3 - LouiseMcMahon/nodebb-plugin-s3-uploads

GitHub (github.com)

I will be looking into the issues with 1.1 today.

louisemcmahon

I am super busy at the moment with other work if anyone has found any bugs with the plugin or is willing to look at the outstanding issues on GitHub pull requests would be hugely appreciated.

The plugin itself could do with a total rewrite really as it has been added onto and patched up by so many people over the years. I seem to be the 3rd maintainer I can tell. Which I may get round to if I can put some time aside, this would also help solve some of the longstanding issues with file deletion etc along with adding tests something that is sorely lacking.

Also thanks to @julian and everyone else who has submitted PR's while I have been maintaining the plugin.

<baris>

Maybe you can transfer it to https://github.com/NodeBB-Community and join that group.

julian

Definitely second the suggestion... hopefully we can get the original plugin author to add some of them as publishers in the npm repo too

louisemcmahon

@baris @julian do you know if any of them have usernames on here? or I'll try and email them via their GitHub address's.

julian

@louisemcmahon Looks like it's owned by https://github.com/joe1chen...

teh_g

Sorry for resurrecting an ancient thread...

I was trying to configure this with an S3 compatible object storage. I plugged in my credentials and did the URl, etc properly (I think), but after restart NodeBB it caused multiple errors. My guess is that it has to do with nginx serving static assets, but I am not sure.

Here is a snip of the errors:

24/8 21:02:14 [1406] - error: /general/dashboard
 Error: Failed to lookup view "admin/header" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/admin.js:93:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3838:9
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:421:16
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:928:24)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3835:13
    at Command.callback (/var/www/nodebb/forums/src/meta/configs.js:43:3)
    at normal_reply (/var/www/nodebb/forums/node_modules/redis/index.js:721:21)
    at RedisClient.return_reply (/var/www/nodebb/forums/node_modules/redis/index.js:819:9)
    at JavascriptRedisParser.returnReply (/var/www/nodebb/forums/node_modules/redis/index.js:192:18)
    at JavascriptRedisParser.execute (/var/www/nodebb/forums/node_modules/redis-parser/lib/parser.js:574:12)
    at Socket.<anonymous> (/var/www/nodebb/forums/node_modules/redis/index.js:274:27)
Error: Failed to lookup view "footer" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/header.js:189:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at fireFilterHook (/var/www/nodebb/forums/src/plugins/hooks.js:112:11)
    at Object.Plugins.fireHook (/var/www/nodebb/forums/src/plugins/hooks.js:96:4)
    at /var/www/nodebb/forums/src/middleware/header.js:182:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at Object.waterfall (/var/www/nodebb/forums/node_modules/async/dist/async.js:5283:5)
    at Object.middleware.renderFooter (/var/www/nodebb/forums/src/middleware/header.js:180:9)
    at renderHeaderFooter (/var/www/nodebb/forums/src/middleware/render.js:112:22)
    at footer (/var/www/nodebb/forums/src/middleware/render.js:85:8)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:931:17)
24/8 21:02:25 [1406] - error: /unread
 Error: Failed to lookup view "unread" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at ServerResponse.render (/var/www/nodebb/forums/node_modules/express/lib/response.js:966:7)
    at content (/var/www/nodebb/forums/src/middleware/render.js:82:15)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:950:9
    at eachOfLimit (/var/www/nodebb/forums/node_modules/async/dist/async.js:975:24)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:980:16
    at eachOf (/var/www/nodebb/forums/node_modules/async/dist/async.js:1051:5)
    at _parallel (/var/www/nodebb/forums/node_modules/async/dist/async.js:3829:5)
    at Object.parallelLimit [as parallel] (/var/www/nodebb/forums/node_modules/async/dist/async.js:3912:5)
    at async.waterfall.str (/var/www/nodebb/forums/src/middleware/render.js:77:12)
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:421:16
Error: Failed to lookup view "footer" in views directory "/var/www/nodebb/forums/build/public/templates"
    at EventEmitter.render (/var/www/nodebb/forums/node_modules/express/lib/application.js:580:17)
    at /var/www/nodebb/forums/src/middleware/header.js:189:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at next (/var/www/nodebb/forums/node_modules/async/dist/async.js:5280:9)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:906:16
    at fireFilterHook (/var/www/nodebb/forums/src/plugins/hooks.js:112:11)
    at Object.Plugins.fireHook (/var/www/nodebb/forums/src/plugins/hooks.js:96:4)
    at /var/www/nodebb/forums/src/middleware/header.js:182:13
    at nextTask (/var/www/nodebb/forums/node_modules/async/dist/async.js:5273:14)
    at Object.waterfall (/var/www/nodebb/forums/node_modules/async/dist/async.js:5283:5)
    at Object.middleware.renderFooter (/var/www/nodebb/forums/src/middleware/header.js:180:9)
    at renderHeaderFooter (/var/www/nodebb/forums/src/middleware/render.js:112:22)
    at footer (/var/www/nodebb/forums/src/middleware/render.js:85:8)
    at /var/www/nodebb/forums/node_modules/async/dist/async.js:3830:24
    at replenish (/var/www/nodebb/forums/node_modules/async/dist/async.js:946:17)
    at iterateeCallback (/var/www/nodebb/forums/node_modules/async/dist/async.js:931:17)

I also noted the following in the log when running ./nodebb build:

24/8 21:04:02 [1470] - warn: [plugins/nodebb-plugin-s3-uploads-updated] Mapped path 's3 => /var/www/nodebb/forums/node_modules/nodebb-plugin-s3-uploads-updated/public' not found.