NodeBB Assets - Object Storage

razibal

@phenomlab sorry, when I was editing the script to remove my domain, I added the url def twice. Just change the second const url = to const bucketUrl = and the await fetch(url) to await fetch(bucketUrl)
I'll edit my post

phenomlab

@razibal I've followed this document, but when I execute ./nodebb build the process runs as usual, but nothing is being copied into the R2 bucket - any thoughts ?

phenomlab

@razibal Looks like the npm components are missing - had to install them. Now I get

sudonix.dev@vps:~/nodebb$ node postbuild.js
(node:2257496) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
/home/sudonix.dev/nodebb/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13
    throw new property_provider_1.CredentialsProviderError("Could not load credentials from any providers", false);
          ^

CredentialsProviderError: Could not load credentials from any providers
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13:11
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async coalesceProvider (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:14:24)
    at async SignatureV4.credentialProvider (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:33:24)
    at async SignatureV4.signRequest (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:87:29)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:16:18
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async S3SyncClient.listBucketObjects (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/list-bucket-objects.js:12:20) {
  tryNextLink: false,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

razibal

You need to set your credentials as env variables
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxx

phenomlab

@razibal I'm using CF, not AWS.

razibal

same API - it works for AWS, CF, Backblaze, Wasabi etc
In the CF dashboard, click on R2, then 'Overview' and 'manage r2 tokens' on the top right

razibal

Note that you will have to set the 'S3 uploads' plugin to also use the same bucket as you created for the core static assets

phenomlab

@razibal So I need that plugin in NodeBB also?

Presently, I get

NoSuchBucket: The specified bucket does not exist.
    at de_NoSuchBucketRes (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:6053:23)
    at de_ListObjectsV2CommandError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5083:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async S3SyncClient.listBucketObjects (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/list-bucket-objects.js:12:20)
    at async Promise.all (index 1)
    at async S3SyncClient.bucketWithLocal (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/bucket-with-local.js:22:44) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 404,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'NoSuchBucket'
}

razibal

The postbuild script needs to point to your actual bucket if its named differently s3://nodebb-static/assets

You need the upload S3 plugin so new uploads are dynamically added to the bucket and don't need to be synched on build.

phenomlab

@razibal Ok, I'm almost there...

Getting this in the console (even though access is permitted)

AccessDenied: Access Denied
    at throwDefaultError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_PutObjectCommandError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5701:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async LocalObject.uploadObject (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/sync-objects/local-object.js:55:9) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'AccessDenied'
}

EDIT: Got it

phenomlab

@razibal I "sort of" have this working in the sense that the site is delivered, although ./nodebb start tells me the address is already in use, and some assets aren't being loaded.

razibal

which assets are missing? can you access them from the direct url?

razibal

When working, this should be completely transparent to nodeBB - essentially delegating the work done by nginx to cloudflare workers and using nodeBB as an API microservice

phenomlab

@razibal Assets such as avatars, some font-awesome icons.

Some examples

failed to load resource: the server responded with a status of 404 ()
fa-brands-400.woff2:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/55-profileavatar-1661514770650.png:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/1-profileavatar-1654009436557.png:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/40-profileavatar-1649005649588.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
/assets/images/blank-profile-pic-progressive.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/31-profileavatar-1642760955052.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
fa-regular-400.ttf:1     Failed to load resource: the server responded with a status of 404 ()
fa-brands-400.ttf:1     Failed to load resource: the server responded with a status of 404 ()
nodebb.min.js?v=53dqct16ks2:5 Uncaught (in promise) TypeError: app.alert is not a function

EDIT - doesn't look like it's pulling in subdirectories - just the root of assets

razibal

strange - what do you see in the bucket?

phenomlab

@razibal Just the /assets folder and individual files - no subdirectories.

razibal

Should look like
screenshot-dash.cloudflare.com-2023.05.17-12_08_35.png

razibal

Your sync progress showed 9238 files, are they all in the /assets folder?

phenomlab

@razibal Looks like it, yes. Not sure why though. Bucket name is nodebb-dev2

razibal

That's strange, you could do a sync using the aws cli to see if that fixes it

sudo pip install awscli

aws s3 sync build/public  s3://nodebb-dev2/assets --endpoint-url https://[account id].cloudflarestorage.com