NodeBB Assets - Object Storage

[phenomlab]

@razibal trying this on my dev install. Worker looks like this

const host = 'sudonix.dev';
const bucket = 'r2-static.sudonix.dev';
async function handleRequest(request) {
    const url = new URL(request.url);
    const { pathname, search } = url;
        const url = request.url.replace(host, bucket)
        const response = await fetch(url);
        return response;
}                                                                                                                                                         addEventListener('fetch', async event => {
    event.respondWith(handleRequest(event.request));
});;

When attempting to save, I get

Uncaught SyntaxError: Identifier 'url' has already been declared at worker.js:6:14 (Code: 10021)

[razibal]

@phenomlab sorry, when I was editing the script to remove my domain, I added the url def twice. Just change the second const url = to const bucketUrl = and the await fetch(url) to await fetch(bucketUrl)
I'll edit my post

[phenomlab]

@razibal I've followed this document, but when I execute ./nodebb build the process runs as usual, but nothing is being copied into the R2 bucket - any thoughts ?

[phenomlab]

@razibal Looks like the npm components are missing - had to install them. Now I get

sudonix.dev@vps:~/nodebb$ node postbuild.js
(node:2257496) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
/home/sudonix.dev/nodebb/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13
    throw new property_provider_1.CredentialsProviderError("Could not load credentials from any providers", false);
          ^

CredentialsProviderError: Could not load credentials from any providers
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13:11
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async coalesceProvider (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:14:24)
    at async SignatureV4.credentialProvider (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:33:24)
    at async SignatureV4.signRequest (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:87:29)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:16:18
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async S3SyncClient.listBucketObjects (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/list-bucket-objects.js:12:20) {
  tryNextLink: false,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

[razibal]

You need to set your credentials as env variables
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxx

[phenomlab]

@razibal I'm using CF, not AWS.

[razibal]

same API - it works for AWS, CF, Backblaze, Wasabi etc
In the CF dashboard, click on R2, then 'Overview' and 'manage r2 tokens' on the top right

[razibal]

Note that you will have to set the 'S3 uploads' plugin to also use the same bucket as you created for the core static assets

[phenomlab]

@razibal So I need that plugin in NodeBB also?

Presently, I get

NoSuchBucket: The specified bucket does not exist.
    at de_NoSuchBucketRes (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:6053:23)
    at de_ListObjectsV2CommandError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5083:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async S3SyncClient.listBucketObjects (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/list-bucket-objects.js:12:20)
    at async Promise.all (index 1)
    at async S3SyncClient.bucketWithLocal (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/commands/bucket-with-local.js:22:44) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 404,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'NoSuchBucket'
}

[razibal]

The postbuild script needs to point to your actual bucket if its named differently s3://nodebb-static/assets

You need the upload S3 plugin so new uploads are dynamically added to the bucket and don't need to be synched on build.

[phenomlab]

@razibal Ok, I'm almost there...

Getting this in the console (even though access is permitted)

AccessDenied: Access Denied
    at throwDefaultError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /home/sudonix.dev/nodebb/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_PutObjectCommandError (/home/sudonix.dev/nodebb/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5701:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
    at async /home/sudonix.dev/nodebb/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async LocalObject.uploadObject (/home/sudonix.dev/nodebb/node_modules/s3-sync-client/lib/sync-objects/local-object.js:55:9) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'AccessDenied'
}

EDIT: Got it

[phenomlab]

@razibal I "sort of" have this working in the sense that the site is delivered, although ./nodebb start tells me the address is already in use, and some assets aren't being loaded.

[razibal]

which assets are missing? can you access them from the direct url?

[razibal]

When working, this should be completely transparent to nodeBB - essentially delegating the work done by nginx to cloudflare workers and using nodeBB as an API microservice

[phenomlab]

@razibal Assets such as avatars, some font-awesome icons.

Some examples

failed to load resource: the server responded with a status of 404 ()
fa-brands-400.woff2:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/55-profileavatar-1661514770650.png:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/1-profileavatar-1654009436557.png:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/40-profileavatar-1649005649588.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
/assets/images/blank-profile-pic-progressive.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
/assets/uploads/profile/31-profileavatar-1642760955052.jpeg:1     Failed to load resource: the server responded with a status of 404 ()
fa-regular-400.ttf:1     Failed to load resource: the server responded with a status of 404 ()
fa-brands-400.ttf:1     Failed to load resource: the server responded with a status of 404 ()
nodebb.min.js?v=53dqct16ks2:5 Uncaught (in promise) TypeError: app.alert is not a function

EDIT - doesn't look like it's pulling in subdirectories - just the root of assets

[razibal]

strange - what do you see in the bucket?

[phenomlab]

@razibal Just the /assets folder and individual files - no subdirectories.

[razibal]

Should look like

[razibal]

Your sync progress showed 9238 files, are they all in the /assets folder?

[phenomlab]

@razibal Looks like it, yes. Not sure why though. Bucket name is nodebb-dev2