NodeBB Assets - Object Storage

[phenomlab]

@razibal Looks like it, yes. Not sure why though. Bucket name is nodebb-dev2

[razibal]

That's strange, you could do a sync using the aws cli to see if that fixes it

sudo pip install awscli

aws s3 sync build/public  s3://nodebb-dev2/assets --endpoint-url https://[account id].cloudflarestorage.com

[phenomlab]

@razibal shouldn't that be

aws s3 sync build/public  s3://nodebb-dev2/assets --endpoint-url https://[account id].r2.cloudflarestorage.com

[razibal]

yes, sorry typo

[phenomlab]

@razibal Looks promising on script execute

upload: build/public/admin.min.js to s3://nodebb-dev2/assets/admin.min.js
upload: build/public/nodebb.min.js to s3://nodebb-dev2/assets/nodebb.min.js
upload: build/public/plugins/nodebb-plugin-ns-embed/js/acp.js to s3://nodebb-dev2/assets/plugins/nodebb-plugin-ns-embed/js/acp.js
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-900-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-900-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-900-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-900-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-900-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-300-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-100-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-800-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-200-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-400-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-500-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-700-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-600-normal.woff
upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-300-normal.woff

However, the bucket itself remains the same.

[razibal]

shouldn't be possible - as I'm sure you already know S3 buckets/Object storage don't really have 'folders' just object keys with a slash to signify a directory - somehow, in your case the keys are getting translated to the file name

[phenomlab]

@razibal created a new bucket and tried again from CLI. It looks fine from here, but everything lands up in /assets/

[razibal]

aws s3api list-objects-v2 --endpoint-url https://[account].r2.cloudflarestorage.com --bucket nodebb-dev3 | more

{
    "Contents": [
        {
            "Key": "assets/acp.min.js.map",
            "LastModified": "2023-05-17T14:37:46.508Z",
            "ETag": "\"30d96ca6de60c70d482689de4f656784\"",
            "Size": 2115263,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "assets/acp.min.js",
            "LastModified": "2023-05-17T14:37:46.182Z",
            "ETag": "\"ac03fc028d895314c458a64783a5d2df\"",
            "Size": 617833,
            "StorageClass": "STANDARD"
        },

[phenomlab]

@razibal

{
    "Contents": [
        {
            "Key": "assets/03371bf1d5cbb1eab58e3a0130d1e9c2.js",
            "LastModified": "2023-05-17T17:37:30.453Z",
            "ETag": "\"733f46933105eaf011a37d00a964aa39\"",
            "Size": 502502,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "assets/10822.4553fc0a94f5e4017c9e.min.js",
            "LastModified": "2023-05-17T17:37:29.881Z",
            "ETag": "\"e825b3003d09c6acaef0960484c1cd43\"",
            "Size": 742,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "assets/10943.83723e39cc9dae85e2e7.min.js",
            "LastModified": "2023-05-17T17:37:29.880Z",
            "ETag": "\"a14d728a69056d99f40de4e9d009c14f\"",
            "Size": 1149,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "assets/11307.fd4abf5601dce6fcfaf8.min.js",
            "LastModified": "2023-05-17T17:37:30.211Z",
            "ETag": "\"b81d4a241111819396108c44b1932acc\"",
            "Size": 327,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "assets/11393.036b406d0bb56dc3b1c4.min.js",

A lot more output here, but I think you get the picture.

It looks like everything is being minified into single files?

[razibal]

just copy a single file as a test

aws s3 cp ./package.json s3://nodebb-dev3/assets/folder1/folder2/package.json --endpoint-url https://[account].r2.cloudflarestorage.com
upload: ./package.json to s3://nodebb-static/assets/folder1/folder2/package.json

aws s3 ls s3://nodebb-dev3/assets/folder1/folder2/package.json --endpoint-url https://[account].r2.cloudflarestorage.co
m
2023-05-17 12:48:58       7920 package.json

[phenomlab]

@razibal files are being taken from /build/public/ - is that correct ? Shouldn't it just be /public/ ??

[razibal]

correct

await sync('./build/public', 's3://nodebb-static/assets', {...

[phenomlab]

@razibal But that is where the minified files are being taken from - and it matches the directory structure.

EDIT - I changed it to just /public/ and it looks much better

[razibal]

You know the nodebb file layout better than I do, all folders that serve static assets would need to be synched to the bucket

[phenomlab]

@razibal Yes, I think this is the disconnect. I'm now seeing folders and files.

[phenomlab]

@phenomlab Hmm - perhaps not. I think you still need ./build/public/ which I am pulling back in now.

EDIT - seems the files are indeed there

However, there is an issue with pulling the files on request.

[razibal]

so, the script can be modified as

async function syncStaticFiles() {
const folders = ['./build/public', './public']
    await Promise.all(folders.map(async folder => {
    await sync(folder, 's3://nodebb-static/assets', {
        monitor,
        maxConcurrentTransfers: 1000,
        commandInput: {
            ACL: 'public-read',
            ContentType: (syncCommandInput) => mime.lookup(syncCommandInput.Key) || 'text/html'
        }
    });
    }));
    process.exit()
}

[razibal]

If you sync using the command line, there will be issues as the content-type won't be set. Your last sync for build/pubic was with the command line

[phenomlab]

@razibal Right.... I think I have it working now with those modifications Running some tests, but all looking good. Will test overnight...

This has a lot of potential using workers and R2, and really needs proper documentation. As you know, it's taken me 4 hours to get this working!

When I run /nodebb build I don't see anything else running afterwards - should it be calling that other script automatically?

[phenomlab]

@baris @julian I think this should be made into a plugin if I'm honest. Obviously, there are other steps that need to be completed outside of just installing the plugin, but this works very well, and it's extremely fast in my testing within dev.