NodeBB Assets - Object Storage
-
@razibal Looks promising on script execute
upload: build/public/admin.min.js to s3://nodebb-dev2/assets/admin.min.js upload: build/public/nodebb.min.js to s3://nodebb-dev2/assets/nodebb.min.js upload: build/public/plugins/nodebb-plugin-ns-embed/js/acp.js to s3://nodebb-dev2/assets/plugins/nodebb-plugin-ns-embed/js/acp.js upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-900-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-900-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-cyrillic-ext-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-900-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-greek-ext-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-greek-ext-900-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-900-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-900-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-300-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-100-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-100-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-latin-ext-800-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-latin-ext-800-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-200-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-200-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-400-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-400-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-500-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-500-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-700-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-700-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-600-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-600-normal.woff upload: build/public/plugins/nodebb-theme-harmony/inter/inter-vietnamese-300-normal.woff to s3://nodebb-dev2/assets/plugins/nodebb-theme-harmony/inter/inter-vietnamese-300-normal.woff
However, the bucket itself remains the same.
-
aws s3api list-objects-v2 --endpoint-url https://[account].r2.cloudflarestorage.com --bucket nodebb-dev3 | more
{ "Contents": [ { "Key": "assets/acp.min.js.map", "LastModified": "2023-05-17T14:37:46.508Z", "ETag": "\"30d96ca6de60c70d482689de4f656784\"", "Size": 2115263, "StorageClass": "STANDARD" }, { "Key": "assets/acp.min.js", "LastModified": "2023-05-17T14:37:46.182Z", "ETag": "\"ac03fc028d895314c458a64783a5d2df\"", "Size": 617833, "StorageClass": "STANDARD" },
-
{ "Contents": [ { "Key": "assets/03371bf1d5cbb1eab58e3a0130d1e9c2.js", "LastModified": "2023-05-17T17:37:30.453Z", "ETag": "\"733f46933105eaf011a37d00a964aa39\"", "Size": 502502, "StorageClass": "STANDARD" }, { "Key": "assets/10822.4553fc0a94f5e4017c9e.min.js", "LastModified": "2023-05-17T17:37:29.881Z", "ETag": "\"e825b3003d09c6acaef0960484c1cd43\"", "Size": 742, "StorageClass": "STANDARD" }, { "Key": "assets/10943.83723e39cc9dae85e2e7.min.js", "LastModified": "2023-05-17T17:37:29.880Z", "ETag": "\"a14d728a69056d99f40de4e9d009c14f\"", "Size": 1149, "StorageClass": "STANDARD" }, { "Key": "assets/11307.fd4abf5601dce6fcfaf8.min.js", "LastModified": "2023-05-17T17:37:30.211Z", "ETag": "\"b81d4a241111819396108c44b1932acc\"", "Size": 327, "StorageClass": "STANDARD" }, { "Key": "assets/11393.036b406d0bb56dc3b1c4.min.js",
A lot more output here, but I think you get the picture.
It looks like everything is being minified into single files?
-
just copy a single file as a test
aws s3 cp ./package.json s3://nodebb-dev3/assets/folder1/folder2/package.json --endpoint-url https://[account].r2.cloudflarestorage.com upload: ./package.json to s3://nodebb-static/assets/folder1/folder2/package.json aws s3 ls s3://nodebb-dev3/assets/folder1/folder2/package.json --endpoint-url https://[account].r2.cloudflarestorage.co m 2023-05-17 12:48:58 7920 package.json
-
@phenomlab Hmm - perhaps not. I think you still need
./build/public/
which I am pulling back in now.EDIT - seems the files are indeed there
However, there is an issue with pulling the files on request.
-
so, the script can be modified as
async function syncStaticFiles() { const folders = ['./build/public', './public'] await Promise.all(folders.map(async folder => { await sync(folder, 's3://nodebb-static/assets', { monitor, maxConcurrentTransfers: 1000, commandInput: { ACL: 'public-read', ContentType: (syncCommandInput) => mime.lookup(syncCommandInput.Key) || 'text/html' } }); })); process.exit() }
-
@razibal Right.... I think I have it working now with those modifications Running some tests, but all looking good. Will test overnight...
This has a lot of potential using workers and R2, and really needs proper documentation. As you know, it's taken me 4 hours to get this working!
When I run
/nodebb build
I don't see anything else running afterwards - should it be calling that other script automatically? -
-
@phenomlab said in NodeBB Assets - Object Storage:
This has a lot of potential using workers and R2, and really needs proper documentation. As you know, it's taken me 4 hours to get this working!
Yeah, it only took me a few minutes to get it working as a POC this morning, but , as you experienced, getting that even partially operationalized took a lot longer.
The nice thing about implementing this is that it makes nodeBB essentially infinitely scalable. All static assets are handled by cloudflare workers while the nodeBB API can be deployed to droplets behind a load balancer. Spin up droplets as demand increases and then spin them back down when no longer needed.