Invalid CSRF Token when Uploading File via REST API

[Jeremy Snyder]

I have been using the REST API to write a tool to import data, and have had good success up to the point where I want to upload images to a post, or just upload images in general. I have tried both the /api/admin/upload/file and the /api/post/upload endpoints, but no matter what combination of headers I seem to try, I am getting the following errors on the server:

2022-08-15T12:35:47.558Z [4567/165] - error: POST /api/admin/upload/file?_uid=1
invalid csrf token
...
2022-08-15T12:38:16.394Z [4567/165] - error: POST /api/post/upload?_uid=1
invalid csrf token

I am using the Python Requests library to access the REST API, and my logic is as follows:

• Create a Requests session
• Attach the generated Bearer token to the session
• Access the /api/config endpoint to retrieve the csrf_token
• Attach retrieved csrf_token to the session header for key x-csrf-token
• Post to the /api/admin/upload/file?_uid=1 endpoint, setting the files[] param to a Python file pointer

The attempts to post to the /api/post/upload are similar, but I am also attaching a cid to the post data for a valid category. Either way, I get the same 403 Client Error: Forbidden for url ... error in the response, and the invalid csrf token on the server.

This is what a raw request (output from Python Requests with debugging turned on) looks like when I debug the POST:

send: b'POST /api/post/upload?_uid=1 HTTP/1.1\r\nHost: localhost:4567\r\nUser-Agent: python-requests/2.28.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Bearer 8bab7c95-da3c-44a7-926c-750158b7f6eb\r\nx-csrf-token: crGjXcZT-unxVCdPwBPmIEoFMgGCvyhGuI44\r\nCookie: express.sid=s%3AXhIQf0bWxYFViKKID5OqOobbG5dRAx7W.sgxJnGcULUbcQZnEZJfeFMMK%2FDiX0m8RIEYkAR%2BKgzQ\r\nContent-Length: 254\r\nContent-Type: multipart/form-data; boundary=aef166f35592d3b4beed61b08ccdf152\r\n\r\n'
send: b'--aef166f35592d3b4beed61b08ccdf152\r\nContent-Disposition: form-data; name="cid"\r\n\r\n12\r\n--aef166f35592d3b4beed61b08ccdf152\r\nContent-Disposition: form-data; name="files[]"; filename="1659448063198-foo.txt"\r\n\r\ntest me\n\r\n--aef166f35592d3b4beed61b08ccdf152--\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Cross-Origin-Opener-Policy: same-origin
header: Cross-Origin-Resource-Policy: same-origin
header: X-DNS-Prefetch-Control: off
header: Expect-CT: max-age=0
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=15552000; includeSubDomains
header: X-Download-Options: noopen
header: X-Content-Type-Options: nosniff
header: Origin-Agent-Cluster: ?1
header: X-Permitted-Cross-Domain-Policies: none
header: Referrer-Policy: strict-origin-when-cross-origin
header: X-XSS-Protection: 0
header: X-Powered-By: NodeBB
header: Content-Security-Policy: frame-ancestors 'self'
header: Content-Type: text/plain; charset=utf-8
header: Content-Length: 9
header: ETag: W/"9-PatfYBLj4Um1qTm5zrukoLhNyPU"
header: Set-Cookie: express.sid=s%3AXhIQf0bWxYFViKKID5OqOobbG5dRAx7W.sgxJnGcULUbcQZnEZJfeFMMK%2FDiX0m8RIEYkAR%2BKgzQ; Path=/; HttpOnly; SameSite=Lax
header: Date: Mon, 15 Aug 2022 12:41:43 GMT
header: Connection: keep-alive
header: Keep-Alive: timeout=5

I have replicated the process in the browser with the Dev Tools open, to see what headers I might be missing, but other than a session ID, I am not seeing the difference, but it is clearly working through the web application. I also tried to step through the tests for uploading an image to see if I was missing something, and other than the cookies not completely matching, nothing jumped out at me there.

Any help here is appreciated, as it seems this is last piece I need working for my import process.

[baris]

Maybe you can debug on the server side and check if x-csrf-token is in the correct place. We use the csurf module and you should be able put a breakpoint at https://github.com/expressjs/csurf/blob/master/index.js#L111 in node_modules/csurf/index.js in your nodebb folder. The module looks for the token in a few different places as you can see here https://github.com/expressjs/csurf/blob/master/index.js#L130-L137

[PitaJ]

It looks like you may be missing cookies? You probably need to add a jar to your request session.

Might help to reference the uploads tests: https://github.com/NodeBB/NodeBB/blob/master/test/uploads.js

[Jeremy Snyder]

Thanks for the quick replies.

I tried setting the csrf token in a couple different ways, per @baris suggestion, but didn't get any different results. Thanks for the pointer to the code that reads the tokens in different ways, that was a good thought.

My suspicion is that @PitaJ is on to something with the cookie jar, I did notice that when I looked through the test code before. I haven't had too much time to spend on this, but did attempt to do some basic cookie jar tests with Requests, and I am still getting the same error.

I hope to have more time later this week to work through this further, and will post any updates and relevant code here if I get something working or have more questions.

[julian]

@Jeremy-Snyder It may be helpful to see how we do it in our testing suite

https://github.com/NodeBB/NodeBB/blob/master/test/helpers/index.js#L120

[Jeremy Snyder]

Finally getting back around to testing this, still not having much luck. After digging through a lot of the code, and reading through the unit tests for the /api/post/upload endpoint, a user is logged in first.

I have been trying to do this with only an API token, as this process is loading data from another system. I really don't want to have to store a password in addition to an API token when doing my data load.

I guess my next question is: Is /api/post/upload even usable without logging in with a username/password?