No shit, if you train a stochastic parrot on biased data, you get biased output?

[xgranade@wandering.shop]

No shit, if you train a stochastic parrot on biased data, you get biased output? Say it ain't so!

[xgranade@wandering.shop]

On a somewhat related note, my efforts to help improve Wikipedia's coverage of LLMs is again stymied by the problem that few people write scholarly articles explaining that the moon is not made of green cheese.

The few that do are absolute fucking *heroes*.

[xgranade@wandering.shop]

Said effort is also not helped by the truly absurd amount of spam hyping up AI, and the incredible SEO efforts thrown behind said spam.

If you search for "what is prompt engineering", for instance, not one fact-based source shows up in the first three pages of DDG for me (searching in a private window). It's *all* AI hype by either LLM vendors, startups that want to sell you "libraries" of prompts, or courseware that takes the above credulously.

[aud@fire.asta.lgbt]

@[email protected] I boosted that thing earlier and I thought about commenting on how "fucking duh" and how it's bullshit that it's up to poorly funded socially responsible researchers to point out the harms after they're inflicted that could have been proven to happen just based on theory (and was pointed out a million times)

but, well

[xgranade@wandering.shop]

So many people have pissed in the pool, it's just a toilet now.

[xgranade@wandering.shop]

@aud The whole inverted burden of proof thing just gets to me so badly. It's not our job to prove that something causes harm, it's their job to prove that it doesn't. It's not our job to prove that something doesn't work, it's their job to prove that it does.

[aud@fire.asta.lgbt]

@[email protected] if “software engineering” was actually a fucking engineering discipline… it’s such bullshit.

[aud@fire.asta.lgbt]

@[email protected] I’ll call it engineering when altman has to go broke paying to undo his harms.

[xgranade@wandering.shop]

I pulled a random arXiv preprint¹ off a Google Scholar search for prompt engineering, and this is the size of the datasets² that they're using for all their conclusions.

The quality of evidence in that field is almost nonexistent. They're reporting on deltas of less than 1% based on a sampling procedure that can at *best* give 3% margins of error.

¹accepted at a major conference, using a preprint to avoid paywalls
²apologies for shitty alt-text, getting alt-text of tables is tricky

https://stockroom.wandering.shop/media_attachments/files/113/409/149/155/302/516/original/a72f8b2770756503.png

[aud@fire.asta.lgbt]

@[email protected] cool. this is one of those times I'm feeling salty that I let that whole "evidence" and "statistically proving or disproving the hypothesis and getting enough data to confidently say anything" get in the way of the publication needed to finish my phd.

[xgranade@wandering.shop]

@aud My whole thesis was basically on how to rigorously evaluate the quality of control in quantum systems... I hear you.

[aud@fire.asta.lgbt]

@[email protected] I had to spend six months reworking our statistical analysis pipeline because things just weren't agreeing and I still couldn't get convergence across the 900-odd simulations I was doing for the fixed set of parameters.

and these people just... they just... ... "prompt engineering". I.

I.

[aud@fire.asta.lgbt]

@[email protected] (also: I am so sorry that they keep trying to make quantum the next snake oil)

[xgranade@wandering.shop]

@aud Yuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuup.

[aud@fire.asta.lgbt]

@[email protected] (fun fact on that one: thanks to github's total fucking inability to let you do name changes, I no longer have credit for that on my github account. You'd have to know my deadname and my prior account)

[xgranade@wandering.shop]

@aud God that sucks, I'm sorry. There's too many places that my deadname appears in the academic record for me to possibly correct at this point, sadly. The fun of transitioning very late in life and *after* becoming famous in a tiny, insular field.

[aud@fire.asta.lgbt]

@[email protected] I was lucky enough that I switched to publishing under my initials because I sensed some discomfort with my actual name (even though I hated my initials) after the first or second paper, so that's helpful. But GitHub just doesn't allow account name changes, or at least didn't in 2019*.

* actually, the real problem is that you can make a new account and delete the old one and have it link, but if anyone ever registers it again all the credit will revert back. So about two days after I registered the new one and got rid of the old one, a friend helpfully registered the old one and gave it to me. So no one could squat on the account... but also now my account lost all my old contributions, sort of... defeating the purpose. I appreciated the intent though.

[xgranade@wandering.shop]

@aud What a fucking mess. The degree to which technical systems are designed with complete disregard for how names work in practice is absurd.

Cool URIs don't change, but cool names absolutely do.

[aud@fire.asta.lgbt]

@[email protected] RIGHT?!

"tech bros give a shit about anyone except themselves" challenge level: impossible

[xgranade@wandering.shop]

@aud Like, account systems have no business assigning anything other than a fucking UUID to individual accounts, then at *most* using stuff like OAuth claims as evidence of names.