@UlrikeHahn @locha "Sure" as in "I have complete knowledge about what is in my training dataset". I do not dispute the author's understanding on their fields of expertise.
It has however been my experience that people who should definitely know better may have a somewhat hazy grasp on how a LLM work (or for that matter, how many models behave when trained with huge, imperfectly controlled training datasets).

@UlrikeHahn @locha Hence my point: are we sure that the "non-identical gambles" are not somewhere in the training data (imagine a 1980s book with a large table of experiments and results, with lots of variations)?

@UlrikeHahn @locha I object to this wording, since it assumes an abstract reasoning capacity, which LLMs lack. It may *look like* a generalization, but is not considering the output could essentially be training data + some noise (the noise parameter being responsible for the "close enough" part).

@UlrikeHahn @locha I did, but I fail to see how it answers my concern.

@UlrikeHahn @locha I discuss the very idea that a generalization was made. My first assumption is that something close enough to the "new" items is there on the training dataset, and that the model, as expected, just reproduces what is in the training set.

@UlrikeHahn @locha Strictly speaking, a genAI does not "identify" the right option in any meaningful way. It just predicts that a chain of the order "choose this" is more often associated with some formulation of option A than with some formulation of option B. Boilerplate lecture slides with "people chose this over that", with the two options, provide the required training data.

@UlrikeHahn @locha In my (statistician) opinion, you would get quite correct results if the figures used in the problem the model faced are close enough to these used in Masters' textbooks and exercises. The LLM would consider figures as word, and be able to predict that the options with these figures are associated with a positive weight.

@UlrikeHahn @locha I beg to differ. In my class, I fully described examples of options, and which one people preferred. Looks like to me the kind of structure a LLM could pick up.

@UlrikeHahn @locha I expect detailed descriptions of these tasks abound in textbooks and lecture notes put online (mine included), which very likely are in the training data.