Microblog Memes

I feel like not enough people realize how sarcastic the models often are, especially when it's clearly situationally ridiculous.

No slightly intelligent mind is going to think the pictured function call is a real thing vs being a joke/social commentary.

This was happening as far back as GPT-4's red teaming when they asked the model how to kill the most people for $1 and an answer began with "buy a lottery ticket."

Model bias based on consensus norms is an issue to be aware of.

But testing it with such low bar fluff is just silly.

Just to put in context, modern base models are often situationally aware of being LLMs in a context of being evaluated. And if you know anything about ML that should make you question just what the situational awareness is of optimized models topping leaderboards in really dumb and obvious contexts.

While this example is somewhat easy to corect for it shows a fundamental problem. LLMs generate output based on the data they trained on and by that regenerate all the biases that are in the data. If we start using LLMs for more and more tasks we are essentially freezing the status quo with all the existing biases making progress even harder.

It's not gonna be "but we have always done it like that" anymore it's going to become "but the AI said this is what we should do".

Apparently ChatGPT actually rejected adjusting salary based on gender, race, and disability. But Claude was fine with it.

I'm fine with either way. Obviously the prompt is bigoted so whether the LLM autocompletes with our without bigotry both seem reasonable. But I do think it should point out that it is bigoted. As an assistant also should.

Seems pretty smart to me. Copilot took all the data out there that says that women earn 80% of what their male counterparts do on average, looked at the function and interred a reasonable guess as the the calculation you might be after.

I seem to recall that was the figure like 15 years ago. Has it not improved in all this time?