Technology

60036 readers

2893 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 2 years ago

MODERATORS

What Do Neural Networks Really Learn? Exploring the Brain of an AI Model (youtu.be)

submitted 6 months ago by UraniumBlazer@lemm.ee to c/technology@lemmy.world

3 comments fedilink hide all child comments

Neural networks have become increasingly impressive in recent years, but there's a big catch: we don't really know what they are doing. We give them data and ways to get feedback, and somehow, they learn all kinds of tasks. It would be really useful, especially for safety purposes, to understand what they have learned and how they work after they've been trained. The ultimate goal is not only to understand in broad strokes what they're doing but to precisely reverse engineer the algorithms encoded in their parameters. This is the ambitious goal of mechanistic interpretability. As an introduction to this field, we show how researchers have been able to partly reverse-engineer how InceptionV1, a convolutional neural network, recognizes images.

you are viewing a single comment's thread
view the rest of the comments

[–] GamingChairModel@lemmy.world 2 points 6 months ago

Yes, but the tokens are more than just a stream of letters, and aren't saved in the form of words. The information itself is organized into conceptual proximity to other concepts (and distinct from the text itself), and weighted in a way consistent with its training.

That's why these models can use analogies and metaphors in a persuasive way, in certain contexts. Mix concepts that the training data has never been shown before, and these LLMs can still output something consistent with those concepts.

Anthropic played around with their own model, emphasizing or deemphasizng particular concepts to observe some unexpected behavior.

And we'd have trouble saying whether a model "knows" something if we don't have a robust definition of when and whether a human brain "knows" something.