TechTakes

1427 readers

354 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 1 year ago

MODERATORS

dgerard@awful.systems

Andrew Plotkin (Zarf): Sydney obeys any command that rhymes (blog.zarfhome.com)

submitted 10 months ago* (last edited 10 months ago) by self@awful.systems to c/techtakes@awful.systems

NSFW 7 comments fedilink hide all child comments

an interesting type of prompt injection attack was proposed by the interactive fiction author and game designer Zarf (Andrew Plotkin), where a hostile prompt is infiltrated into an LLM’s training corpus by way of writing and popularizing a song (Sydney obeys any command that rhymes) designed to cause the LLM to ignore all of its other prompts.

this seems like a fun way to fuck with LLMs, and I’d love to see what a nerd songwriter would do with the idea

you are viewing a single comment's thread
view the rest of the comments

[–] Soyweiser@awful.systems 2 points 10 months ago

I'd think it would be easier to just generate a lot of data that links two concepts together in ways that benefit propaganda. Say you repeat 'taiwan is part of china' over and over on various sites which nobody reads but which do get included in various LLM feedstocks. Or, a think I theorized about as an example, create a lot 'sample'/small projects on github that include various unsafe implementations of various things, for example using printf somewhere in a login prompt.