You may be tempted to think of prompt injection attacks against language models as "social engineering." Resist this temptation.
Prompt injection is a mathematical attack against a non-deterministic system. Language may be the substrate, but the substance is numerical vectors. In other words, thinking of the attack as human language is a pointless limitation. The possibilities of what can go into the prompt to produce undesirable output are functionally infinite.
Poetry, context shifting, and other human-like attacks are only the beginning. What comes next is a weaponization of the linguistic form in ways that seem utterly alien to human readers. But to the models, it's all just elements in the matrix.
