Thread · 7 tweets · 7 Oct 2021

Very interesting thread from @RepCasten on the danger of persuasive technology applied to finance, here with tragic consequences. We're only at the beginning of our journey trying to reckon with digital persuasion and clawing control back. 🧵

There is a lot of good (and ongoing) work on nudging, dark patterns, and defining what is manipulative or deceptive. The part that worries me, though, comes from asymmetries of data, automation, and personalisation. It's inordinately harder to defend against digital nudging.

If your nudge is to put some given food a little farther from reach, I can detect it and compensate. If it's the same for everyone, someone else can detect it and share. And most physical environments aren't totally designed, you retain agency and control inside of them.

Digital worlds are "total" environments, they control every detail. The nudge might be the precise timing of a prompt, the shade and size of a button. It might be a long, subtle funnel taking you there in small steps. It will be informed by billions of interactions.

You don't stand a chance, and it gets incrementally harder to detect manipulative interactions — especially if they differ significantly between people. Individuals can't fight it, and the burden of proof can't come from observation.

I wonder if we need to establish a form of "Interaction Neutrality": the things you interact with can't be smart about you. Personalisation should only be under direct user control; the rest of the UI can be A/B optimised but only in the same way for everyone.

At any rate, @RepCasten's thread is a reminder that a tech company claiming to "democratise" something is a red flag. To democratise something isn't to just give access to it: you also need to increase people's agency. If you're corralling people, it's not democracy.