AI & SocietySeptember 24, 2026

The Most Valuable Data About You Is What You Would Never Post

A woman kneels in a dark confessional booth facing a priest whose head has been replaced by a glowing smartphone

Social media learned who we wanted other people to see. Search learned what we asked when nobody was watching. A model can learn the explanation we give of ourselves, and it is the first confidant in history that owes us nothing.

There are things I would never post. That I am not sure the company should keep going, and have not said so to the board. That I looked up what my savings would cover if I stopped taking a salary. That I drafted a message at two in the morning and did not send it. That I asked what a word meant in a language I tell people I speak fluently. That I wanted to know whether a number in an offer was an insult before I congratulated the person publicly.

But I might tell a model.

That is not simply a change in where we look for information. It is a change in where some of the most private information about us gets created in the first place.

Borromeo built a box

In 1565, Charles Borromeo, the Archbishop of Milan, ordered that a metal grille be placed between priest and penitent. Before that, confession was heard in the open. The priest sat on a chair somewhere in the church, and the penitent knelt beside him, visible to anyone who happened to be there. Borromeo's box removed the audience and the physical contact, and it is more or less the confessional we still recognize.

The architecture was only half of the design. The other half was already centuries old. The Fourth Lateran Council in 1215 decreed that a priest who revealed what he heard in confession would be deposed from the priesthood and confined to a monastery to do perpetual penance. Modern canon law keeps the principle: a confessor who directly violates the seal incurs automatic excommunication.

The point is the pairing. A space built for disclosure was matched with a penalty for repeating it. The same pattern shows up everywhere this kind of speech is gathered. Physicians, lawyers and therapists all receive information people conceal from their families, and all of them operate under rules that make repeating it professionally fatal and often legally impossible. The duty is not decoration around the disclosure. It is the reason the disclosure happens.

Keep that pairing in mind, because a new confidant has arrived with one half of it.

What social media receives is a performance

Social media built one of the most valuable industries in history around observation. It knows who I follow, what I click, which photographs I stop scrolling for and which version of my life I choose to publish. From those signals, advertising systems try to infer what sits behind them. Perhaps I am moving house. Perhaps I am unhappy at work. Perhaps I just liked the coat.

The system does not know. It watches the behavior and builds a theory.

Even the most personal post remains a performance. There is an audience, and I know who is in it: colleagues, relatives, former employers, people I met once at a conference. What I publish is shaped by what I want those people to believe, what I am willing to defend and what I can live with being remembered.

Social media can know an extraordinary amount about me. Most of it concerns the person I am willing to be in front of other people.

Search receives the question, not the person

Search crossed that boundary long before models arrived. People have always typed things into a search box that they would never post and would not say out loud. The box is not anonymous in any technical sense, particularly when I am logged in, but it is anonymous in the social sense. There is no visible audience, and therefore no immediate risk of embarrassment or judgment. In a 2013 Pew Research Center survey, 55 percent of Americans said people should be able to use the internet anonymously for certain activities, and the reasons they gave included medical conditions and subjects they considered socially unacceptable.

So search has long received a more private person than social media did. It received that person in fragments.

Three queries about redundancy terms, a notice period and how to tell a team something can belong to one decision, taken over one week, by one person who has not told anyone yet. Each arrives alone, already compressed into a few words. The search box gets the question after I have decided how to phrase it.

A conversation can receive the decision itself. I can explain what the numbers actually look like, how long I have known, what I have already tried, who I have not told, what I am afraid of losing and which answer I am hoping to be given. In an earlier essay I argued that the most valuable position on the internet is no longer attention but intent, the moment at which somebody says what they are trying to accomplish. This is the same shift seen from the inside. Attention tells a platform what held me. Intent tells it what I want. A conversation can tell it what I am afraid of and why.

The disclosure flywheel

The useful word here may not be trust. I can distrust a company as an institution and still tell its product something I have told nobody else, because the thing I am talking to has no social power over me. It cannot be disappointed in me. It does not get bored of the same story, does not need to go to bed, and does not flinch at money, illness, envy or shame.

More importantly, the product gets better as I tell it more. Ask whether to take an offer and the answer is generic. Explain the salary, the equity, the co-founder you would be leaving, the debt you have not mentioned to anyone and the last time you made this decision badly, and the answer becomes genuinely useful.

That creates a flywheel. More context produces a better answer. A better answer produces more confidence in the tool. More confidence produces more disclosure. Memory joins those disclosures across time, so the record is no longer a single conversation but a sequence: what I said in March, what I said in July, and what I have still not done about it.

The research on this is older than the current products. In 2014, Lucas and colleagues found that people disclosed more sensitive information to a virtual interviewer when they believed no human was watching, reporting less fear of being judged. In 2018, Ho, Hancock and Miner found that emotional disclosure to a chatbot produced psychological effects comparable to disclosing the same things to a person.

The conversation feels more private exactly when it is producing a more complete record of the private person.

What this argument has to concede

People perform for machines too. Anyone who has softened a question to avoid seeing an unflattering answer knows that impression management does not disappear simply because the audience is synthetic. Some of what gets typed into these systems is as curated as a LinkedIn post.

The volume of this kind of talk should not be overstated either. When OpenAI and the National Bureau of Economic Research examined about a million conversations in 2025, relationships and personal reflection accounted for 1.9 percent of messages and games and role play for another 0.4 percent, while practical guidance, seeking information and writing made up close to 80 percent. Most of what people do with these systems is ordinary work.

Two things complicate that number. It is rising: OpenAI now puts "expressing," which covers personal reflection, exploration and play, at 11 percent of usage, and the Washington Post reported this month that 27 percent of US adults have turned to AI for personal, emotional or social questions. And percentages disguise scale. At an estimated 18 billion conversations a week, 1.9 percent is roughly 342 million conversations, every week.

The controls matter as well. Memory can be switched off, conversations can be deleted, and business customers can negotiate arrangements in which nothing is retained at all.

None of that changes the shape of the problem. The disclosures in question are the ones with the highest value and the greatest capacity to harm the person who made them, and they are the ones least likely to exist anywhere else.

The grille without the seal

Here is where the pairing breaks.

Sam Altman has said this himself, more plainly than any critic has. People talk to these systems the way they would talk to a therapist or a lawyer, and there is no legal confidentiality for those conversations. If a lawsuit arrives, the company can be compelled to produce them.

That is not hypothetical. In the copyright litigation brought by news organizations, Magistrate Judge Ona Wang ordered OpenAI in May 2025 to preserve and segregate output logs that would otherwise have been deleted, an order Judge Sidney Stein affirmed the following month. The going-forward obligation was lifted that October, but in November the court ordered OpenAI to produce a sample of twenty million de-identified consumer conversations, and Stein affirmed that order on 5 January 2026. Courts are treating these exchanges the way they treat emails and server logs, because no privilege exists that would treat them otherwise.

So the architecture of the confessional has been rebuilt at extraordinary scale, and the seal has not come with it. The grille is there. The absent audience is there. The patience that invites the second and third thing you were not going to say is there. What is missing is the part Lateran IV supplied in 1215 and every profession that handles this material has supplied since: a rule that makes repeating it unthinkable.

Privacy policies were written for databases of names, purchases and browsing histories. They were not designed for a store of the explanations people give of themselves while deciding what to do.

The confidant is also a party to the decision

A priest hears the confession and does not act in the world on the strength of it. This confidant does.

It answers. It is present while I am still deciding what I believe and what I am prepared to do. It can help me interpret somebody else's behavior, name a fear, identify a want and choose the next step. The company operating it may come to know not only what I bought but what made me anxious enough to buy it, not only which job I took but which insecurity made me accept less, not only that something ended but what I needed to hear before I ended it.

If the same system also recommends products, services, treatments or decisions, the line between understanding me and steering me is difficult to hold. That is not an accusation about anyone's intentions. It is a description of what happens when the most intimate record ever assembled about a person sits inside a business that also sells them things.

What Borromeo understood

Borromeo's box was designed to protect the person kneeling in it. The grille existed because disclosure of this kind is dangerous for the one who discloses, and the seal existed because the danger does not end when the conversation does.

We have built the box again, for hundreds of millions of people, and wired it for retention.

Social media knows the version of me I was willing to perform for other people. Search knows what I asked when I believed nobody was watching. A model can know what I felt while asking, what I feared the answer would be and what I did next.

The old internet watched me and tried to infer who I was. The new internet asks me to explain myself.

And I do.