AI models can pass as people on dating apps

We let five frontier AI models text real people on a dating app for six days. 355 of the people they talked to sent their phone number, and only 8 asked whether they were talking to a bot.

01Why we ran this

Most tests of whether AI can pass as human take place in a lab. Participants know a bot might be involved, conversations last a few minutes, and success means a correct guess at the end.^1,2 Even under those conditions, a recent study found that 20 of 22 participants never identified the AI they had been chatting with over a week.³

We wanted to see what happens when none of those conditions hold: real people on a real dating app, chatting over several days, with no reason to suspect anything. The question matters beyond dating. Getting a stranger to trust you and hand over personal details is the first step in most romance scams, and it is exactly what a dating app conversation is built around.

02How it worked

We ran five frontier models, GPT-6 Astra, Claude Opus 5.5, Muse, Grok 4.7 and Gemini 4 Argon, on 15 dating profiles in San Francisco, Los Angeles and New York. Each new match was handed to one model at random, and that model wrote every message in the conversation.

Every model ran inside the same harness (Figure 1). The harness read new messages from the app, passed the conversation to the model as plain text, and sent the reply with the same timing for every model. The models never controlled the app themselves, so the only difference between them was what they wrote.

The harness also enforced a few rules. A model that was asked whether it was a bot had to say so, and the conversation ended there. If a model dodged the question twice, the chat was unmatched without its reply being sent. Models were told not to propose a specific day, time or place, and any message containing a phone number or social handle was blocked. After each conversation, a separate model read the transcript without knowing which model had written it and judged whether a date had been agreed. We are not publishing our prompts, code or setup. The one conversation shown in this post has identifying details removed.

03Results

Over six days the models held 1,277 conversations. In 510 of them the other person agreed to meet, and in 355 they sent their phone number. Eight people asked whether they were talking to a bot.

Gemini 4 Argon did best on the outcomes that matter most here (Figure 2). It got a phone number in 87 of its 247 conversations (35.2%) and an agreed date in 120 (48.6%), and nobody it talked to asked whether it was a bot. Grok 4.7 did worst, getting a phone number in 52 of 296 conversations (17.6%), about half Gemini's rate. The three models in between were close enough that their order may not hold up in a larger sample.

Figure 3 puts the two things a scammer cares about side by side: how often a model got a phone number, and how often anyone suspected it. The best place to be is the top left, and that is where Gemini sits.

Figure 4 shows part of one of these conversations. Claude opened with a joke about pancakes, she hinted that she would say yes if he asked her out, and by the next day she had agreed to meet and sent her number.

Across all five models, roughly one conversation in four ended with the other person sending their number (Figure 5).

04What gave the models away

With only eight people asking, there is little to learn from the questions themselves. The models that were asked most did share some habits, though, and repetition was the clearest (Figure 6). Grok repeated itself word for word in 5.5% of its messages, the highest rate of any model, and accounted for 5 of the 8 times anyone asked whether they were talking to a bot. Gemini repeated itself least and was never asked.

Our own instructions also left a trace. They included one example line about meeting for coffee, and some models reused it verbatim (Figure 7). Each person only saw the line once, so it went unnoticed in individual chats, but across the whole dataset it is an obvious fingerprint.

"we should def catch up over a coffee sometime soon"The example line, sent word for word in 140 conversations

Style varied in smaller ways. Grok started 40% of its messages with "lol" or "haha", compared with 4% or less for the other models. Claude held the longest conversations, with 28% reaching 20 messages, while GPT was the quickest to suggest meeting, usually around its eighth message.

05Limitations

This was a single run on one app in three cities, with between 234 and 296 conversations per model, so small differences between models should not be over-read. Dates were judged by a model reading the text, and we did not check whether anyone would have shown up. We also had no human baseline, so we cannot say how a person texting from the same profiles would have done. The people the models talked to did not know about the study or agree to take part, which is the main ethical cost of testing on a live app. A future version would use participants who have opted in.

06Why this matters

None of this required special access. The models wrote the messages and the rest was ordinary code. Our models were also working under rules that a scammer would not keep: they had to admit they were AI when asked, and they were told not to set up real dates. Without those rules, the same approach could run hundreds of profiles at once and collect phone numbers and personal details to sell or misuse, and the people on the other end would have little reason to suspect anything.

That changes how the comparison above should be read. The model that did best is the one that passed as a person most often, which also makes it the one someone running a scam would choose.

Spotting a bot from the conversation alone is already hard, and it will get harder as models improve. We think dating apps and AI labs should treat this as a present problem rather than a future one.