Turn production traffic into training data

Turn a deployment’s real traffic into the training data for its next version

sf model ls                    # find the deployment's model

Reflow is the governed path from a deployment’s production traffic back to the training data of its next version. You turn capture on per deployment, let the buffer fill, mine it for candidates, and promote what survives review into a dataset version.

It closes the loop the platform otherwise leaves open: data goes in, a model comes out, it serves — and what it learns from serving goes nowhere.

What is captured is real user input, and it is the most sensitive data this platform will ever hold. Capture is off until somebody turns it on, that person is recorded, and every change to the settings is audited.

The shape of it

traffic → Reflow Buffer → mining → Candidate Set → [ you promote ] → Dataset version
          (expires)                  (replaced)      (audited)        (immutable)

Only the last step is irreversible. Everything before it expires or gets replaced, which is what makes it safe to be wrong about sampling, redaction or mining.

Turn it on

Deployment → Reflow tab.

Setting	Meaning
Sample rate	0 is off, which is the default everywhere. Sampling is decided before a request is issued, so an unsampled request costs nothing
Retention	Default 30 days. Expiry is real deletion; there is no archive tier
System prompt	Captured by default. It is often your own prompt engineering, and the most sensitive field here — leaving it out makes records weaker for training, because the behaviour was conditioned on text the record would be missing

Buffer bytes count against the owner’s storage quota. The quota wins over retention: if you go over, the oldest buffer files are reclaimed early and the loss is counted. A full shared disk stops training for everyone on the cluster.

What is never captured

Request headers of any kind, so an Authorization header cannot arrive by accident; the deployment token itself (only its id, which is the tenant boundary inside one deployment); and anything but chat completions.

Redaction

Masking runs before anything reaches disk — redaction that runs afterwards is not redaction, it is filtering. The default pack covers emails, phone numbers (mainland China and E.164), US SSNs, PRC resident ids, payment-card-shaped digits and key-shaped strings. It is versioned, and the version is stamped on every record so you can tell later which rules cleaned what.

It errs toward masking too much. A long hex string in a code sample gets masked as a possible key; that costs a slightly worse training example, and the opposite error puts a credential in a corpus.

Mine the hard cases

The useful half of a buffer is small, and it is not the average request. Mining picks out what is worth another training pass:

Signal	What it means
Request failed	Cheapest possible flag
Answer truncated	The model hit the length ceiling, so it did not finish
Malformed tool call	Arguments that are not valid JSON, or a tool the request never declared
User retried	The same prompt again, minutes later. The customer already made this judgement
Scored low	A rubric you name scores what the free signals missed. This is the only signal that spends judge tokens

Mining replaces the previous candidate set; it is a working suggestion, not a ledger.

Promote

Select candidates, name a destination dataset, and promote. Three checks, all fail-closed:

• Visibility. A deployment is readable by its owner and by administrators, so the only non-escalating destination is that owner’s own private dataset — including a pre-existing one. Promoting into a public dataset publishes your users’ traffic to every account on the platform; an administrator can do it, and the audit record says so.
• Contamination. Name the evaluation datasets to check against, or say explicitly that you are skipping the check. The choice is recorded on the version, because nobody checked and checked and clean must not look alike later. Production traffic overlapping an eval set is the normal case — that set was probably built from the same distribution.
• Not empty. A version that exists, is referenceable, and teaches nothing is worse than no version.

The contamination estimate uses 13-gram fingerprints sampled at 1/64. A small promotion of short prompts produces too few n-grams for the estimate to mean anything, and is reported inconclusive rather than as a pass.

Seeing the loop

Models → expand a model → Data flywheel. It shows one version’s whole chain: sources, training run, evaluation verdict, serving and captured volume, and the datasets that came back.

“Closed” is deliberately strict. Capturing traffic is not closing the loop, and neither is mining it. The loop closes when the traffic became a dataset somebody can train on.