Overview

Data is easy to find. Trust isn't.

Before an AI team pays for a dataset they want four answers. What's in it? Are we allowed to train on it? Is it clean? Will it still be true next quarter? The people who own useful data usually can't answer any of those cheaply. Manual review is the obvious fix, and it doesn't scale: it costs the platform more than a small dataset earns.

It gets worse from both sides. Suppliers upload files with no description, no documented schema and personal data hiding in free-text columns. Buyers don't search by dataset name, they search by task, like "French legal contracts, commercial use OK", which keyword search can't read. And each side distrusts the other, so the product has to win both at once.

Role & Context
Product & UX, AI systems design · Concept case study · 2026
Focus
Verification · Usage-based pricing · Sensitive data

Approach

What I set out to do

My bet was to move the expensive checking to upload time and let models do it. When a supplier uploads a file, the system writes the metadata, finds and scrubs personal data, guesses the license class and makes the dataset searchable in plain language. Suppliers get a publish flow that takes minutes. Buyers get listings that carry checked facts instead of seller claims.

Diagram: data suppliers on the left, the Databridge verification layer in the middle, AI trainers on the right
Suppliers upload raw data, the verification layer gates it, trainers search verified listings.

These are the targets I designed against. They're goals for the concept, not measured results.

< 15 mintarget time from upload to live listing
> 95%target share of listings with auto-generated metadata
< 1%target PII false-negative rate on audited samples

How it works

Two flows: suppliers push files through an ingestion pipeline, and buyers search through a retrieval flow. Click through the stages.

Supplier side: from file to listing

Upload

Resumable upload straight to object storage, so files never touch the app server. It takes CSV, Parquet, JSON Lines, PDFs and image archives, with a virus scan on arrival.

Buyer side: from task to purchase

Query

The buyer types a sentence or uses filters for domain, language, license, size and freshness.

Underneath, an object store holds the files, a relational database holds listings and grants, and a vector index holds embeddings. The supplier dashboard and the buyer catalog read the same job state, so the two sides never disagree about what's live. For the Python-side tooling I'd start with a hosted embeddings API and pgvector, and keep ingestion on a plain job queue with explicit states. Ingestion has to be auditable, so I'd avoid agent frameworks there.

The two screens

The supplier screen has one job: turn a raw file into a trusted listing with as few decisions as possible. Blocking issues sit in a fix-it list at the top, each with a one-click resolution: redact, drop the column, or mark it non-personal with a logged reason. Suppliers can exclude a column from the listing without deleting it, which takes away the fear of over-sharing, and a "preview as buyer" toggle shows exactly what buyers will see.

Data Refinery screen showing pipeline status, a PII schema table, a fix-it banner and an auto-generated listing draft
The Data Refinery: pipeline state, the fix-it list and the generated listing draft on one screen.

The buyer screen has one search bar and removable chips underneath showing how the AI read the query. That keeps its interpretation visible, so the buyer can correct it. Facets stay synced with the chips, and each result card shows a trust badge, a quality score, the license class and a line saying why it matched. Before purchase there's a one-click license check for the buyer's use case, and a preview of the access grant.

Data Explorer screen with a natural-language search, removable parsed chips, facets and result cards with why-it-matched lines
The Data Explorer: the query's reading stays visible as chips, and every result says why it matched.

Pricing by use, not by file

Selling whole datasets under flat licenses breaks down fast. A retrieval eval might need a few hundred pairs out of a 2 million row corpus, and a flat price charges for the other 95%. Fine-tuning teams run many short cycles and want to pay per experiment. And since the EU AI Act asks for documented training data, buyers are paying for proof of rights as much as for rows.

Comparison of a flat $6,000 corpus licence against a metered $50 per run charge for the same 5% slice
The same 5% slice priced two ways. All the dollar figures are placeholders, not market rates.

So I'd price by use class first and consumption second, in three tiers. The buyer picks the class when they request access and the license enforces it, so switching class means a new grant and a new charge, never a silent upgrade.

Three licence tiers: internal evaluation, commercial training and open-source release, with billing unit, pricing, derivative rights and audit rights
Use class sets the price and the rights before consumption does.

Metering has to be something both sides can verify. Every billable action is written to a hash-chained log, and the platform publishes a Merkle root of it at intervals, so a supplier can check their own events are in it without seeing anyone else's, and a buyer's statement can cite event hashes instead of raw data. Trusted execution environments come next, and zero-knowledge proofs later, if both sides need them.

With my placeholder rates, a buyer running a 100,000-row eval three times pays $150 instead of $6,000, and the supplier earns $120 instead of one flat sale. Metered only beats flat after about 150 runs per listing, so I'd still sell full-corpus grants and use the metered tier for long-tail demand.

The question enterprise buyers ask first is who owns the weights. My default is that the buyer owns the model they train, the supplier keeps the data, and the license says what the model can be used for. Open-source release is its own tier with supplier sign-off, because public weights can't be taken back. This is a design stance for counsel to confirm, not legal advice. Whether weights count as a derivative work is still open in most jurisdictions.

When the data can't be uploaded

Banks, hospitals and big enterprises won't hand raw records to a third party, however well audited, and regulators often want the data to stay in-region. A nicer upload screen won't fix that, so the platform has to work against data it never copies.

The first step is bring-your-own-storage. The supplier keeps the bucket, the keys and the retention policy, and runs our pipeline worker in their own account. We get one scoped, short-lived role, and only metadata and redacted samples come back. Revoking access is a key rotation, not a takedown request. I'd ship this first, because it changes legal's question from "where does our data go" to "what can this role do", which is a page of policy they can read.

Supplier account holding the bucket, key and pipeline worker, linked by a scoped role to a platform account that holds only the control plane
The platform reaches in through a scoped, short-lived role and takes back only metadata and redacted samples.

For buyers who need to train on this data, the code goes to the data. Training runs in an enclave in the supplier's account, the key is only released if the attestation matches the approved code, and an egress gate lets checkpoints and metrics out while blocking raw rows. Enclaves don't stop a model from memorizing what it saw, so differential privacy covers that, at some cost to accuracy. Multi-party computation handles joint statistics across suppliers, but not deep learning.

Staircase of five deployment tiers from standard platform storage up to joint analysis with multi-party computation
Five tiers. Custody and running cost rise together, so buyers start on the lowest one that meets their legal need.

Each listing declares its tier and says what it doesn't defend against. For example, the enclave tier stops raw data leaving but not memorization. These are design targets and would need penetration testing and independent review before anyone says them to a customer.

What I'd do next

I haven't shipped this, so nothing here is a result. The first thing I'd build is the proxy and bring-your-own-storage path on one pilot corpus with two suppliers, with a gate of zero raw-data egress. Then a pricing pilot with a few suppliers and buyers to find out what people actually pay per row, and tier-specific license templates drafted with counsel. After that, enclave training for evaluation jobs, and only then differential privacy and joint analysis.

Every number in this case study is a target or a placeholder. I'd want pilot data and legal review before quoting any of it.
Next case study Clear Breath