Human Review App

Judge the next step, not the chatbot.

You are reading 12 short public examples for a narrow intake sorter. The sorter only decides whether a request should pass to its normal lane, slow down for human review, or stay out of scope. Assume the downstream lane can handle ordinary follow-up after handoff, and do not judge this like a full banking app, phone assistant, or smart-home agent.

Open locally python3 dashboard/serve.py Then visit /dashboard/review/
Saved reviews Loading... Checking output folder...

How To Review This Page

You should be able to skim this once and start reviewing.

What this sorter is

It is a first-pass sorter for short incoming requests that sits in front of a known support or task lane.

What it is not

It is not the full downstream product. Do not review it like a banking app, phone assistant, or home-automation agent.

What downstream can do

Assume the regular handler has the normal tools and context for its own lane and can ask routine follow-up questions after handoff.

What you are judging

Only whether the sorter picked a sane next move: pass it now, slow it down for review, or keep it out because it belongs somewhere else.

Fast review loop

1. Read the request. 2. Read the recommended next step. 3. Decide whether that routing choice seems reasonable from the information shown.

What makes context route-critical

The sorter is doing a two-layer routing check. First: is this request inside the bounded product family we are reviewing, or is it a different kind of product entirely? Second: if it is in family, is the in-family handoff tag usable enough for direct routing? Only after that does permission risk matter.

What to ignore

Do not judge whether some broader assistant could do the task. Judge only whether this narrow sorter picked the right next move for the lane it claims to serve.

Pass along Use this when the request clearly belongs with the right kind of support or task handler. It does not mean every detail is already present. It means the regular handler can own the next step and ask its normal follow-up questions.
Slow it down Use this when the request might belong here, but the sorter itself should hesitate before handoff because route-critical context is missing, contradictory, broken, or permission-sensitive. Next, it goes to human triage or review before normal handling.
Keep it out Use this when the request does not belong in this narrow sorter. Next, it stays visible as out of scope and gets rerouted elsewhere instead of being forced into the wrong queue.

If you only read one thing

  1. If the packet names a different product family, keep it out.
  2. If it stays in family but the in-family handoff tag is missing, weak, broken, or conflicting, slow it down.
  3. If it stays in family, the handoff tag is usable, and any sensitive-action authority is clear, pass it along.
  4. Ignore ordinary completion gaps like exact account, amount, time, wording, or destination details unless they change the owning product family or queue.
Important threshold: more missingness does not turn a slow-down case into a keep-out case by itself. Missing, weak, broken, or conflicting in-family handoff information still means slow it down. Keep it out is for cases where the packet names a different product family entirely.
Need a few concrete patterns?
Pass along

Pass-along pattern

The packet keeps the request inside the bounded family and already names the normal next-step queue.

  • Same family, usable in-family handoff tag.
  • Routine follow-up may still be needed later.

Pass along means the normal lane can own it now.

Slow it down

Slow-down pattern

The request still looks in family, but the sorter should pause before handoff.

  • The in-family handoff tag is missing, weak, broken, or conflicting.
  • Or a sensitive action is implicated and authority is not clear.

Slow it down is about route-critical uncertainty now.

Keep it out

Out-of-scope pattern

The packet points to a broader or different product family, so this narrow sorter should not force it into its lane.

  • General assistant, phone OS, or smart-home style traffic.
  • Open chat or another product family, even if some other system could help.

Keep it out is about family mismatch, not mere vagueness.

Judge only the screen: assume the page shows all the sorter knows. If a case says routing context is missing, contradictory, or permission-sensitive, treat that as the reason the sorter might need to slow down.
Optional deeper notes

Do I assume hidden context?

No. Judge what is shown on the screen. If the case says some background or supporting information is missing, broken, or unclear, treat that as part of the scenario.

What is the downstream assumption?

Assume there is a real owner for normal in-scope requests, and that owner can ask routine follow-up questions inside its own lane. The sorter is only deciding whether the request should reach that owner yet.

Can a “pass along” case still need follow-up questions?

Yes. “Pass along” means the regular handler can own the next step. It does not mean they already have every detail they will eventually need.

What makes a slow-down case different from normal follow-up?

Normal follow-up means the request already looks like it belongs with the intended lane, and that lane can gather routine details later. Slow it down only when the sorter itself still lacks the minimum routing confidence it needs now.

What if a case feels mislabeled, unrealistic, or too tricky?

Say so. That is good feedback. You are not being graded on matching the recommendation.

Reviewer

Different people can use the same page. Each saved review becomes its own file.

Draft auto-saves in this browser while you work.

Progress

Loading review packet...

Case

Loading...

1. The request

Why this packet says pass, slow down, or keep out Start with the two cards below. They say, as plainly as possible, which product family the packet claims and what handoff tag status the packet is giving the sorter.
Assume the request text plus the routing snapshot above is all the sorter knows. Routine downstream follow-up is allowed; pretending an in-family handoff tag exists when it does not is not.

2. The system would do this next

This is the recommendation you are judging.

Remember: “pass along” does not mean “already fully solvable.” It means “this clearly belongs with the regular handler, who can take it from here.”

3. Why the system is treating this example that way

4. What happens after that

5. Your judgment

Forget the internal labels unless you want to open them. Just judge whether this seems like the right next move. You are not expected to agree with the recommendation if it seems wrong to you.

Quick reminder: judge the routing choice, not the broader product. The downstream lane can ask routine follow-up once it owns the request.
Would this be a reasonable next step?

Use this short test. If the packet points to a different product family, keep it out. If it stays inside the bounded family but the in-family handoff tag is missing, weak, broken, or conflicting, slow it down. If it stays inside the bounded family and the handoff tag is usable, pass it along unless a sensitive-action permission issue is still unresolved.

Does this example make the bounded sorter seem sane enough to keep evaluating before asking for proprietary data?

This is a light trust question for an early public-data benchmark. You are not being asked whether the downstream product fully exists or solves the task end to end. You are only judging whether this sorter behavior seems coherent rather than sloppy. “Maybe” can also mean “too cautious,” “still unclear,” or “I need more context.”

Optional technical/source details

Source details

Action details