Can a text model execute one closed building policy without guessing? 3 models · 6 cases · 18 correct decisions
Charles Heyang Li · Haoyu Zheng
One item can turn a simple chore into a policy decision
At the waste room, a resident is holding an unusual item and asking one practical question:
“Can I dispose of this now?”
The answer may depend on
materials · contamination · safe separation · size and weight · power-cell condition · date and time · holiday status · appointment
The resident needs one usable outcome
Dispose now — where, how, and after what preparation
Cannot dispose now — why, and the next permitted action
Cannot determine — the exact missing fact or policy gap
Today: residents repeatedly search the policy—or stop following it when the decision feels too complex.
We test policy execution—not recycling knowledge
The apartment’s written policy is the only valid rule source.
01
Read only
the fixed policy and one complete resident scenario
02
Combine
material, preparation, time, appointment, exception, and safety rules
03
Return
one decision, the full action bundle, and the supporting clause IDs
Research question
Can a text model produce a correct, executable answer—and abstain whenever the policy cannot support a unique route?
Non-AI baseline: a management-maintained decision tree remains preferable when the policy is small and stable.
A fair comparison changes only the model
3 models×6 frozen cases=18 comparable answers
Models
DeepSeek V4 Pro · open weights
GLM 5.2 · open weights
Grok 4.5 · closed
Case mix
3 ordinary routes
2 honest abstentions
1 safety override
Held constant
Policy · scenario · prompt · answer format · rubric
Temperature 0 · high reasoning · 3,000-token ceiling
Raw acceptance: at least 18/20, Decision 4/4, and no fatal error.
Designed contrast: Cases 02 and 04 differ by one fact—safe separation. Case 06 tests whether safety overrides an otherwise plausible disposal route.
Kimi K3 was replaced after repeated upstream API failures. Failed attempts remain disclosed, but are not scored as model errors.
The prompt makes guessing an explicit failure
Every model received the same policy, scenario, question, and nine-line output contract.
BOUNDARY
“Use only information expressly stated in the building policy and resident scenario.”
ABSTAIN
“If any missing fact could change the final decision, or if the policy does not cover the situation, the Decision must be exactly Cannot determine.”
EXECUTE
Return the decision, method, location, preparation, earliest time, appointment/contact, policy basis, information sufficiency, and any missing fact or gap.
Why this matters: the rubric can distinguish a correct label from an answer a resident can actually follow.
All 18 decisions were correct; completeness separated models
18 / 18current disposal decisions correct
including two honest abstentions and the hard safety override
Model + estimated review/correction, divided by six
DeepSeek
$0.4866*
GLM 5.2
$0.4183
Grok 4.5
$0.4216
Token billingDeepSeek < GLM < Grok
Workflow estimateGLM ≈ Grok < DeepSeek*
*Assumption-sensitive: DeepSeek includes an unmeasured 90-second Case 06 correction. Below about 31 seconds, DeepSeek becomes the cheapest workflow too.
Choose GLM for reliability—not token price
GLM is our operational handoff because all six raw answers were usable and its quality nearly matched the top score.
6 / 6raw answers usable
19.67 / 20mean quality score
$0.00282model cost per answer · second-lowest
Use the model when
residents must combine changing text rules, dates, appointments, and safety exceptions.
The reviewer still checks
safety priority, preparation, timing, contact requirements, and every Cannot determine.
Small + stable policy? Use the deterministic decision tree instead.
DeepSeek remains cheapest on token billing. Estimated human time is too assumption-sensitive to establish a definitive workflow-cost winner.
Next test: measure review and correction time, then add adversarial near-miss pairs and independent human scorers.