GoldiBench

Right Tools, Wrong Amount: Benchmarking Agents on Effect Judgement in Professional Editing

Agents pick the right editing tools, but they cannot judge how much of each tool a look needs.

Wang Yang1, Xinpeng Li1, Yanyan Zhang1, Chaoda Song1, Shouren Wang1, Debargha Ganguly1, Vikash Singh1, Chuang Ma2, Xiaoran Cai3, Kai Ye1, Xiaotian Han1, Vipin Chaudhary1
1 Case Western Reserve University2 NII3 Columbia University
TARGET render
Target
The same edit at the chosen strength
Edit at ×0.5
The question

Same tool. How much of it?

darktable task b000: the answer is exposure +0.943 EV. Move the dial to apply the same tool at other strengths.

FAIL ΔE to target 3.32 closeness 0.48

Pass: closeness ≥ 0.75 and mean CIEDE2000 ΔE ≤ 4. Half and double strength fail on every task.

Findings

Operation is solved. Judging the amount is not.

99–100%

pass rate with programs from 4B parameters up when the tools and values are given (L1).

19% · 33%

best pass rate with programs · through the GUI when only the tools are given (L2).

×2

pass rate of strong models when forced to compare their render with the target before answering.

Paper figure 1: a request describes the look; agents operate but miss the amount; reading the measured difference helps

(1) The request describes a look; GPT-6.1 picks the right tools at the wrong amounts. (2) Pass rates collapse from L1 to L2 and L3 on both channels. (3) Measuring the TARGET − START difference helps; the manual or the difference as an image does not.

Benchmark

500 tasks with an exact answer

Each task pairs a START image with a TARGET rendered from a hidden program in the application's own parameters, so the amount has an exact answer. The agent edits by writing a program or by operating the GUI; the benchmark re-renders its edit and compares it with TARGET.

L1

Tools and values

Can the agent apply a stated edit? Measures operation.

L2

Tools only

START, TARGET and the tool set. How much of each tool? Measures effect judgement.

L3

Goal only

START and TARGET. Tool choice and effect judgement together.

How a task is built

1 · Start asset

A photo (PSBench), an SVG drawing (Openclipart) or a 3D scene (Poly Haven).

2 · Propose

An open model writes the look in words and proposes tools and value ranges.

3 · Render

Values are sampled and rendered one step at a time by the application itself: that is TARGET.

4 · Check

Replay gives TARGET exactly; ×0.5 and ×1.5 strength fail; every step is needed; two VLM judges accept the look.

Results

A cliff between L1 and L2, for every model

Pass rate per level, mean over the five applications (100 tasks each). Open models: mean of 5 seeds (program) or 3 seeds (GUI).

Full table with closeness
Analysis

What helps, and what does not

L2 pass rate (%) on the intervention task subsets, before → after the intervention (same tasks, same models).

Scale does not close the gap

Qwen3.5 from 0.8B to 397B parameters, writing programs: L1 reaches 100% from 4B on, while L2 moves from 1.4% to 10.5%.

Examples

Same tools, different amounts

One L2 task per application from the main run. Each result is the benchmark's re-render of the agent's final program.

Watch the agents operate the GUI

One task per application: a passing episode (left) and a failing one on the same task (right). Every step is one screenshot with the model's own words for that step; the last frame shows the final state next to TARGET.

Citation

BibTeX

@misc{yang2026goldibench,
  title  = {GoldiBench: Right Tools, Wrong Amount -- Benchmarking Agents on
            Effect Judgement in Professional Editing},
  author = {Yang, Wang and Li, Xinpeng and Zhang, Yanyan and Song, Chaoda and
            Wang, Shouren and Ganguly, Debargha and Singh, Vikash and Ma, Chuang and
            Cai, Xiaoran and Ye, Kai and Han, Xiaotian and Chaudhary, Vipin},
  year   = {2026}
}