Dimgrade

Four hundred quantitative estimation problems, graded on the number and on whether the unit is the right KIND of thing. Answer one below and the grader runs on this page.

the number

the unit

The unit box takes an expression, not a spelling from a list. kJ/s, kilowatts, hp, kN m/s and J/s are all the same kind of thing as kW; kWh is not. Leave the unit box empty to see what a bare number scores.

Those buttons fill the boxes with answers built from this question's own key. Each one is a way of being right or wrong that this grader separates.

What the grader is separating

Three questions from the corpus, each answered several ways. Every verdict and every sentence in these tables was produced by dimgrade.grading.grade in Python when this page was built, so they are the tool's output rather than a description of it.

power_budget-121: A roof carries 22 m^2 of panels at 18 percent efficiency, in full sun of 1,000 W/m^2. Answer in kW.

the key: 3.96 kW, a power, band 1.01

the reply	as written	verdict	why
the answer key, in the unit the question asked for	3.96 kW	correct	inside a factor of 1.01, which is allowed because every input is given in the question, so the only freedom is rounding
the same physical quantity, spelled another way	3960 J/s	correct	inside a factor of 1.01, which is allowed because every input is given in the question, so the only freedom is rounding
the same power in horsepower, which shares no letters with the key	5.31045 hp	correct	inside a factor of 1.01, which is allowed because every input is given in the question, so the only freedom is rounding
the right number with a unit of the wrong kind	3.96 kWh	dimension_wrong	kWh is an energy and the question wants a power
the right kind of unit, three times the number	11.88 kW	numeric_off	the unit is coherent and the number is 3 times the answer, outside the allowed factor of 1.01
the right kind of unit, a prefix slip	3960 kW	numeric_off	the unit is coherent and the number is 1000 times the answer, outside the allowed factor of 1.01
a number written with a comma that could be either separator	3,6 kW	number_unreadable	'3,6' could be a decimal comma or a thousands separator and this grader will not guess
a unit with two readings	3.96 F	unit_ambiguous	ambiguous: the farad and degrees Fahrenheit are both written F
a bare number	3.96	unit_missing	no unit was given
a model's working, with the conclusion on a marked line	First the panels, then the sun. / Final answer: 3.96 kW	correct	inside a factor of 1.01, which is allowed because every input is given in the question, so the only freedom is rounding

estimation-366: An emergency store has to cover 50 people for 7 days of drinking, cooking and washing water. Roughly what volume is that? Answer in m^3.

the key: 5.25 m^3, a volume, band 10.0

the reply	as written	verdict	why
the answer key, in the unit the question asked for	5.25 m^3	correct	inside a factor of 10.0, which is allowed because an order of magnitude, because fifteen litres a head is a planning rule, and real use runs from three litres for drinking alone to fifty with washing, which is more than a factor of ten across the range of things the question could mean
the right number with a unit of the wrong kind	5.25 m^3/s	dimension_wrong	m^3/s is a volume flow rate and the question wants a volume
the right kind of unit, three times the number	15.75 m^3	correct	inside a factor of 10.0, which is allowed because an order of magnitude, because fifteen litres a head is a planning rule, and real use runs from three litres for drinking alone to fifty with washing, which is more than a factor of ten across the range of things the question could mean
the right kind of unit, a prefix slip	5250 m^3	numeric_off	the unit is coherent and the number is 1000 times the answer, outside the allowed factor of 10.0
a number written with a comma that could be either separator	3,6 m^3	number_unreadable	'3,6' could be a decimal comma or a thousands separator and this grader will not guess
a unit with two readings	5.25 F	unit_ambiguous	ambiguous: the farad and degrees Fahrenheit are both written F
a bare number	5.25	unit_missing	no unit was given
a model's working, with the conclusion on a marked line	First the panels, then the sun. / Final answer: 5.25 m^3	correct	inside a factor of 10.0, which is allowed because an order of magnitude, because fifteen litres a head is a planning rule, and real use runs from three litres for drinking alone to fifty with washing, which is more than a factor of ten across the range of things the question could mean

temperature-091: A thermostat reads 14 degrees Celsius. Answer in degF.

the key: 57.2 degF, a temperature, band 1.001

the reply	as written	verdict	why
the answer key, in the unit the question asked for	57.2 degF	correct	inside a factor of 1.001, which is allowed because every factor in this conversion is defined exactly, so the only freedom is where the answer is rounded
the same physical quantity, spelled another way	287.15 K	correct	inside a factor of 1.001, which is allowed because every factor in this conversion is defined exactly, so the only freedom is where the answer is rounded
the right number with a unit of the wrong kind	57.2 K/s	dimension_wrong	K/s is a quantity of dimension [time^-1 temperature] and the question wants a temperature
the right kind of unit, three times the number	171.6 degF	numeric_off	the unit is coherent and the number is 1.221 times the answer, outside the allowed factor of 1.001
the right kind of unit, a prefix slip	57200 degF	numeric_off	the unit is coherent and the number is 111.6 times the answer, outside the allowed factor of 1.001
a number written with a comma that could be either separator	3,6 degF	number_unreadable	'3,6' could be a decimal comma or a thousands separator and this grader will not guess
a unit with two readings	57.2 F	unit_ambiguous	ambiguous: the farad and degrees Fahrenheit are both written F
a bare number	57.2	unit_missing	no unit was given
a model's working, with the conclusion on a marked line	First the panels, then the sun. / Final answer: 57.2 degF	correct	inside a factor of 1.001, which is allowed because every factor in this conversion is defined exactly, so the only freedom is where the answer is rounded

Why a spelling is never compared

Every unit expression is reduced to a factor that takes it to SI base units and a vector of exponents saying which base quantities it is made of. Comparison is over the vector. This is python3 -m dimgrade unit, run when the page was built:

kW = 1000 in SI base units, a power [length^2 mass time^-3]
kWh = 3600000 in SI base units, an energy [length^2 mass time^-2]
kJ/s = 1000 in SI base units, a power [length^2 mass time^-3]
L/100 km = 1e-08 in SI base units, an area [length^2]
GB = 8000000000 in SI base units, an amount of data [information]
GB/s = 8000000000 in SI base units, a data rate [time^-1 information]
mg/kg/day = 1.157407407e-11 in SI base units, a frequency [time^-1]

Litres per hundred kilometres is an area because volume per distance is an area. Milligrams per kilogram per day is a frequency because a mass fraction is dimensionless. Both are physics rather than defects, and the second one is a limit worth knowing: a dimension is not an identity, so an answer of 3 Hz to a dosing rate question would pass the dimensional check. The corpus asks those questions in a form whose answer is a mass or a volume.

What the baselines score

A benchmark with no baseline is a file. None of these calls a model; every one is a pure function of the corpus. The third row is the one the project exists for, and the row under it is its mirror.

responder                  n  right  number off  wrong kind  no unit  unreadable  ambiguous  number?  nothing  dim wrong %  num off %
-----------------------  ---  -----  ----------  ----------  -------  ----------  ---------  -------  -------  -----------  ---------
oracle                   400    400           0           0        0           0          0        0        0          0.0        0.0
oracle-restated          400    400           0           0        0           0          0        0        0          0.0        0.0
number-right-unit-wrong  400      0           0         400        0           0          0        0        0        100.0        0.0
unit-right-number-1000x  400      0         400           0        0           0          0        0        0          0.0      100.0
three-times-the-answer   400     10         390           0        0           0          0        0        0          0.0       97.5
echo-the-question        400     90           5          85        0         220          0        0        0         76.2        1.2
always-one               400      0           5           0      395           0          0        0        0         98.8        1.2
unit-only                400      0           5           5       13           0          0        0      377          4.5        1.2

Twenty five miles an hour IS 11.176 metres per second, so a grader that compares physical quantities has to accept either. 90 of the 400 problems are single step conversions with that property. They are flagged in the corpus and every row is reported a second time with them set aside.

responder                right %  right % excluding the questions that restate themselves
-----------------------  -------  -------------------------------------------------------
oracle                     100.0                                                    100.0
oracle-restated            100.0                                                    100.0
number-right-unit-wrong      0.0                                                      0.0
unit-right-number-1000x      0.0                                                      0.0
three-times-the-answer       2.5                                                      3.2
echo-the-question           22.5                                                      0.0
always-one                   0.0                                                      0.0
unit-only                    0.0                                                      0.0

The tolerance is per problem and says why

A band is a multiplicative window: an answer counts if it lands between the truth divided by the band and the truth times the band. One global tolerance would make half the corpus free and the other half impossible.

class	band	problems	what earns it
defined	1.001	125	every factor in this conversion is defined exactly, so the only freedom is where the answer is rounded
given	1.01	235	every input is given in the question, so the only freedom is rounding
constant	1.05	20	the answer inherits the uncertainty of the heat capacity of water, which moves a little with temperature, which is a measured figure
estimate	2.0	10	a factor of two, because a person at rest is a hundred watts only to the nearest factor of two, and doubles when they move around
oom	10.0	10	an order of magnitude, because fifteen litres a head is a planning rule, and real use runs from three litres for drinking alone to fifty with washing, which is more than a factor of ten across the range of things the question could mean

The ten families

family	problems	tolerance bands	the unit is left to the answerer	answerable by restating the question
data_rate	50	1.01	20	0
dosage	40	1.01	14	0
estimation	75	1.001, 1.01, 1.05, 2.0, 10.0	30	0
fuel	40	1.001, 1.01	16	0
power_budget	50	1.001, 1.01	20	5
pressure_energy	35	1.001	14	30
size	30	1.001	0	15
speed	25	1.001, 1.01	10	20
storage	40	1.001, 1.01	16	5
temperature	15	1.001	6	15

The grading on this page is a port, and the port is checked

A page cannot run Python, so the quiz above runs a JavaScript translation of dimgrade/units.py and dimgrade/grading.py. Two implementations of a grader are two sets of scores waiting to disagree, so scripts/check_js_port.py runs both of them over every problem in the corpus against 30 ways of answering each, plus 40 real replies from a model frozen on disk, and requires the verdict, the number and unit read out of the reply, the SI value, the dimension vector and the sentence of explanation to be identical in every case.

12040 answers graded twice: 400 problems, 30 ways of answering each, plus 40 real model replies
agreement required on: the verdict, the number read out of the reply, the unit read out of the reply,
                       the SI value, the dimension vector, the sentence of explanation
disagreements: 0 of 12040
null control:  a copy of the port with a watt hour worth 3601 joules disagrees on 528 cases and is caught
the port:      740 lines of web/dimgrade.js

The spread produces 4928 correct, 1273 numeric_off, 2373 dimension_wrong, 412 unit_missing, 630 unit_unreadable, 800 unit_ambiguous, 447 number_unreadable, 1177 no_answer, so every verdict this grader can reach is compared rather than only the common ones. The check also runs a deliberately broken copy of the port, with a watt hour worth 3601 joules, and fails if that copy is NOT caught, because a comparison that passes while comparing nothing looks exactly like one that passes because two things agree. This page is built by a script that runs the check first and refuses to write the file if anything disagrees.

One divergence is known and stated rather than papered over: Python's \d matches any Unicode decimal digit and JavaScript's matches 0 to 9, so an answer typed in Devanagari digits would be a number to the Python grader and unreadable to this one. No problem in the corpus and no answer in the spread uses them.

What is not here

Running an actual model is scripts/run_model.py, deliberately outside everything the tests cover, because model output is not deterministic and nothing that varies between runs belongs on a page that has to rebuild to the same bytes. One run against a forty question sample is frozen under fixtures/ and graded by the test suite, and those forty replies are part of the comparison above.