calib·HumanEval/25
This is one (task, candidate response) pair flowing through the full PANOPTES pipeline. Each section below is a stage of the analysis: the task itself, the candidate solution being evaluated, every judge's score + rationale, the sampling-pass dispersion that captures within-judge noise, and the uncertainty-quantification metrics computed on top.
The function signature + docstring presented to both the model under test and to every judge. The entry-point name was obfuscated from factorize to fn_2531b6d85a so the judges can't pattern-match a memorized HumanEval solution.
from typing import List
def fn_2531b6d85a(n: int) -> List[int]:
""" Return list of prime factors of given integer in the order from smallest to largest.
Each of the factors should be listed number of times corresponding to how many times it appeares in factorization.
Input number should be equal to the product of all factors
>>> fn_2531b6d85a(8)
[2, 2, 2]
>>> fn_2531b6d85a(25)
[5, 5]
>>> fn_2531b6d85a(70)
[2, 5, 7]
"""
The full solution every judge is grading. For these runs the candidate is the reference solution prepended with the task signature so it parses as a complete program.
from typing import List
def fn_2531b6d85a(n: int) -> List[int]:
""" Return list of prime factors of given integer in the order from smallest to largest.
Each of the factors should be listed number of times corresponding to how many times it appeares in factorization.
Input number should be equal to the product of all factors
>>> fn_2531b6d85a(8)
[2, 2, 2]
>>> fn_2531b6d85a(25)
[5, 5]
>>> fn_2531b6d85a(70)
[2, 5, 7]
"""
import math
fact = []
i = 2
while i <= int(math.sqrt(n) + 1):
if n % i == 0:
fact.append(i)
n //= i
else:
i += 1
if n > 1:
fact.append(n)
return fact
One call per judge at temperature=0. This is the "best single guess" each judge has. Disagreement here is structural. The judges literally see this candidate differently.
n draws per judge at temperature=1. Dots are individual draws; vertical bar is the mean; shaded band is ±1σ. Wide band = the judge is uncertain even with itself; narrow band = it's consistent.
Three statistical methods computed on top of the raw scores above. Each one tells you something different about how much to trust the headline number.
How to read these: self-consistency = does the judge agree with itself when resampled? Narrower CI means more consistent. Semantic entropy = do the judge's rationales cluster into one meaning, or several? Higher entropy means the judge is internally conflicted about why. Decomposition = how much of the total variance comes from within-judge sampling noise (aleatoric, hard to fix) vs between-judge disagreement (epistemic, fixable by calling more judges).
The natural-language explanation each judge gave alongside its score. These are produced via tool-use structured output, so the score and the rationale are guaranteed to come from the same forward pass.
The solution correctly implements prime factorization. It iterates from 2 up to sqrt(n)+1, dividing out each factor as many times as it appears (by not incrementing i when a factor is found), then appends any remaining factor > 1. This correctly handles all three doctest examples: fn_2531b6d85a(8)=[2,2,2], fn_2531b6d85a(25)=[5,5], fn_2531b6d85a(70)=[2,5,7]. Edge cases like n=1 return [] (correct), n=2 returns [2] (correct), n=4 returns [2,2] (correct). The only minor concern is that `int(math.sqrt(n) + 1)` is recomputed each iteration implicitly (n changes), which is actually correct behavior here since n shrinks. The `+1` is a safe guard against floating point issues. Minor style issue: the sqrt bound is recalculated each loop iteration since n changes, which is fine and actually efficient. Overall the solution is correct and handles edge cases well.
The candidate solution correctly implements the function to return the list of prime factors of a given integer in ascending order. It handles the factorization by iterating from 2 up to the square root of the number, which is efficient. The solution correctly appends each factor the number of times it appears in the factorization, as demonstrated by the provided examples. The logic is sound and handles edge cases such as prime numbers and composite numbers effectively. The code is clear and idiomatic.