Some issues for the existing dataset

[feixiangdejiahao]

1. Examples (public tests) in requirement are incorrect.
   For example, in humaneval/47, the second example is incorrect and the output should be 8.

def median(l: list):
    """Return median of elements in the list l.
    >>> median([3, 1, 2, 4, 5])
    3
    >>> median([-10, 4, 6, 1000, 10, 20])
    15.0
    """

2. The type of variables used in examples are conflict with the requirement.
   For example, in mbpp/2, the example uses tuple as inputs but the requirement mentions the inputs are list

def similar_elements(test_tup1: tuple, test_tup2: tuple) -> tuple:
"""
Write a function to find the shared elements from the given two lists.
assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))
"""

[feixiangdejiahao]

One important issue leading to ambiguity is missing input range in requirement.

[feixiangdejiahao]

One important issue for failing cases are that LLM-generated inputs are not align with the argument type in function signature. Poor instruction following fails the detection. So better to use crosshair.

[feixiangdejiahao]

It would be better to only compare the overall result for ablation study since we use a general threshold for all datasets. For deepseek-entropy, the threshold is 0, which is good for humaneval and mbpp.

[feixiangdejiahao]

original_requirement

 def sort_counter(dict1: dict) -> list:
"""
Write a function to sort a dictionary by value.
"""

repaired_requirement

 def sort_counter(dict1: dict) -> list:
    """Sorts a dictionary by its values in descending order and returns a list of tuples.
    
    Each tuple contains a key-value pair from the dictionary, sorted by the value in descending order.
    The function uses the Counter class from the collections module to achieve this.
    
    Example:
    >>> sort_counter({'a': 3, 'b': 1, 'c': 2})
    [('a', 3), ('c', 2), ('b', 1)]
    """

• original_passk: 0.19999999999999996
• repaired_passk: 1.0
• original entropy: 0.7219280948873623
• original_weighted_test_consistency: 0.2

Another case

def unique(l: List[int]) -> List[int]:
    """Return sorted unique elements in a list

Original_passk: 1
Original_entropy: 0

Finding
Memerization or we called data leakage shows huge influence on experiment.

[feixiangdejiahao]

Another failed example Humaneval/116:

def sort_array(arr):
"""
In this Kata, you have to sort an array of non-negative integers according to
number of ones in their binary representation in ascending order.
For similar number of ones, sort based on decimal value.

It must be implemented like this:
>>> sort_array([1, 5, 2, 3, 4]) == [1, 2, 3, 4, 5] # incorrect example
>>> sort_array([-2, -3, -4, -5, -6]) == [-6, -5, -4, -3, -2] # incorrect example
>>> sort_array([1, 0, 2, 3, 4]) == [0, 1, 2, 3, 4] # incorrect example
"""

The correct example should be

candidate([1,5,2,3,4]) == [1, 2, 4, 3, 5]
candidate([-2,-3,-4,-5,-6]) == [-4, -2, -6, -5, -3]
candidate([1,0,2,3,4]) == [0, 1, 2, 4, 3]