File tree 2 files changed +4
-4
lines changed
2 files changed +4
-4
lines changed Original file line number Diff line number Diff line change @@ -461,10 +461,10 @@ def _generate_data(
461
461
dataset : data .Dataset ,
462
462
index : data .Index ,
463
463
) -> Iterator [Tuple [List [int ], List [int ]]]:
464
- """Helper function to manage data encoding for SED."
464
+ """Helper function to manage data encoding for SED.
465
465
466
- We want encodings without BOS or EOS tokens . This encodes only raw
467
- source-target text for the Maxwell library.
466
+ We want encodings without padding . This encodes only raw source-target
467
+ text for the Maxwell library.
468
468
469
469
Args:
470
470
dataset (data.Dataset): dataset for generating expert vocabulary.
Original file line number Diff line number Diff line change @@ -103,7 +103,7 @@ def predict(
103
103
# Beam search.
104
104
tsv_writer = csv .writer (sink , delimiter = "\t " )
105
105
for predictions , scores in trainer .predict (model , loader ):
106
- predictions = util .pad_tensor_after_eos (predictions )
106
+ predictions = util .pad_tensor_after_end (predictions )
107
107
# TODO: beam search requires singleton batches and this
108
108
# assumes that. Revise if that restriction is ever lifted.
109
109
targets = [
You can’t perform that action at this time.
0 commit comments