Skip to content

Commit

Permalink
switch to the API in torchtext 0.9.0
Browse files Browse the repository at this point in the history
  • Loading branch information
Guanheng Zhang committed Feb 20, 2021
1 parent fcbeeb2 commit 2b9a280
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion examples/BERT/cross_lingual_mlm_task.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ def text_transform(x: str) -> List:

for epoch in range(1, args.epochs + 1):
train_data = CC100('/datasets01/cc100/031720/', {'*.txt'}, start_line=args.start_line, num_lines=args.num_lines)
from torchtext.experimental.datasets.raw import WikiText2
from torchtext.datasets import WikiText2
val_data, = WikiText2(data_select='valid')
val_data = [(17, item) for item in val_data if item != ' \n'] # english language type is 17 in CC100 dataset

Expand Down

0 comments on commit 2b9a280

Please sign in to comment.