-
Notifications
You must be signed in to change notification settings - Fork 24
Open
Description
Produced checkpoints are using absolute path where they should really be using relative ones.
# ./best_dev_checkpoint
model_checkpoint_path: "/mnt/checkpoints/best_dev-221133"
all_model_checkpoint_paths: "/mnt/checkpoints/best_dev-221133"
Here is English checkpoint for reference.
# ./best_dev_checkpoint
model_checkpoint_path: "best_dev-3663881"
all_model_checkpoint_paths: "best_dev-3663881"
Fix should probably go into package.sh I think.
Related : coqui-ai/STT#2338
^ STT's lm_optimizer throws a NotFoundError from TF, since the path points to nowhere in transfer-learning configuration, as checkpoint path is /transfer-checkpoint/ instead of /mnt/checkpoints/.
commonvoice-fr/DeepSpeech/CONTRIBUTING.md
Lines 76 to 78 in 5699e59
| To use an existing checkpoint, just ensure the `docker run` includes a mount such as: | |
| `type=bind,src=PATH/TO/CHECKPOINTS,dst=/transfer-checkpoint`. Upon running, data | |
| will be copied from that place. |
commonvoice-fr/DeepSpeech/train.sh
Lines 14 to 17 in 5699e59
| if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.pb" ]; then | |
| echo "Using checkpoint from ${TRANSFER_CHECKPOINT}" | |
| cp -a /transfer-checkpoint/* /mnt/checkpoints/ | |
| fi; |
That was for DS but with STT, we have access to --load_checkpoint_dir and --save_checkpoint_dir flags instead.
if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.tflite" ]; then
echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
# use --load_checkpoint_dir for transfer learning
LOAD_CHECKPOINT_FROM="--load_checkpoint_dir /transfer-checkpoint --save_checkpoint_dir /mnt/checkpoints"
else
LOAD_CHECKPOINT_FROM="--checkpoint_dir /mnt/checkpoints/"
fi;Metadata
Metadata
Assignees
Labels
No labels