Skip to content

feat(bert): pretrain and finetune implementation #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
shenxiangzhuang merged 67 commits into from
Sep 30, 2025
Merged

feat(bert): pretrain and finetune implementation #43

shenxiangzhuang merged 67 commits into from
Sep 30, 2025

Conversation

@shenxiangzhuang
Copy link
Collaborator

@shenxiangzhuang shenxiangzhuang commented Sep 9, 2025 •
edited
Loading

Close #34

@shenxiangzhuang
init bert
ccd0ca2
@shenxiangzhuang
wip: tokenizer
a03c6df
@shenxiangzhuang
marimo
051b48c
@shenxiangzhuang
wip
a47b76d
@shenxiangzhuang
wip
02452ec
@shenxiangzhuang
clean
61bce64
@shenxiangzhuang
fix
0c8899d
@shenxiangzhuang
wip: model
d5e7d0f
@shenxiangzhuang
fix model arch
b616835
@shenxiangzhuang
fix model arch
1c616e4
@shenxiangzhuang
add: dataset core process
2465aff
@shenxiangzhuang
upload dataset
26c65f7
@shenxiangzhuang
train seems works
b641a78
@shenxiangzhuang
fix: configs
7c5c265
@shenxiangzhuang
fix: loss
1182e73
@shenxiangzhuang
use 3M training data
d31f324
@shenxiangzhuang
train done: learning rate should be lower
38d12f9
@shenxiangzhuang
try overfitting
0ec8edf
@shenxiangzhuang
fix: embedding dropout
cf364bb
@shenxiangzhuang
still can not overfitting
7dcac67
@shenxiangzhuang
fix:val loss log
8c9bd78
@shenxiangzhuang
works, we can overfitting one sample, with 1e-4 lr and simple mlm head
452d9e9
@shenxiangzhuang
works: the lr is the only problem
2b253ed
@shenxiangzhuang
works too
4a81aab
@shenxiangzhuang
works
3edc4f8
@shenxiangzhuang
wip: lr warmup
1631da1
@shenxiangzhuang
start training larger dataset
ed9d4d1
@shenxiangzhuang
fix: model arch(got exactly same parameter count with huggingface tra…
bb07d94 
…nsformers lib)
@shenxiangzhuang
enable val and test
b8e2b9f
@shenxiangzhuang
fix: gelu
58ca7f0
@codecov
Copy link

codecov bot commented Sep 25, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.42%. Comparing base (1d90093) to head (3c0952b).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master      #43      +/-   ##
==========================================
+ Coverage   92.11%   94.42%   +2.30%     
==========================================
  Files           8        9       +1     
  Lines         368      520     +152     
==========================================
+ Hits          339      491     +152     
  Misses         29       29

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shenxiangzhuang shenxiangzhuang marked this pull request as ready for review September 30, 2025 01:21
@shenxiangzhuang shenxiangzhuang merged commit 460d9ad into master Sep 30, 2025
6 checks passed
@shenxiangzhuang shenxiangzhuang deleted the feat/bert branch September 30, 2025 01:22
@shenxiangzhuang shenxiangzhuang added the enhancement New feature or request label Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@shenxiangzhuang shenxiangzhuang

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

BERT(2018)

2 participants

@shenxiangzhuang