Skip to content

Partial lexer implementation #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 24 commits into from
Jan 14, 2025
Merged

Partial lexer implementation #8

merged 24 commits into from
Jan 14, 2025

Conversation

@wpkelso
Copy link
Owner

@wpkelso wpkelso commented Jan 14, 2025

Adds partial support for lexing token chains that include the AND, ADD, LD, LDI, LDR, LEA, ST, STI, STR, and NOT instructions, as well as passing tests for all but NOT. Merging this unblocks working on the assembler frontend, but does not guarantee compatibility with all (indeed most) programs.

Adds strum as a dependency to ease enum matching

wpkelso and others added 24 commits December 20, 2024 13:26
@wpkelso
add foundational structures for code->machine translation
fde1fe4
@wpkelso
Merge branch 'main' into assembler
68bc5a3
@wpkelso
lay tokenizer groundwork
c09d59f
@wpkelso
Switch to using regex
8fba3da
@wpkelso
add all regex strings
f175c9e
@wpkelso
add some matching and a failing test
a625f0e
@wpkelso
cleaned up register matching logic
85d5f51
@wpkelso
add tests for comments and unclean register strings
f26d5ff
@wpkelso
instruction tokenization and test for add
9312b4b
@wpkelso
tokenize pseudoops
166b73f
@wpkelso
return vector of tokens
921f5dd
@wpkelso
split into multiple files
217288b
@wpkelso
num tokens and tests
0dae0e3
@wpkelso
add prefix label pass for lexer
1d9ce2c
@wpkelso
Add test for stripping prefix labels
bfae0de
@wpkelso
Add wrapper function for lexer interface
469ab05
@wpkelso
token validation fn shell & fn inlining
06e7699
@wpkelso
Implementation up to instruction formation, then stuck
c0007c8
@wpkelso
use strum macros instead of shell tokens
0ba9d4e
@Bennett-Petzold
Initial lexer skeleton
e5cd945
@Bennett-Petzold
Merge branch 'main' into assembler
c28b353
@wpkelso
add load lexing
c00c7e8
@wpkelso
add all multi-field instructions to lexer
e418601
@wpkelso
Add tests for currently implemented instructions
b4c9da9
github-advanced-security[bot]
github-advanced-security bot found potential problems Jan 14, 2025
View reviewed changes
src/assembler/lexer.rs
@@ -0,0 +1,327 @@
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
@@ -0,0 +1,327 @@
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
defs::{LC3Word, RegAddr},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
defs::{LC3Word, RegAddr},
instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
defs::{LC3Word, RegAddr},
instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
defs::{LC3Word, RegAddr},
instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
use crate::{
assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
defs::{LC3Word, RegAddr},
instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},

Check warning

Code scanning / clippy

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE Warning

unused imports: ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, PseudoOp, RegAddr, and TRAP_OPCODE
src/assembler/lexer.rs
instr: &mut MaybeUnresolvedInstr,
) -> Result<(), anyhow::Error> {
if let Token::REGISTER(reg) = token {
instr.value |= (LC3Word::from(*reg) << SHIFT);

Check warning

Code scanning / clippy

unnecessary parentheses around assigned value Warning

unnecessary parentheses around assigned value
src/assembler/lexer.rs
instr: &mut MaybeUnresolvedInstr,
) -> Result<(), anyhow::Error> {
if let Token::REGISTER(reg) = token {
instr.value |= (LC3Word::from(*reg) << SHIFT);

Check warning

Code scanning / clippy

unnecessary parentheses around assigned value Warning

unnecessary parentheses around assigned value
src/assembler/lexer.rs
instr: &mut MaybeUnresolvedInstr,
) -> Result<(), anyhow::Error> {
if let Token::REGISTER(reg) = token {
instr.value |= (LC3Word::from(*reg) << SHIFT);

Check warning

Code scanning / clippy

unnecessary parentheses around assigned value Warning

unnecessary parentheses around assigned value
src/assembler/lexer.rs
instr: &mut MaybeUnresolvedInstr,
) -> Result<(), anyhow::Error> {
if let Token::REGISTER(reg) = token {
instr.value |= (LC3Word::from(*reg) << SHIFT);

Check warning

Code scanning / clippy

unnecessary parentheses around assigned value Warning

unnecessary parentheses around assigned value
@wpkelso wpkelso merged commit bb90fa6 into main Jan 14, 2025
4 of 9 checks passed
@wpkelso wpkelso deleted the assembler branch January 14, 2025 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Bennett-Petzold Bennett-Petzold Bennett-Petzold approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wpkelso @Bennett-Petzold