How to get all tokens as a list in py and c++

[sae-as-me]

Hi,
may I ask how I can use yaramod to retrieve all tokens from the yara rule file(regardless of whether the rule is valid or complete)?
For example, the contents of my file are as follows:

rule abc {
    condition:
        xyz
}

and what I hope to get is:

['rule','abc','{','\n','condition',':','\n','xyz','\n','}']

Excuse me.

[MatejKastak]

Hello @sae-as-me, thank you for the question.

You can use the following snippet to generate your output

import yaramod

# Initialize yaramod
ymod = yaramod.Yaramod()

# Parse complete YARA file
yara_file = ymod.parse_string("""rule abc {
    condition:
        true
}""")

# Get the tokens
tokens = [token.pure_text for token in yara_file.tokenstream.tokens]

# Print the tokens
print(f'Complete rule {tokens=}')

# Parse incomplete YARA file
yara_file = ymod.parse_string("""rule abc {
    strings:
        $a = "test
""", yaramod.ParserMode.Incomplete)

# Get the tokens
tokens = [token.pure_text for token in yara_file.tokenstream.tokens]

# Print the tokens
print(f'Incomplete rule {tokens=}')

This will yield the following

env λ python yaramod_tokens.py
Complete rule tokens=['rule', 'abc', '{', '\n', 'condition', ':', '\n', 'true', '\n', '}']
Incomplete rule tokens=['rule', 'abc', '{', '\n', 'strings', ':', '\n', '$a', '=']

With that being said, you should be extra careful when using yaramod.ParserMode.Incomplete as Yaramod was never intended to reliably parse malformed YARA input. You will need to test it yourself if it suits your use case.

You should also be able to replicate that for C++ fairly easily. Hope it helps.

[sae-as-me]

Hello @sae-as-me, thank you for the question.

You can use the following snippet to generate your output

import yaramod

Initialize yaramod

ymod = yaramod.Yaramod()

Parse complete YARA file

yara_file = ymod.parse_string("""rule abc {
condition:
true
}""")

Get the tokens

tokens = [token.pure_text for token in yara_file.tokenstream.tokens]

Print the tokens

print(f'Complete rule {tokens=}')

Parse incomplete YARA file

yara_file = ymod.parse_string("""rule abc {
strings:
$a = "test
""", yaramod.ParserMode.Incomplete)

Get the tokens

tokens = [token.pure_text for token in yara_file.tokenstream.tokens]

Print the tokens

print(f'Incomplete rule {tokens=}')

This will yield the following

env λ python yaramod_tokens.py
Complete rule tokens=['rule', 'abc', '{', '\n', 'condition', ':', '\n', 'true', '\n', '}']
Incomplete rule tokens=['rule', 'abc', '{', '\n', 'strings', ':', '\n', '$a', '=']

With that being said, you should be extra careful when using yaramod.ParserMode.Incomplete as Yaramod was never intended to reliably parse malformed YARA input. You will need to test it yourself if it suits your use case.

You should also be able to replicate that for C++ fairly easily. Hope it helps.

Oh, it works!
Thanks for your answer, have a nice day!

[sae-as-me]

@MatejKastak Excuse me, I have another question.
When encountering illegal or incomplete rules using the above method, the parsing work stops.
For example,

rule abc {
    strings:
        $a = "test
    condition:
        true

and using the method you provided will result in the following content:

Incomplete rule tokens=['rule', 'abc', '{', '\n', 'strings', ':', '\n', '$a', '=']

Can I obtain more tokens after that through other methods, like:

['rule', 'abc', '{', '\n', 'strings', ':', '\n', '$a', '=', '"test', '\n', 'condition', ':', '\n', 'true']

Please forgive my foolishness.

[MatejKastak]

Unfortunately, as I said above Yaramod was never intended to reliably parse malformed YARA input. It is probably aborting parsing the rule after the first error. I believe that this will require non-trivial changes to make it work as you expect.

If you want a more reliable solution, you will likely need to experiment with other parsers. My suggestion would be take a look at the following (but I haven't tried them personally with regards to incomplete parsing yet):

• https://github.com/VirusTotal/yara-x/tree/main/parser
• https://github.com/virustotal/gyp
• https://github.com/egibs/tree-sitter-yara

[sae-as-me]

Unfortunately, as I said above Yaramod was never intended to reliably parse malformed YARA input. It is probably aborting parsing the rule after the first error. I believe that this will require non-trivial changes to make it work as you expect.

If you want a more reliable solution, you will likely need to experiment with other parsers. My suggestion would be take a look at the following (but I haven't tried them personally with regards to incomplete parsing yet):

* https://github.com/VirusTotal/yara-x/tree/main/parser

* https://github.com/virustotal/gyp

* https://github.com/egibs/tree-sitter-yara

Thank you, I will learn it