Add a WildcardAssembler Plugin. #6118

plucia-mitre · 2024-01-15T17:41:53Z

Allows assembling instructions containing wildcards. Includes support for filtering assembly results to only include some completions for each wildcard.

This starts to address issue #5800 which may also address several other existing issues including #679 and #2516.

This pull request builds on the assembler changes merged in e7458ed and was developed alongside @nsadeveloper789 (to whom we are very thankful for his support!)

This WildcardAssembler is not specific to Pickled Canary (though certainly useful for it). In addition to Pickled Canary and other assembly search tools, this assembler would be useful for tools which could now more easily surface architectural details about assembly instructions to end-users or other tools.

For more background on this pull request, see the now-closed #5801

Allows assembling instructions containing wildcards. Includes support for filtering assembly results to only include some completions for each wildcard.

plucia-mitre · 2024-01-15T17:43:11Z

Ghidra/Features/WildcardAssembler/src/main/help/help/TOC_Source.xml

It's unclear to me if I need to add something else to this file to make the documentation work properly.

Yes, you would need to add some entries to make this work. I'd say look at any project except Features Base as an example. Before you go too far, though:

Usually everything in Help is meant to describe things in the GUI. I suppose, in anticipation of a GUI, it might make sense to include this.

For API-only features, I'd recommend instead including example GhidraScripts. You can often borrow code from your unit tests, and this gives you an opportunity to demonstrate one or two use cases in prototypical fashion. I think a basic instruction pattern search could be demoed? Keep it as simple as possible, and remember what seems simple to you, the developers, may not be simple others. Even skilled users seeing this for the first time will appreciate simple examples to start.

It's true, there are parts of our Help that describe the internals of Ghidra, e.g., some of the decompiler and Sleigh documentation. If this is the approach you intend to take, please indicate in the text (html) that the feature is currently only available as an API for plugins and scripts to use. If you add example scripts, you can refer to them in the help. (Scripts are things the user can definitely see from the GUI.)

If you continue down the "Help" road, the way to validate your help is:

Run the help builder:
gradle buildHelp

Refresh all projects in Eclipse so that it copies the resources over into the build.

Run Ghidra from Eclipse, hit F1 to open help, and search for your topic.

Verify your HTML displays correctly in Ghidra's help viewer. (Sadly, Swing is stuck in HTML 3, CSS 2 times.)

I might recommend formatting your help as well. Beware that Tidy (which our formatter uses) may mangle your nested list, though, so commit first, then:

gradle WildcardAssembler:tidyHelp

Because it's Gradle, you can do the build and the tidy in the same command. Don't omit the WildcardAssembler: part, though. Doing just gradle buildHelp tidyHelp will mangle all the other projects' help. Tidying is a newer practice that not all modules follow.

Thanks! I'll do some combo of the above and we can edit from there :-)

I've added two example scripts and tweaked the documentation. I've verified that the documentation looks as expected in the generated help (Help > Contents).

I certainly could be missing something, but the gradle WildcardAssembler:tidyHelp gradle command gives me:

Cannot locate tasks that match 'WildcardAssembler:tidyHelp' as task 'tidyHelp' not found in project ':WildcardAssembler'.

Furthermore, I'm not able to find any references to 'tidyHelp' in the repo unless GitHub search is letting me down! Thanks for your ongoing help!

The tidyHelp task is simply a call to an HTML beautification library. It appears as though it is not in the repo, but instead inside of a build file that we use internally. It is not necessary for you to run this task.

That makes sense; thanks for letting me know I'm not completely crazy! 😄

plucia-mitre · 2024-01-15T22:59:21Z

For reference (though certainly not critical to this request), there's a pull request for Pickled Canary here (mitre/pickled-canary#3) which refactors Pickled Canary to use this Wildcard Assembler.

nsadeveloper789

The scripts and the help are a nice improvement! Thank you. Many coding style comments, then a few usability comments for the scripts. These aren't hard a fast "you musts," but strong suggestions. If you've already tried some of the suggestions and ran into issues, let me know.

nsadeveloper789 · 2024-01-25T18:09:04Z

.../WildcardAssembler/src/main/help/help/topics/WildcardAssemblerPlugin/Wildcard_Assembler.html

+        </DL>
+      </BLOCKQUOTE>
+
+      <P class="providedbyplugin">Provided by: <I>Wildcard Assembler Plugin</I></P>


A Plugin is an actual configurable object in the Ghidra GUI. Because this is API only, I'd just say "Wildcard Assembler Module"

nsadeveloper789 · 2024-01-25T18:12:45Z

.../WildcardAssembler/src/main/help/help/topics/WildcardAssemblerPlugin/Wildcard_Assembler.html

+    <BLOCKQUOTE>
+
+      <P><B>This feature is currently only available as an API for Ghidra
+        scripts and plugins.</B></P>


I'd go ahead and name your example scripts here, and even direct the user to find them in Ghidra's Script Manager. Something like:

For an example of how to use the API, see the FindInstructionWithWildcard and WildSleighAssemblerInfo scripts in the Script Manager.

nsadeveloper789 · 2024-01-25T18:16:36Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+	}
+
+	/**
+	 * Use an x86_64 WildSleighAssembler to assemble the given {@code wildcardedInstruction}


Should use the {@link ...} syntax on WildSleighAssembler.

nsadeveloper789 · 2024-01-25T18:19:48Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+		// Get our current program or build a new program if our current binary isn't x86_64
+		Program baseProgram = currentProgram;
+		if (!baseProgram.getLanguageCompilerSpecPair().languageID.getIdAsString()
+				.equals("x86:LE:64:default")) {


I think you should just use the current program, whatever it is. We can always warn (at least in general) that the user's mileage may vary depending on the processor language.

nsadeveloper789 · 2024-01-25T18:21:12Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+		// either "R12D" or "R13D". The instruction and/or wildcard here can be modified
+		// to an instruction found in your x86_64 binary, or patch your binary to have
+		// the bytes "0x45 0x31 0xed" somewhere.
+		var allValidResults = getAllResolvedPatterns("XOR R13D,`Q1/R1(2|3)D`");


Why not askString()? You can always include some examples in the comments above, or even include some examples in the prompt.

nsadeveloper789 · 2024-01-25T18:56:36Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+
+		println("Hit at address: " + matchAddress.toString());
+
+		// Search over all the resolutions we were searching for and find the one which matches and


"Search" is overloaded here. Consider rewording.

nsadeveloper789 · 2024-01-25T18:58:28Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+		// Search over all the resolutions we were searching for and find the one which matches and
+		// use it determine what the wildcard values are for the given hit.
+		//
+		// It'd likely be much faster the deduplicate similar WildAssemblyResolvedPatterns based on


faster ~~the~~ to deduplicate

nsadeveloper789 · 2024-01-25T19:01:42Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

+	 * {@code WildAssemblyResolvedPatterns}
+	 * 
+	 * @param results
+	 * @return


Missing descriptions for param and return

nsadeveloper789 · 2024-01-25T19:02:57Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/WildSleighAssemblerInfo.java

+		// Make sure that if one of the example instructions was chosen the current binary has the
+		// correct architecture.
+		if (sampleInstructions.contains(wildcardedInstruction) &&
+			!language.getLanguageID().toString().equals("x86:LE:64:default")) {


Ditto from the other script.

This is different from the other script because here I am just raising a warning if the user is using one of the example instructions with an unexpected architecture). I've removed the "return" statement right after this to allow us to try anyways (and updated the warning text slightly). Do you think I should do something different here?

Oh. Honestly, I missed the if statement on Line 75. I'll re-read as I check your other revisions.

nsadeveloper789 · 2024-01-25T19:15:18Z

Ghidra/Features/WildcardAssembler/ghidra_scripts/FindInstructionWithWildcard.java

Nice script. Even though it's a demo, for something provided out of the box, we should strive to make it as generally useful as possible without requiring the user to modify it. I marked particulars below as I was reading through it, but some usablility tips:

Just use the current program and its language. The assembler is supposed to accommodate all Ghidra's languages, anyway (even though we know it's not perfect.)

Allow the user to input the instruction. I see you use askString() in one script but not the other....

I had suggested using a table, but IIRC, addresses become links in the script Console? If so, then don't worry about it. Otherwise, consider just finding the next result and calling goTo, or using the table.

Ideally, the user would only need to modify the script to add a feature or change its operation, not just as a means of providing input.

nsadeveloper789 · 2024-01-25T19:38:07Z

.../WildcardAssembler/src/main/help/help/topics/WildcardAssemblerPlugin/Wildcard_Assembler.html

+      <CODE>!</CODE> character. For example, in a x86:LE:32:default binary:</P>
+      <BLOCKQUOTE>
+        <DL>
+          <DT>Assembles (no wildcard):</DT>


Probably remove the "Assembles" and parentheses for these. Just, e.g.:

No wildcard:

For the examples that doesn't work:

Single token (does not assemble):

plucia-mitre · 2024-01-31T18:25:39Z

I think I've addressed most of the above feedback but I've left a few comments on several of the points. Let me know if you'd like me to mark the other threads as resolved or leave them open for someone else to confirm my fixes.

nsadeveloper789

This looks good. Thanks for the revisions! At this point, I think any remaining revisions should be minor, and we can take care of them during our internal review process. No guarantees, but we should be good to go.

plucia-mitre · 2024-02-01T18:23:30Z

Great! Thanks! Let me know if there's anything else here, else I'll be working on cleaning up the search engine and pattern parser portions of Pickled Canary as time permits.

astrelsky · 2024-02-07T15:34:28Z

I couldn't help myself, someone had to do it.

Add a WildcardAssembler Plugin.

a0d6a5e

Allows assembling instructions containing wildcards. Includes support for filtering assembly results to only include some completions for each wildcard.

plucia-mitre commented Jan 15, 2024

View reviewed changes

This was referenced Jan 15, 2024

Initial nearly-minimal changes to assembler for Pickled Canary #5801

Closed

Refactor to work with the proposed Ghidra WildcardAssembler. mitre/pickled-canary#3

Open

ryanmkurtz assigned nsadeveloper789 Jan 16, 2024

ryanmkurtz added the Status: Triage Information is being gathered label Jan 16, 2024

Update documentation and add example scripts for WildSleighAssembler.

8470e35

nsadeveloper789 requested changes Jan 25, 2024

View reviewed changes

nsadeveloper789 added Type: Enhancement New feature or request Feature: Assembler Status: Internal This is being tracked internally by the Ghidra team and removed Status: Triage Information is being gathered labels Jan 25, 2024

nsadeveloper789 reviewed Jan 25, 2024

View reviewed changes

Address pull request feedback.

67bb4d1

nsadeveloper789 reviewed Feb 1, 2024

View reviewed changes

nsadeveloper789 added Status: Prioritize This is currently being prioritized Status: Internal This is being tracked internally by the Ghidra team and removed Status: Internal This is being tracked internally by the Ghidra team Status: Prioritize This is currently being prioritized labels Feb 1, 2024

ryanmkurtz added this to the 11.1 milestone Feb 6, 2024

ryanmkurtz closed this in 8abc0ce Feb 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a WildcardAssembler Plugin. #6118

Add a WildcardAssembler Plugin. #6118

plucia-mitre commented Jan 15, 2024 •

edited

Loading

plucia-mitre Jan 15, 2024

nsadeveloper789 Jan 16, 2024

plucia-mitre Jan 16, 2024

plucia-mitre Jan 24, 2024

dragonmacher Jan 24, 2024

plucia-mitre Jan 24, 2024

plucia-mitre commented Jan 15, 2024

nsadeveloper789 left a comment

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

plucia-mitre Jan 31, 2024

nsadeveloper789 Feb 1, 2024

nsadeveloper789 Jan 25, 2024

nsadeveloper789 Jan 25, 2024

plucia-mitre commented Jan 31, 2024

nsadeveloper789 left a comment

plucia-mitre commented Feb 1, 2024

astrelsky commented Feb 7, 2024


		println("Hit at address: " + matchAddress.toString());

		// Search over all the resolutions we were searching for and find the one which matches and

Add a WildcardAssembler Plugin. #6118

Add a WildcardAssembler Plugin. #6118

Conversation

plucia-mitre commented Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plucia-mitre commented Jan 15, 2024

nsadeveloper789 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plucia-mitre commented Jan 31, 2024

nsadeveloper789 left a comment

Choose a reason for hiding this comment

plucia-mitre commented Feb 1, 2024

astrelsky commented Feb 7, 2024

plucia-mitre commented Jan 15, 2024 •

edited

Loading