Refactor parser pipeline with streaming input #12
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix/gpo json domainsid prop
README Update
Allow schemaIDGUID to be formatted as UUID
Fix: Prevent KeyError when 'operatingsystem' is missing during service pack append
* fix: Ensure self.certtemplates exists before iterating in resolve_published_templates * update changelog
* tweak logging; start mythic parser * add quiet flag * updated ldapsearch BOF syntax in examples * fix securityidentifier in test data * uploader class to send files to BHCE * update server default, move Path() cast * readme/help edits * suppress unverified https warning * prep readme/changelog * require mythic api token instead of user/pass * prep merge
* make search for non-cobalt log files more generic * version and changelog
objectguids to uppercase
* Fix to brc4 computer object parsing for msds-allowedtodelegateto * add another variant for never expries * Check if sid and object_type are solved, bloodhound-ce import fails if either is set to None * bump version and changelog --------- Co-authored-by: Tw1sm <[email protected]>
* Bugfix - Neo.ClientError.Statement.TypeError In some cases, msds-allowedtodelegateto generates an invalid map type that breaks the Bloodhound and Neo4ldap parser. Furthermore, this information is redundant, as the attribute is also stored as AllowedToDelegate. * bump version & changelog --------- Co-authored-by: Tw1sm <[email protected]>
* Add NoneType check for x509Certificate in build_certificate_chain * bump version + changelog --------- Co-authored-by: Tw1sm <[email protected]>
…ile contents into memory
* Remove MythicCallback wrapper
- Updated attribute names in LocalGroupMembership, LocalPrivilegedSession, LocalRegistrySession, and LocalSession classes to use lowercase for consistency. - Removed the GenericParser class and replaced its functionality with specific parsers for netloggedon, netsession, netlocalgroup, and regsession BOF outputs. - Introduced new parsers: NetLocalGroupBofParser, NetLoggedOnBofParser, NetSessionBofParser, and RegSessionBofParser, each with specific start and end boundary patterns. - Enhanced ParsingPipeline to register and process new parsers, allowing for better organization and handling of parsed objects. - Updated tests to utilize the new parser structure and ensure correct parsing of BOF outputs. - Removed unused shared parsers and generic parser code to streamline the codebase.
…; add minimal OU GPLink log fixture for testing
…onsolidate local group/session object retrieval
… streamline parser registration based on type
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Refactors the parsing pipeline to use streaming input instead of loading entire files into memory. This architectural change prepares the codebase for better memory efficiency and sets the foundation for future optimizations.
Changes
DataSourceandDataStreamabstractions with generator-based processingFileDataStream,OutflankDataStream, andMythicDataStreamBoundaryBasedParserpatternParsingPipelineFactoryTesting
Notes