Use bodyfile info to reduce file system traversals

[halpomeranz]

When UAC hits systems with large disks it can take a significant time to run. Part of the speed issue is that the program traverses most or all of the file system multiple times. We could reduce the number of file system traversals by creating a bodyfile and then deriving many of the other outputs from that information.

For example, all of the following artifacts could be derived from the body file, thereby eliminating multiple extra file system traversals:

live_response/system/socket_files.yaml
system/group_name_unknown_files.yaml
system/hidden_directories.yaml
system/hidden_files.yaml
system/sgid.yaml
system/suid.yaml
system/user_name_unknown_files.yaml         
system/world_writable_directories.yaml      
system/world_writable_files.yaml

There are probably other instances as well.

I could think of multiple possible solutions:

• Create a post-processing script separate from UAC to produce the artifacts from the bodyfile, and exclude the artifacts from the UAC profile.
• Create UAC artifacts that parse the bodyfile rather than traversing the file system.
• Modify the "find" collector to intelligently decide whether the search criteria exist in the body file, and if so get the info from there rather than a file system traversal.

Just another one of my random ideas. Feel free to ignore me. But lately I've been dealing with machines with large file systems.

[halpomeranz]

We could add this as a post-processing step in the uac script itself. Perhaps connected with a command-line option. Honestly, though, I think it makes sense to make this the default behavior and to comment-out the above artifacts in the default profiles.

[tclahr]

I was thinking about some options for this. And yes, those artifacts take forever to run, and as this information is been collected by the bodyfile artifact, we could use it to save time and resources.

One solution would be changing the _find_based_collector function to search across the bodyfile.txt file (when available). It would be challenging "translating" all find parameters/options into grep/sed/awk... But I can try.

Another solution would be changing those artifacts, adding conditions to check if the bodyfile.txt file exists, and if so, grep for the proper information. Something like this:

version: 2.1
output_directory: /system
artifacts:
  -
    description: List all world writable directories using the bodyfile.txt as the source.
    supported_os: [all]
    collector: command
    condition: if [ -s bodyfile.txt ]; then true; else false; fi
    command: grep "rwxrwxrwx" bodyfile.txt
    output_file: world_writable_directories.txt
  -
    description: List all world writable directories.
    supported_os: [all]
    collector: find
    condition: if [ -s bodyfile.txt ]; then false; else true; fi
    path: /
    file_type: [d]
    permissions: [777]
    exclude_file_system: [proc, procfs]
    output_file: world_writable_directories.txt

The first option could speed up collection by any artifact that uses find (which I like most), but, it could also miss artifacts if, for instance, the user changes the bodyfile artifact to max_depth: 2.

[halpomeranz]

I like your second option: modifying the artifacts to do the right thing if bodyfile exists. We just have to make sure that the bodyfile artifact runs early, before the other artifacts that might refer to it. The downside is re-reading the bodyfile multiple times. I've generated 3GB bodyfiles which can take a long time to read through.

Btw, since I was in a hurry and we needed it for my current investigation, I coded a modern bash script for parsing the bodyfile into the different output files: bodyfile2filelists.sh - if you run it as bodyfile2filelists.sh -U <uacdir> it will read <uacdir>/bodyfile/bodyfile.txt and output files to <uacdir>/system

I've added some additional artifacts here, per issues #417 and #418