Fixes found porting java/cpp/python (#14)

* Removed toolchain dependence on env, and simplified arg parsing, always work from source_dir

* Make sure docker images are re-generated for every PR to builder

* base dirs off source_dir, not launch_dir

* Fixed config mashing

* Added output when selecting git branch

* Reduced redundant cmake args

* Removed redundant cmake_binaries option

* Ensure that download script in docker images busts the cloudfront cache

* Added util to API

* Moved libcrypto to using python for download/tarball

* Prevent GC from destroying our late loaded classes

* Fixed env in cross-compile container

* All S3 uploads have .pyz extension

* Containers must download with .pyz extension

* Scoped build_env, pre_build_env, and post_build_env to just those steps

Author: Justin Boswell

.github/docker-images/entrypoint.sh

@@ -11,7 +11,7 @@ if [ "${args[0]}" == "--version=*" ]; then
 fi
 
 # download the version of builder requested
-curl -sSL -o /usr/local/bin/builder.pyz --retry 3 --retry-delay 3 --retry-max-time 30 https://d19elf31gohf1l.cloudfront.net/${version}/builder
+curl -sSL -o /usr/local/bin/builder.pyz --retry 3 --retry-delay 3 --retry-max-time 30 https://d19elf31gohf1l.cloudfront.net/${version}/builder.pyz?date=`date +%s`
 builder=/usr/local/bin/builder.pyz
 chmod a+x $builder
 

.github/docker-images/ubuntu-16-x64/Dockerfile

@@ -23,7 +23,7 @@ RUN apt-get update -qq \
 ###############################################################################
 # Python/AWS CLI
 ###############################################################################
-RUN python3 -m pip install --upgrade pip setuptools virtualenv \
+RUN python3 -m pip install --upgrade pip setuptools \
     && python3 -m pip install --upgrade awscli \
     && aws --version
 

.github/workflows/docker-images.yml

@@ -10,6 +10,11 @@ on:
       - '.github/docker-images/entrypoint.sh'
       - '.github/workflows/docker-images.yml'
       - '.github/workflows/*.sh'
+  # need images created at least once per branch, even if there are no docker changes
+  # so that downstream projects can use the branch channel
+  pull_request:
+    types: [opened]
+  # Make new images for every published release
   release:
     types: [published]
 
@@ -82,7 +87,7 @@ jobs:
         run: |
           mkdir -p build
           python3 -m zipapp --python="/usr/bin/env python3" --output=build/builder builder
-          aws s3 cp build/builder s3://$AWS_S3_BUCKET/${{ steps.release.outputs.release_tag }}/builder
+          aws s3 cp build/builder s3://$AWS_S3_BUCKET/${{ steps.release.outputs.release_tag }}/builder.pyz
 
       - name: Artifact builder
         uses: actions/upload-artifact@v1
@@ -102,7 +107,7 @@ jobs:
       run: |
         mkdir -p build
         python3 -m zipapp --python="/usr/bin/env python3" --output=build/builder builder
-        aws s3 cp build/builder s3://$AWS_S3_BUCKET/LATEST/builder
+        aws s3 cp build/builder s3://$AWS_S3_BUCKET/LATEST/builder.pyz
 
   linux-x86-x64:
     name: ${{ matrix.variant }}

README.md

@@ -1,18 +1,16 @@
 # AWS CRT Builder
 
-This is a central repository for build scripts and docker images for the [AWS Common Runtime Team](https://github.com/orgs/awslabs/teams/aws-sdk-common-runtime-team)
+This is a central repository for the build tool and docker images for the [AWS Common Runtime Team](https://github.com/orgs/awslabs/teams/aws-sdk-common-runtime-team)
 
-## Builder
+## Using Builder
 Builder is bundled into a zipapp. Within a given project using builder, builder.json in the root of the project will provide configuration data. 
-If you wish to add custom actions or programmatically generate data you can add python scripts in the <root>/.builder/actions directory. All 
-scripts in this directory will be loaded and scanned for classes.
+If you wish to add custom actions or programmatically generate data you can add python scripts in the <root>/.builder/ directory. All 
+scripts in this directory will be loaded and scanned for subclasses of Project, Import, and Action.
 
 ### Requirements
-* Python 3.4+
+* Python 3.5+
 * docker (if cross compiling with dockcross)
 * CMake 3.1+ (if compiling native code)
-* curl (linux only)
-* tar (linux only)
 
 ### CLI Arguments
 Usage: ```builder.pyz [build|inspect|<action-name>] [spec] [OPTIONS]```
@@ -33,29 +31,6 @@ Usage: ```builder.pyz [build|inspect|<action-name>] [spec] [OPTIONS]```
 ### Example build
 ```builder.pyz build --project=aws-c-common downstream```
 
-### Example action
-```python
-import Builder
-
-class MyAction(Builder.Action):
-    def run(self, env):
-        print('My Action did the thing')
-```
-
-This can be run with ```builder.pyz my-action``` or ```builder.pyz myaction``` or ```builder.pyz MyAction```
-
-See api.py for the available API to actions.
-
-#### Action chaining
-The ```run(self, env)``` method of any action can return an Action or list of Actions to run before considering this action complete.
-The ```Builder.Script``` class can encapsulate a list of python functions, actions, or shell commands to run. Most compound actions
-return ```Builder.Script([additional, commands, to, run])```
-
-#### The Virtual Shell
-There is a virtual shell available via ```env.shell```. It abstracts away dry run behavior, and allows for cross-platform implementations
-of common shell operations (cd, cwd, pushd, popd, setenv, getenv, pushenv, popenv, where) and the ```exec()``` function for running 
-arbitrary commands.
-
 ## Projects
 Each project is represented at minimum by its name, path on disk, and github repo url. If there is no builder.json file in the project, nor
 any python scripts to describe it, then this is enough for the builder to at least build the project, assuming it will build with defaults
@@ -78,6 +53,8 @@ that instead. There are a few external dependencies (s2n and libcrypto, for inst
 ```
 
 #### Detailed config:
+NOTE: Any key can be prefixed with a ```!``` to overwrite the config value, rather than to add to it.
+
 ```json
 {
     "name": "my-project",
@@ -128,6 +105,8 @@ that instead. There are a few external dependencies (s2n and libcrypto, for inst
         "al2": {},
         "al2012": {},
         "alpine": {},
+        "raspbian": {},
+        "manylinux": {},
         "macos": {},
         "windows": {}
     },
@@ -161,14 +140,47 @@ that instead. There are a few external dependencies (s2n and libcrypto, for inst
                 // example, disable on clang 3
                 "3": {
                     "enabled": false
-                }
+                },
             }
         }
     }
 }
 ```
 
-## Docker Images
+### Cross-compiling
+On linux and macos, builder supports cross compiling via [dockcross](https://github.com/dockcross/dockcross). It installs a small docker
+container locally that contains the toolchain and creates a shell to run commands in this environment. Builder then wraps build commands
+with that shell.
+
+## Actions
+Actions are just arbitrary python code that can be run in place of shell commands. Minimally, Actions derive from ```Builder.Action```
+and provide a ```run(self, env)``` method.
+
+### Example action
+```python
+import Builder
+
+class MyAction(Builder.Action):
+    def run(self, env):
+        print('My Action did the thing')
+```
+
+This can be run with ```builder.pyz my-action``` or ```builder.pyz myaction``` or ```builder.pyz MyAction```
+
+See api.py for the available API to actions.
+
+#### Action chaining
+The ```run(self, env)``` method of any action can return an Action or list of Actions to run before considering this action complete.
+The ```Builder.Script``` class can encapsulate a list of python functions, actions, or shell commands to run. Most compound actions
+return ```Builder.Script([additional, commands, to, run])```
+
+#### The Virtual Shell
+There is a virtual shell available via ```env.shell```. It abstracts away dry run behavior, and allows for cross-platform implementations
+of common shell operations (cd, cwd, pushd, popd, setenv, getenv, pushenv, popenv, where) and the ```exec()``` function for running 
+arbitrary commands.
+
+## Developing on builder
+### Docker Images
 Each docker image has a script which will fetch the builder app baked into it, and will then call the builder with the arguments provided.
 Any push to the .github/docker-images directory will cause a rebuild of all of the docker images (see docker-images.yml). The 
 image layers are cached, so this should be quick unless you made a fundamental modification. Any push to the builder source, or any 

builder/__main__.py

@@ -43,8 +43,6 @@ def run_action(action, env):
     config = env.config
     # Set build environment from config
     env.shell.pushenv()
-    for var, value in config.get('build_env', {}).items():
-        env.shell.setenv(var, value)
     for var, value in getattr(env, 'env', {}).items():
         env.shell.setenv(var, value)
 
@@ -104,53 +102,29 @@ def build_consumers(env):
 
 
 def default_spec(env):
-    target, arch = env.platform.split('-')
+    target, arch = current_platform().split('-')
     host = current_host()
-    compiler, version = Toolchain.default_compiler(env, target, arch)
-    print('Using Spec:')
-    print('  Host: {} {}'.format(host, current_arch()))
-    print('  Target: {} {}'.format(target, arch))
-    print('  Compiler: {} {}'.format(compiler, version))
+    compiler, version = Toolchain.default_compiler(target, arch)
     return BuildSpec(host=host, compiler=compiler, compiler_version='{}'.format(version), target=target, arch=arch)
 
 
 def inspect_host(env):
     spec = env.spec
-    toolchain = Toolchain(env, spec=spec)
+    toolchain = Toolchain(spec=spec)
     print('Host Environment:')
     print('  Host: {} {}'.format(spec.host, spec.arch))
     print('  Default Target: {} {}'.format(spec.target, spec.arch))
-    compiler_path = toolchain.compiler_path(env)
+    compiler_path = toolchain.compiler_path()
     if not compiler_path:
         compiler_path = '(Will Install)'
     print('  Compiler: {} (version: {}) {}'.format(
         spec.compiler, toolchain.compiler_version, compiler_path))
     compilers = ['{} {}'.format(c[0], c[1])
-                 for c in Toolchain.all_compilers(env)]
+                 for c in Toolchain.all_compilers()]
     print('  Available Compilers: {}'.format(', '.join(compilers)))
     print('  Available Projects: {}'.format(', '.join(Project.projects())))
 
 
-def parse_extra_args(env):
-    args = getattr(env.args, 'args', [])
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--compiler', type=str,
-                        help="The compiler to use for this build")
-    parser.add_argument(
-        '--target', type=str, help="The target to cross-compile for (e.g. android, linux-x86, aarch64)")
-    # parse the args we know, pass the rest on to actions to figure out
-    args, env.args.args = parser.parse_known_args(args)
-
-    if args.compiler or args.target:
-        compiler, version = (None, None)
-        if args.compiler:
-            compiler, version = args.compiler.split('-')
-        spec = str(env.spec) if hasattr(
-            env, 'spec') else getattr(env.args, 'spec', None)
-        env.spec = BuildSpec(compiler=compiler,
-                             compiler_version=version, target=args.target, spec=spec)
-
-
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('-d', '--dry-run', action='store_true',
@@ -165,10 +139,12 @@ def parse_args():
     parser.add_argument('--build-dir', type=str,
                         help='Directory to work in', default='.')
     parser.add_argument('-b', '--branch', help='Branch to build from')
-    parser.add_argument(
-        '--platform', help='Target platform to compile/cross-compile for', default='{}-{}'.format(current_os(), current_arch()),
-        choices=data.PLATFORMS)
     parser.add_argument('--cli_config', action='append', type=list)
+    parser.add_argument('--compiler', type=str,
+                        help="The compiler to use for this build")
+    parser.add_argument('--target', type=str, help="The target to cross-compile for (e.g. android-armv7, linux-x86, linux-aarch64)",
+                        default='{}-{}'.format(current_os(), current_arch()),
+                        choices=data.PLATFORMS)
     parser.add_argument('args', nargs=argparse.REMAINDER)
 
     # hand parse command and spec from within the args given
@@ -201,19 +177,33 @@ def parse_args():
     args, extras = parser.parse_known_args(argv)
     args.command = command
     args.cli_config = cli_config
-    if not args.spec:
-        args.spec = spec
-    args.args += extras
+    args.spec = args.spec if args.spec else spec
     # Backwards compat for `builder run $action`
     if args.command == 'run':
         args.command = args.spec
         args.spec = None
 
-    return args
+    if args.spec:
+        spec = BuildSpec(spec=args.spec, target=args.target)
+
+    if args.compiler or args.target:
+        compiler, version = (None, None)
+        if args.compiler:
+            compiler, version = args.compiler.split('-')
+        spec = str(spec) if spec else None
+        spec = BuildSpec(compiler=compiler,
+                         compiler_version=version, target=args.target, spec=spec)
+
+    if not spec:
+        spec = default_spec()
+    # Save unknown args for actions to parse later
+    args.args += extras
+
+    return args, spec
 
 
 if __name__ == '__main__':
-    args = parse_args()
+    args, spec = parse_args()
 
     if args.build_dir != '.':
         if not os.path.isdir(args.build_dir):
@@ -226,19 +216,10 @@ def parse_args():
         'args': args,
         'project': args.project,
         'branch': args.branch,
-        'platform': args.platform,
+        'spec': spec,
     })
 
-    parse_extra_args(env)
-
-    if not getattr(env, 'spec', None):
-        build_name = getattr(args, 'spec', getattr(args, 'build', None))
-        if build_name:
-            env.spec = BuildSpec(spec=build_name, platform=env.platform)
-        else:
-            env.spec = default_spec(env)
-
-    if env.platform == current_platform():
+    if env.spec.target == current_os() and env.spec.arch == current_arch():
         inspect_host(env)
     if args.command == 'inspect':
         sys.exit(0)
@@ -247,20 +228,16 @@ def parse_args():
         print('No project specified and no project found in current directory')
         sys.exit(1)
 
-    # Build the config object
-    env.config = env.project.get_config(
-        env.spec,
-        env.args.cli_config,
-        source_dir=env.source_dir,
-        build_dir=env.build_dir,
-        install_dir=env.install_dir,
-        project_dir=env.project.path)
+    print('Using Spec:')
+    print('  Host: {} {}'.format(spec.host, current_arch()))
+    print('  Target: {} {}'.format(spec.target, spec.arch))
+    print('  Compiler: {} {}'.format(spec.compiler, spec.compiler_version))
 
     if not env.config.get('enabled', True):
         raise Exception("The project is disabled in this configuration")
 
     if env.config.get('needs_compiler', True):
-        env.toolchain = Toolchain(env, spec=env.spec)
+        env.toolchain = Toolchain(spec=env.spec)
 
     if args.dump_config:
         from pprint import pprint