- Remove deprecated aws object and increase required provider
- Following input variables removed from the main module
emr_ec2_iam_policy_nameemr_managed_core_sg_nameemr_managed_master_sg_nameemr_service_access_sg_nameenable_http_port
- Following input variables removed from
aws-emr-configmodule:cluster_name
- Following input variables removed from
aws-emr-iammodule:emr_ec2_iam_policy_names3_bucket_name_for_root_directory
- Removes unused variables from examples
- Following input variables removed from the main,
aws-emr-config, andaws-emr-clustersmodulesjson_configuration_bucket_key
- Following output variables removed
- main module
json_config_s3_key
aws-emr-configmoduleemr_config_file_pathjson_config_s3_key
- main module
- Following output variables added to the main module
emr_configuration_json
- Adjusts AWS provider constraints to allow newer versions
- Updates example HBase configuration in
emr-config-template.jsonwith new root directory path
- Updates HBase configuration in
emr-config-template.jsonwith new recommended values
- Updates version file to prevent the major upgrade to the AWS provider version 4.0.
- Updates examples to use the newest version of the s3-module.
- Adds example usage of cloudwatch log collection in examples/static-hbase-logs.
- Adds example usage of cloudwatch log collection in examples/static-spark-logs.
- Deprecates
s3_policy_arnsin favor ofadditional_policy_arns.
- Adds new variable
security_configurationwhich allows specifying an existing EMR Security Configuration by name
- Switch to using instance groups instead of instance fleets when multiple master nodes specified and no spot instances
- Adds/renames variables to
aws_emr_portsmodule for differentiating between ports required for pre-6x and 6x EMR - Adds variables:
is_pre_6xmaster_ports_hbase_commonmaster_ports_hbase_pre_6xmaster_ports_hbase_6xcore_ports_emr_pre_6xcore_ports_emr_6x
- Removes variables:
master_ports_hbasecore_ports_emr
- Adds new variable
require_abac_for_subnetwhich disables ABAC for the subnet if set to false
- Adds new variable
emr_service_access_sg_idsto the nested security group module to support EMR 5.30 and newer - Updates
master_bid_price_as_percentage_of_on_demand_pricedefault value
- Adds new variable
abac_valid_tagsto be used in IAM Policies conditions for creating EMR Resources using ABAC
- Fixes an issue in the way
emr_managed_master_sg_idsandemr_managed_core_sg_idsare passed to the AWS provider that would make Terraform need to recreate the cluster in every plan
- Removes creation of security groups except for internal use within the cluster
- Removes security group input variables:
emr_managed_master_sg_idemr_additional_master_sg_idemr_managed_core_sg_idemr_additional_core_sg_idemr_service_access_sg_id
- Replaces above 5 variables with:
emr_managed_master_sg_idsemr_managed_core_sg_ids
- Adds a new ports module
- Updates examples to use new ports module
- Deprecates
additional_tagsin favor oftags
- Adds new variable
permissions_boundaryto set the permissions boundary for all IAM Roles created by the module
- Remove wildcards from IAM policies where possible
- Removes policy entirely for the ec2 role
- Open port 16030 on region servers to allow collection of HBase metrics
- Update cluster to use instance fleets, for a mix of on-demand and spot instances
- Changes variables:
- master_instance_group_name ==> master_instance_fleet_name
- core_instance_group_name ==> core_instance_group_name
- Removes variables:
- master_group_instance_count
- core_group_instance_count
- Adds variables:
- master_instance_on_demand_count
- master_instance_spot_count
- master_bid_price_as_percentage_of_on_demand_price
- master_block_duration_minutes
- master_timeout_action
- master_timeout_duration_minutes
- core_instance_on_demand_count
- core_instance_spot_count
- core_bid_price_as_percentage_of_on_demand_price
- core_block_duration_minutes
- core_timeout_action
- core_timeout_duration_minutes
- Changes output:
- core_group_instance_count ==> core_fleet_instance_count
- Adds output:
- master_fleet_instance_count
- Upgrades and pins
terraform-aws-modules/key-pair/awsto version 1.0.0
- Updates minimum AWS provider version to 3.36.0
- Sets default value for
custom_ami_idtonullto achieve the intended behavior - Updates the versions of other modules in the examples
- Adds new variables
core_bid_priceandmaster_bid_pricefor setting bid price for spot instances
- Upgrades module to require terraform 13.x
- Fixes a bug that sometimes required running apply twice when making configuration changes
- Adds new variable
arn_partitionto set the partition of any ARNs referenced in this module
- Adds new variable
custom_ami_idto set the ID of a custom Amazon EBS-backed Linux AMI
- Allows
utility_script_bucket_keyto have any name - Removes support for consistent view, which is no longer necessary since Amazon S3 supports strong read-after-write consistency.
- Removes variables:
emrfs_metadata_table_nameemrfs_metadata_read_capacityemrfs_metadata_write_capacityaws_region_of_dynamodb_table
- To upgrade, remove those variables and run
terraform applytwice. The first apply will update the configuration, and the second apply will recreate the cluster.
- Adds optional input
bootstrap_actionsto run ordered list of actions on cluster nodes before Hadoop starts. - Adds example usage of
bootstrap_actionsin examples/static-hbase
- Exposes variable json_configuration_bucket_key to make the path to the EMR configuration file settable.
- Adds new variable utility_script_bucket_key to make path of the utility script that uploads emr configuration files to s3 settable.
- Updates the config.json to match state in AWS
- Logs path in examples updated to match state in AWS
- Removes support for EMR security configuration due to issues with resource deletion
- Fixes for issues with the outputs.tf when a resource does not exist
- Upgrades the s3 module in the examples
- Adds the following module outputs -
hbase_config_path,release_label,core_group_instance_count,core_ebs_size,core_ebs_type,core_instance_type,master_instance_type,master_ebs_volumes_count,master_ebs_size,master_ebs_type,log_uri,subnet_id - Adds input
bucket_path_to_logs - Fixes bug that causes EMR cluster creation to occur prematurely before cluster configuration module has posted to S3
- Adds submodule for EMR cluster creation
- Adds submodule for cluster configuration resources
- Updates ephemeral-spark example to directly invoke submodules
- Adds
tamr_emr_cluster_nameoutput
- Fixes a bug where hbase configuration did not upload to s3
- Adds the MapReduce JobHistory server webapp port to the security group rules
- Adds http port to the security group for the master and config enable_http_port to toggle
- Refactor module to support common patterns of use
- Supports creation of infrastructure for static HBase cluster
- Supports creation of infrastruction for static Spark cluster
- Supports creation of infrastructure for ephemeral Spark cluster
- Adds example for each pattern of use
- Removes logs and root directory S3 bucket creation in root module.
- Deprecates S3 submodule.
- Adds s3_policy_arns input variable to pass bucket access policies to EMR EC2 instance profile.
- Modifies minimal example to rely on the terraform-aws-s3 module for the creation of bucket(s).
- Removes aws_account_id variable.
- Adds EMR security configuration.
- Add AES256 server side encryption on aws_s3_bucket_object resources
- Uploads HBase configuration to S3.
- Enables SSE for EMRFS data in S3.
- Enforces SSE-S3 encryption on S3 buckets.
- Adds support for EMR 5.30.0
- Moves the EMR cluster creation to the root of the module
- Adds EBS configuration variables
- Adds more hbase configuration
- Updates READMEs to use generated documentation
- More consistent formatting applied
- Makes the list of applications configurable
- Added creation of Dynamodb table for EMRFS for better control of the resources through terraform module. Previously it was controlled by EMR.
- Force destroy set to true for S3 buckets. Buckets will be deleted even when they still have objects stored in them.
- The new policy can be applied as an in-place change to existing buckets.
- Root module for EMR Hbase
- Sub modules:
- S3 module
- Security Groups module
- IAM module
- EMR Hbase cluster module