Added support for creating a marketplace image in Alibaba #43

russellseymour · 2017-09-07T10:22:00Z

Changes to the cookbook to allow Marketplace images to be created on the Alibaba platform.

As mentioned in chef-partners/omnibus-marketplace#93 packages need to b uploaded to the OSS service so that they can be downloaded without the danger of time out. A number of changes were required to the cookbook to support this and the Alibaba cloud. This will mean that when new version of Omnibus Marketplace, Automate and Chef Server are created they will need to be uploaded to OSS before a package can be created.

The kitchen file for Alibaba states that Packer version 1.0.4 is used. This version has the Alibaba provider contained within it. It is also now one binary instead of lots of little files.

A new attribute file for Alibaba (alibaba.rb) has been created that holds the necessary configuration. It was in this file that I tried to pass the URL for the software packages that are held in OSS, but it would not pass them through to the chef-marketplace cookbook as attributes.

There is also an attribute set here that states the URL for the Omnibus Marketplace debian file. There is a companion template install_marketplace_oss.sh.erb, but although I tried to pass in the URL as a variable in _publish.rb using the packer_provisioner resource it would not render properly hence why it is hardcode at the moment. (I really do not now why this was the case and I will try it again).

The necessary builders and products have been set for Alibaba, but in the case of the products only Automate has been configured.

The credentials for Alibaba can only be set as environment variables so this has been set in the _packer.rb file if a credential for Alibaba exists in the databag.

The format for the databag item for Alibaba is:

{
    "id": "publishing_credentials",
    "alibaba": {
        "access_key": "abcdefghijklm",
        "secret_key": "abcdefghijklm"
    }
}

I have created an image in Alibaba that has been created with this and the omnibus package.

Signed-off-by: Russell Seymour <[email protected]>

jjasghar · 2017-09-07T21:46:05Z

attributes/alibaba.rb

+  'support_email' => '[email protected]',
+  'sales_email' => '[email protected]',
+  'reporting_cron_enabled' => false,
+  'doc_url' => 'https://docs.chef.io/aws_marketplace.html',


Should we create an Alibaba site for this?

jjasghar · 2017-09-07T21:47:42Z

How is the licence done here? Is it pulled in from the cookbook? Does it default to the 30 day Automate license?

Added URLs to download products to attributes. A new packer file is created so that the Omnibus marketplace recipe reads it in to get the URLs for download Signed-off-by: Russell Seymour <[email protected]>

russellseymour · 2017-09-08T15:13:41Z

@jjasghar Not sure about that at the moment. I think it will default to the 30 day license.

Still trying to work out how all of this hangs together.

russellseymour · 2017-09-08T15:14:25Z

Added new packer_provisioner when building an image on the Alibaba platform that writes out a JSON file of the product URLs so that the omnibus_marketplace is able to download from OSS.

Signed-off-by: Russell Seymour <[email protected]>

ryancragun · 2017-09-11T21:27:52Z

.kitchen.yml

+      packman:
+        version: 1.0.4
+        checksums:
+          "1.0.4": 646da085cbcb8c666474d500a44d933df533cf4f1ff286193d67b51372c3c59e


While I think it's good that we're upgrading packer, I'd prefer to see the default version updated to 1.0.4 and breaking changes in other packer definitions fixed. We're tracking that work on: CLOUD-261 and CLOUD-262

Agreed - did not want to assume upgrading the default version whilst I was getting the initial Alibaba work going

ryancragun · 2017-09-11T21:28:25Z

attributes/alibaba.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+cred_dir = ::File.expand_path(::File.join('~', '.alibaba'))
+credential_file = ::File.join(cred_dir, 'credentials')


I think we should probably move the alibaba credentials into the same data_bag that has the Azure, GCP, and AWS creds.

I have used the same format as the data bag so it should just be a case of copying and paste from what I have. I do not know where the data bag for those is to update.

To get this working I used a local version of the data bag that kitchen could read.

ryancragun · 2017-09-11T21:31:19Z

attributes/alibaba.rb

+default['marketplace_image']['alibaba']['public']['automate']['enabled'] = false
+
+default['marketplace_image']['alibaba']['product_urls'] = {
+  'marketplace' => 'http://chef-software.oss-cn-beijing.aliyuncs.com/chef-marketplace_0.2.5%2B20170911103840.git.2.1c6a67f-1_amd64.deb',


I suspect that there's a firewall preventing the instance from accessing our package mirrors? Do we want to require the full URL or just the version of packages? What's the process of uploading a new package version to the Alibaba mirror? That should at least be documented.

It is not a preventing of getting to mirrors, it is throttling. It is so slow to download that most of the time things time out.

At the moment we need the full URL to the package that needs to be downloaded. It was the simplest way I could think to get it working.

Currently it is a case of logging into the Alibaba cloud with the account in LastPass and manually uploading. There are command line tools that allow you to this but I have not investigated thoe yet.

ryancragun · 2017-09-11T21:31:44Z

attributes/alibaba.rb

+  'role' => 'automate',
+  'platform' => 'alibaba',
+  'user' => 'ubuntu',
+  'support_email' => '[email protected]',


definitely don't want to use the aws alias for alibaba.

Agreed - my mistake I missed this.

Have changed it to [email protected] (although it will need creating on our side).

ryancragun · 2017-09-11T21:32:11Z

attributes/alibaba.rb

+  'support_email' => '[email protected]',
+  'sales_email' => '[email protected]',
+  'reporting_cron_enabled' => false,
+  'doc_url' => 'https://docs.chef.io/aws_marketplace.html',


we should probably make this specific to alibaba as well.

Have changed this to an Alibaba specific address - again it needs creating.

ryancragun · 2017-09-11T21:32:46Z

attributes/alibaba.rb

+  'sales_email' => '[email protected]',
+  'reporting_cron_enabled' => false,
+  'doc_url' => 'https://docs.chef.io/aws_marketplace.html',
+  'disable_outbound_traffic' => false,


We probably want to set this to true. Right now there are several tools that expect to be able to hit outside mirrors. Since it appears that isn't the case we'll want to set this to true.

As mentioned its not that we cannot get to external mirrors, they are just very slow.

When using the Alibaba mirrored Ubuntu repos everything is OK, packages install as expected. This is the default in the images.

ryancragun · 2017-09-11T21:33:09Z

attributes/alibaba.rb

+  'reporting_cron_enabled' => false,
+  'doc_url' => 'https://docs.chef.io/aws_marketplace.html',
+  'disable_outbound_traffic' => false,
+  'license_type' => 'flexible',


This should be fixed since we're not doing metered billing.

Modified in latest changes

Signed-off-by: Russell Seymour <[email protected]>

Changed URL for the marketplace image Signed-off-by: Russell Seymour <[email protected]>

* [CLOUD-262] packer > 0.10.1 uses its own `azure-arm` builder, obsoleting use of the `packer-azure` plugin. This requires new azure builder attributes in the `publishing_credentials.json` file. * packer >= 1.0.1 supports alicloud, enabling #43. Signed-off-by: Yvonne Lam <[email protected]>

…n Alibaba Added script to perform a dist-upgrade on Ubuntu 14.04 Signed-off-by: Russell Seymour <[email protected]>

Added support for creating a marketplace image in Alibaba

96f7764

Signed-off-by: Russell Seymour <[email protected]>

jjasghar reviewed Sep 7, 2017

View reviewed changes

Fixed issue with packer_provisioner template

90a31f7

Added URLs to download products to attributes. A new packer file is created so that the Omnibus marketplace recipe reads it in to get the URLs for download Signed-off-by: Russell Seymour <[email protected]>

Updated URL for chef-marketplace

88ef674

Signed-off-by: Russell Seymour <[email protected]>

ryancragun suggested changes Sep 11, 2017

View reviewed changes

russellseymour added 5 commits September 14, 2017 10:26

Updated URL for chef-marketplace

3bcf07e

Signed-off-by: Russell Seymour <[email protected]>

Changed support email address.

61d812e

Signed-off-by: Russell Seymour <[email protected]>

Modified Doc URL

109a88b

Signed-off-by: Russell Seymour <[email protected]>

Changed to fixed licence type

bf32d9c

Signed-off-by: Russell Seymour <[email protected]>

Added script to ensure that the ipv6 loopback is enabled

c5a2087

Changed URL for the marketplace image Signed-off-by: Russell Seymour <[email protected]>

yzl mentioned this pull request Oct 11, 2017

[CLOUD-261][CLOUD-262] Update to packer 1.1.0 #44

Merged

Changed instance size as the one we were using is now not available i…

71efa9c

…n Alibaba Added script to perform a dist-upgrade on Ubuntu 14.04 Signed-off-by: Russell Seymour <[email protected]>

Added support for creating a marketplace image in Alibaba #43

Are you sure you want to change the base?

Added support for creating a marketplace image in Alibaba #43

Uh oh!

Conversation

russellseymour commented Sep 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjasghar commented Sep 7, 2017

Uh oh!

russellseymour commented Sep 8, 2017

Uh oh!

russellseymour commented Sep 8, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants