Skip to content

Added support for Alibaba cloud platform #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Added support for Alibaba cloud platform #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8d5bc1c
Added support for Alibaba cloud platform
russellseymour Sep 7, 2017
1c6a67f
Releavent recipes now read in a JSON file for the URL do download Aut…
russellseymour Sep 8, 2017
4b24680
Corrected synatx error in default attributes file
russellseymour Sep 11, 2017
cce18c9
Reset cloud-init config
russellseymour Sep 14, 2017
fafb927
Enabled 30 day trial license for Alibaba as well as Azure
russellseymour Oct 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions files/chef-marketplace-cookbooks/chef-marketplace/libraries/helpers.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,16 @@ def biscotti_token_hmac
node["chef-marketplace"]["biscotti"]["token"],
node["chef-marketplace"]["biscotti"]["uuid"])
end

def download_url(product)
Copy link
Contributor

@ryancragun ryancragun Sep 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principal I like this idea assuming we're forced to put the packages in their cloud because of firewall issues. I think that we need to formarlize and unify our approach in this and the image generation.

  1. Both the upgrade and marketplace_image install should use the same mechanism for determining available package versions.
  2. Perhaps we should have a latest version that points to the latest version for each software package.
  3. Docs and/or code to upload packages and modify that config file should be present.

Alternatively, have we considered mirroring the repos in Alibaba?

Copy link
Author

@russellseymour russellseymour Sep 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a speed issue when downloading packages hence the reason for using their storage service.
For example when I did this manually it would take almost all day to download the marketplace image!

I agree with the same method for all scenarios - just trying to get it to work in the first place ;-)

The easiest approach would be to mirror into the Alibaba repos. This would keep everything as is for other cloud platforms - just need to take out the specifics for Alibaba that I have put in.

Copy link
Author

@russellseymour russellseymour Sep 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not mean the main Ubuntu repos, it would be good if we could get a packages repo in China.

Copy link
Contributor

@yzl yzl Sep 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schisamo, fyi about downloads from packages.chef.io in China.

Copy link

@schisamo schisamo Sep 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yzl There should be no technical restriction on mirroring packages.chef.io data in China, it is not on our embargoed country list:
https://github.com/chef/es-package-router/blob/05f8910a7a77904e02b4ad40e7b493c717c3ce4f/.delivery/build-cookbook/recipes/_fastly.rb#L3-L9

Copy link
Author

@russellseymour russellseymour Oct 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schisamo @ryancragun @yzl Interesting.

I am not sure what the issue is then, but there is very definite throttling going on when trying to get a package from packages.chef.io. It is so slow that it more often than not times out.

I am hoping to get some contact details for Alibaba as there is another issue I have with the approval of the image which has been rejected. (The error states that we running Windows Update will fix it!!!)

Copy link
Contributor

@yzl yzl Oct 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read about this a bit, and the possibilities that come to my mind are:

  1. The package isn't cached at whatever fastly POP is closest to Alibaba so they're actually downloading from the origin in the US,
  2. The package is cached locally, but we need some support from fastly, e.g. https://community.fastly.com/t/fastly-cdn-content-blocked-by-the-great-firewall-of-china/447
  3. The package is cached locally and fastly is set up for our particular use case, and something else is going on.

file_path = '/tmp/product_download_urls.json'
if node["chef-marketplace"]["platform"] == "alibaba" && File.exist?(file_path)
data_hash = JSON.parse(File.read(file_path))

# return the selected product url
data_hash["urls"][product]
end
end
end
end

Expand Down
4 changes: 2 additions & 2 deletions files/chef-marketplace-cookbooks/chef-marketplace/recipes/_automate_enable.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@
only_if { node['chef-marketplace']['platform'] == 'aws' }
end

# Use a 30 day trial license if we're on Azure
# Use a 30 day trial license if we're on Azure or Alibaba
cookbook_file 'var/opt/delivery/license/delivery.license' do
source 'delivery.license'
action :create_if_missing
only_if { node['chef-marketplace']['platform'] == 'azure' }
only_if { node['chef-marketplace']['platform'] == 'azure' || node['chef-marketplace']['platform'] == 'alibaba' }
end
2 changes: 1 addition & 1 deletion files/chef-marketplace-cookbooks/chef-marketplace/recipes/_config_biscotti.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
case node['chef-marketplace']['platform']
when 'google'
['Project Name', node['gce']['project']['projectId']]
when 'azure'
when 'azure', 'alibaba'
['VM Name', node['hostname']]
else # aws, testing
['Instance ID', node['ec2']['instance_id']]
Expand Down
5 changes: 4 additions & 1 deletion files/chef-marketplace-cookbooks/chef-marketplace/recipes/_publishing_enable.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,14 @@
home "/home/#{node['chef-marketplace']['user']}"
shell "/bin/bash"
action [:create, :lock]

not_if { node["chef-marketplace"]["user"] == "root" }
end

package "cloud-init" do
action :install
only_if { mirrors_reachable? }

only_if { mirrors_reachable? && node["chef-marketplace"]["platform"] != "alibaba" }
end

directory "/var/lib/cloud/scripts/per-instance" do
Expand Down
6 changes: 6 additions & 0 deletions files/chef-marketplace-cookbooks/chef-marketplace/recipes/_security.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@

node.normal["openssh"]["server"]["client_alive_interval"] = 180 if node["chef-marketplace"]["platform"] == "azure"

# Modify parameters if running on Alibaba
if node["chef-marketplace"]["platform"] == "alibaba"
node.normal["openssh"]["server"]["permit_root_login"] = "yes"
node.normal["openssh"]["server"]["challenge_response_authentication"] = "yes"
end

template "/etc/ssh/sshd_config" do
source "sshd-config.erb"
mode sshd_config_mode
Expand Down
12 changes: 12 additions & 0 deletions files/chef-marketplace-cookbooks/chef-marketplace/recipes/_upgrade_automate.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,21 @@
end

# Chef Automate

# If running on Alibaba download the Automate package for local installation
url = download_url("automate")
target_path = File.join(Chef::Config[:file_cache_path], File.basename(url))
remote_file target_path do
source url
only_if { node["chef-marketplace"]["platform"] == "alibaba" }
end

chef_ingredient 'delivery' do
action :upgrade

# Use the package sourec if this is running on Alibaba
package_source target_path if node["chef-marketplace"]["platform"] == "alibaba"

notifies :run, 'bash[delivery-ctl reconfigure]', :immediately
notifies :run, 'bash[yum-clean-all]', :immediately
notifies :run, 'bash[apt-get-clean]', :immediately
Expand Down
11 changes: 11 additions & 0 deletions files/chef-marketplace-cookbooks/chef-marketplace/recipes/_upgrade_chef_server.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,20 @@
only_if { chef_server_configured? }
end

# If running on Alibaba download the Chef Server package for local installation
url = download_url("chef_server")
target_path = File.join(Chef::Config[:file_cache_path], File.basename(url))
remote_file target_path do
source url
only_if { node["chef-marketplace"]["platform"] == "alibaba" }
end

chef_ingredient "chef-server" do
action :upgrade

# Use the package sourec if this is running on Alibaba
package_source target_path if node["chef-marketplace"]["platform"] == "alibaba"

notifies :run, "bash[chef-server-ctl reconfigure]", :immediately
notifies :run, "bash[chef-server-ctl upgrade]", :immediately
notifies :run, "bash[yum-clean-all]", :immediately
Expand Down
2 changes: 1 addition & 1 deletion files/chef-marketplace-cookbooks/chef-marketplace/templates/default/biscotti.lua.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ if ngx.var.http_x_ops_userid then
end
<% end -%>

<% if node['chef-marketplace']['platform'] == 'azure' -%>
<% if node['chef-marketplace']['platform'] == 'azure' || node['chef-marketplace']['platform'] == 'alibaba' -%>
-- When we're on Azure we don't need to validate anything so we'll just return
if true then
return
Expand Down
15 changes: 14 additions & 1 deletion ...tplace-cookbooks/chef-marketplace/templates/default/chef-marketplace-cloud-init-setup.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
#!/bin/bash

echo "mkdir: /var/opt/chef-marketplace" > /var/log/marketplace-preconfigure.log 2>&1
mkdir -p /var/opt/chef-marketplace/

echo "touch: /var/opt/chef-marketplace/cloud_init_running" >> /var/log/marketplace-preconfigure.log 2>&1
touch /var/opt/chef-marketplace/cloud_init_running
chef-marketplace-ctl setup --preconfigure

echo "sleep: 10" >> /var/log/marketplace-preconfigure.log 2>&1
sleep 10

echo "run: chef-marketplace-ctl setup --preconfigure" >> /var/log/marketplace-preconfigure.log 2>&1
chef-marketplace-ctl setup --preconfigure >> /var/log/marketplace-preconfigure.log 2>&1

echo "touch: /var/opt/chef-marketplave/preconfigured" >> /var/log/marketplace-preconfigure.log 2>&1
touch /var/opt/chef-marketplace/preconfigured

echo "rm: /var/opt/chef-marketplace/cloud_init_running" >> /var/log/marketplace-preconfigure.log 2>&1
rm -rf /var/opt/chef-marketplace/cloud_init_running