== NOTE ==
The `master` branch is now used as a reference only. Active development continues in the `main` branch.
Configuration files are still uploaded to this branch until we switch to main.


This builds maas ephemeral images from candidate builds of cloud images.
Ie, it creates stuff that goes on
  http://images.maas.io/ephemeral-v3/candidate
from stuff on
  http://cloud-images.ubuntu.com/daily

== setup ==
  Run './system-setup' to set up the system.
  This will install necessary dependencies.

  the tools expect that the 'bin' dir is in $PATH.

    export PATH=$PWD/bin:$PATH

== building images ==
You can build images and kernel in one of 2 ways

=== build a single release/arch combination ===
This will build output for a single release and architecture.
You must provide a url or file to use as the source image.

  meph2-build arch release version source-image output.d

example:
  $ burl=http://cloud-images.ubuntu.com/daily/server/
  $ meph2-build --config=conf/meph-v3.yaml amd64 xenial 20160101 \
     $burl/xenial/current/xenial-server-cloudimg-amd64-root.tar.xz \
     xenial-output.d/

This will build from source-image into output.d for 'arch' and 'release'.
The 'version' provided is a serial (YYYYMMDD such as 20160101).
source-image can be either a url or a local file of type
  '.tar.gz', '-root.tar.gz', or '-root.tar.xz' or '.img'

Note, using conf/meph-v2.yaml will build the v2 format stream
and will include d-i kernels and initramfs.  To disable that, use
--disable-di.

=== syncing against cloud-images.ubuntu.com ===
This will pull the latest images down from cloud-images.ubuntu.com
and build maas images and kernel data all in one go.  It only builds
if it notices that maas images are out of date with cloud-images.
To force a build, use '--target=force'

  To run:
    meph2-cloudimg-sync out.d

    You can then add filters to the end, like 'release=trusty' or
    'arch=amd64'.  The resultant out.d/streams/v1/ will have information
    copied from maas.ubuntu.com for all other products, but will only
    create new content for products that match the provided filters.

  If the maas ephemeral data is up to date with the latest cloud image candidate
  then meph2-cloudimg-sync will not do anything.

In order to 'rebuild' something.  To create a new '20160101.1' even when
there is an existing '20160101' maas image and the latest cloud image
is 20160101, then pass '--rebuild=20160101:20160101.1'.

For faster debugging, you can disable the crawling of d-i kernel/initramfs
by passing '--disable-di'

=== debugging ===
To debug and use the fake 'maas-cloudimg2eph2', run like:
  env MAAS_CLOUDIMG2EPH2=fake-maas-cloudimg2eph2 meph2-cloudimg-sync out.d

  This will still download di-kernels and di-initramfs.

It will produce output that looks much like the final output, but
runs in seconds rather than minutes.

=== Enabling Proposed ===
For development and test purposes using Ubuntu proposed or PPA can
be very useful.

You can enable the proposed archive by simply editing the config
file conf/meph-v3.yaml and specifying '--conf=conf/meph-v3.yaml'.

    enable_proposed: True

Alternatively, you can simply pass '--proposed' to 'meph2-build' or
to 'meph2-cloudimg-sync'.

=== Adding PPAs ===
You can build images with any number of PPAs added and additional
packages installed into the image through the environment variables
M2E_ADD_REPOS and M2E_ADD_PACKAGES which are read by
maas-cloudimg2ephemeral.

M2E_ADD_REPOS is a space-delimited list of repositories to pass
to 'add-apt-repository'.  M2E_ADD_PACKAGES is a space-delimited
list of packages that will be installed into the image, and will
thus affect both the image and the initramfs produced from it.

For example:

    $ rel=xenial ; arch=amd64
    $ burl=http://cloud-images.ubuntu.com/
    $ file=$rel-server-cloudimg-amd64.squashfs
    $ wget $burl/$rel/current/$rel-server-cloudimg-amd64.squashfs -O "$file"

    $ export M2E_ADD_REPOS=ppa:smoser/lp1711760
    $ meph2-build --config=conf/meph-v3.yaml \
        $arch $rel $(date +"%Y%m%d") $file $rel-output.d/

== streams management ==
Building as above will create out.d that has *metadata* describing
everything that is in the source stream and also the things that were
just built.  data (images) will only be present for the newly built things.

So after building new data and metadata, you'll then insert the new data into
an existing stream.  This option (meph2-util insert) is additive only.


=== insert data ===
Given output of a meph2-cloudimg-sync command above in build-output.d
and local candidate stream directory candidate-stream.d you can add the new data
with:

  meph2-util insert build-output.d candidate-stream.d

If you do not want to sign (create .gpg and .sjson files), then you will
have to use '--no-sign'.

You can use '--dry-run' to only report what would be done.

=== Promoting Builds ===
Promoting is done on a specific version of a product.
You promote a candidate build to a released build with a given label.
Likely that label is 'release'.

   # list the product-ids and version are available for promotion
   $ sstream-query --output-format="%(product_name)s %(version_name)s" --max=1 \
      candidate.d/streams/v1/index.json arch=i386 | sort -u
   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-p 20140414
   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q 20140414
   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-r 20140414
   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-s 20140414
   com.ubuntu.maas.candidate:v3:boot:13.10:i386:hwe-s 20140416
   com.ubuntu.maas.candidate:v3:boot:14.04:i386:hwe-t 20140416.1

   # promote the listed version
   $ meph2-util promote --label=release \
        candidate.d/streams/v1/index.json stable.d/ 20140414 \
        product_name=com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q


== cleaning old images ==
Cleaning is done in 2 parts:
  * cleaning meta-data (removing entries from the products files)
  * finding orphans: identify which files in the tree are no longer referenced.
  * purging orphans: removing files that are known to have been orphaned for
                     a given amount of time (3 days).

=== cleaning meta-data ===

clean-md is the meph-util subcommand that allows you to keep only a given
number of builds.  For example, to keep the latest 3 versions of every
product in the meta-data, you would run:

  meph2-util clean-md --max=3 candidate-stream.d/

This *only* modifies the meta-data, it does not delete files, use find-orphans
and reap-orphans for that.

There is a --dry-run and --no-sign arguments also.

=== finding orphans ===

finding orphans creates a json formated file that contains a entry for each
file that is no longer referenced, and a timestamp of "now" (when it was first
found to be orphaned).

Example usage:
 The following will populate my-orphans.json.  Subsequent runs will update
 my-orphans.json with any new items, but will not update first-orphaned
 timestamp.

  $ meph2-util find-orphans my-orphans.json ./candidate/ ./candidate/streams/v1/*.json

More complex usage:
 Its possible that 2 streams dirs could share a data directory, similar
 to the 'pool' concept of apt for debian packages.  For example, the following
 directory tree layout (note, 'path' elements would have 'data' in them in
 this example):

   candidate/
     data -> ../data/
     streams/v1/index.json
     streams/v1/candidate.json
   stable/
     data -> ../data
     streams/v1/index.json
     streams/v1/released.json
   data/
     trusty/...
     utopic/...

 Then, we could run meph-util find-orphans with:

   $ meph2-util find-orphans my-orphans.json ./data/ \
       ./candidate/streams/v1/index.json stable/streams/v1/index.json

 That will look through files in data/ that are not referenced by path
 elements in either the candidate/streams/v1/ or stable/streams/v1/ files.

 There is no --dry-run for finding orphans as it is non-destructive.


=== cleaning / reaping orphans ===
Reaping orphans is what actually removes files.  There is a '--dry-run' to
indicate what would be done.

its usage is fairly simple.

  $ meph2-util reap-orphans --older 7d my-orphans.json ./data/

'--older' takes input like:
   Ni
 Where N is a number and i can be one of:
   d: days
   h: hours
   m: minutes
   s: seconds

The default is '3d'.

=== diff ===
It is often useful to compare the difference between two MAAS streams. While
SimpleStreams supports multiple labels in the same stream this tool is written
to work with the MAAS streams, as such it assumes each stream consistently uses
a different label. The tool will identify missing product streams, missing
products, differences in the value of keys, and missing versions. Stream and
product names have their label removed. The streams may be local or remote.
Output defaults to stdout but may be written to a file with '-o'.
'--new-versions-only' may be used to only output new versions found in the
target stream. '--latest-only' may be used to only show the latest version
missing. The YAML output may be modified and passed to the 'patch' command.

Example usage:

$ meph2-util diff --latest-only -o conf/cpc-patch.yaml \
    http://images.maas.io/ephemeral-v3/candidate \
    http://images.maas.io/ephemeral-v3/stable

== patch ===
Modifying a stream can be done using a structured YAML file. A patch may
modify metadata, add and remove products, and add or remove product versions.
Products may be specified as a regular expression. When modifying or removing
only a target stream is required. When adding a product or version a source
stream must also be given. The source stream may be local or remote over
HTTP(S). Streams are resigned after being modified unless the '--no-sign'
flag is passed. '--dry-run' may be used to test the patch without actually
modifying the stream. The YAML file references streams and products without
a label included in the name, it is automatically added based on the stream.

See doc/example-patches

Example usage:

$ meph2-util patch -i conf/cpc-patch.yaml \
    http://images.maas.io/ephemeral-v3/candidate \
    /path/to/stable

=== Known callers ===
This is just a list of known callers to reference when changing things.

 - MAAS c-i uses meph2-cloudimg-sync
   meph2-cloudimg-sync --config=conf/meph-v3.yaml \
      --target=force [--proposed] --arches=arch1,arch2 \
      --max=1 out.d/ 'release~(xenial|bionic)'
 - cloud image build process maas v2
    meph2-build --image-format squashfs-image -vv --enable-di \
       amd64 bionic 20181018 livecd.ubuntu-cpc.squashfs binary/maas/
 - cloud image build  process maas v3
    meph2-build --config conf/meph-v3.yaml \
       --image-format squashfs-image -vv \
       amd64 bionic 20181018 livecd.ubuntu-cpc.squashfs binary/maas_v3/

Trusty images are built like this:
 - v2 (jenkins_kvm/templates/img-maasv2.tmpl):
    meph2-cloudimg-sync -vvv out.d arch=amd64 release=trusty
 - v3:
    meph2-cloudimg-sync --config=conf/meph-v3.yaml -vvv out.d \
       arch=$arch release=trusty


=== Release Notifications ===

The release notification stream is used to let MAAS know when new versions have
been released so users can then be informed.

This is a bit of a non-standard way of using simplestreams as we only want to
deliver the metadata and no files. There is a requirement for a file however,
so we do include a small token file with no real content.

There will only ever be one release notification, as there is only ever one
latest version of MAAS. To insert a new release notification into the stream
the config file in `conf/release-notifications.yaml` should be updated to
reflect the new MAAS release and the message we want to show users running a
older version. The `version: 1.0` field should stay the same and should only
be incremented if we changes to the release notifications and want older MAAS
versions to ignore it.