Deploying a Hosted Site

This is a draft page documenting our process for deploying a hosted CKAN site, should become a step-by-step doc for everything for a standard hosted CKAN deployment, including setting up the servers, installing extensions, and doing basic customisations.

Create a CKAN extension for the site

Each site has a ckanext-SITE extension (where SITE is the ID of the site) in the OKFN organization on GitHub.

• Create an extension using CKAN's paster create command:

    $ paster create -t ckanext ckanext-edo
    Selected and implied templates:
      ckan#ckanext  CKAN extension project template
    
    Variables:
      egg:      ckanext_edo
      package:  ckanextedo
      project:  ckanext-edo
    Enter version (Version (like 0.1)) ['']: 
    Enter description (One-line description of the package) ['']: 
    Enter author (Author name) ['']: 
    Enter author_email (Author email) ['']: 
    Enter url (URL of homepage) ['']: 
    Enter license_name (License name) ['']: 
    Creating template ckanext
    Creating directory ./ckanext-edo
      Recursing into ckanext
        Creating ./ckanext-edo/ckanext/
        Recursing into +project+
          Creating ./ckanext-edo/ckanext/edo/
          Copying __init__.py to ./ckanext-edo/ckanext/edo/__init__.py
        Copying __init__.py to ./ckanext-edo/ckanext/__init__.py
      Recursing into ckanext_+project+.egg-info
        Creating ./ckanext-edo/ckanext_edo.egg-info/
      Copying setup.py_tmpl to ./ckanext-edo/setup.py
    Running /home/seanh/.virtualenvs/ckan/bin/python setup.py egg_info
    $ _ 
  

  See CKAN's writing extensions docs.

• cd into the extension's root dir (eg. ckanext-edo) and initialise a git repo in this dir:

    $ cd /path/to/ckanext-edo
    $ git init
  

  Use a .gitignore file in the extension's root dir to avoid committing certain unwanted files:

    *.pyc
    *~
    *.swp
    *.swo
    .DS_Store
    *.egg-info/*
  

  Now commit the gitignore file and all unignored files:

    $ git add .
    $ git commit -m "Initial commit"
  

• Create a new public repo named ckaneext-edo in the OKFN organization on GitHub, and follow the instructions on GitHub to push the ckanext-edo repo on your machine to the new github repo.

Add Custom Templates and Public Dirs

An extension can register its own templates and public dirs, and then use these to add new template and public files or to override files in CKAN.

• Create a file ckanext-edo/ckanext/edo/plugin.py with an EdoPlugin class that implements the IConfigurer interface and registers the extension's public and templates dirs:

    import ckan.plugins as plugins
    import ckan.plugins.toolkit as tk
  
  
    class EdoPlugin(plugins.SingletonPlugin):
        plugins.implements(plugins.IConfigurer)
  
        def update_config(self, config):
  
            # Add this plugin's templates dir to CKAN's extra_template_paths, so
            # that CKAN will use this plugin's custom templates.
            tk.add_template_directory(config, 'templates')
  
            # Add this plugin's templates dir to CKAN's extra_template_paths, so
            # that CKAN will use this plugin's custom templates.
            tk.add_public_directory(config, 'public')
  
            # Add this plugin's fanstatic dir.
            tk.add_resource('fanstatic', 'ckanext-datagm')
  

• Add EdoPlugin to the entry points in ckanext-edo's setup.py file:

    entry_points='''
    [ckan.plugins]
    edo=ckanext.edo.plugin:EdoPlugin
    ''',
  

• Create the templates, public and fanstatic dirs. Here we add empty .gitignore files to the dirs to allow the empty dirs to be committed to git:

    ckanext-edo $ mkdir ckanext/edo/templates
    ckanext-edo $ touch ckanext/edo/templates/.gitignore
    ckanext-edo $ mkdir ckanext/edo/public
    ckanext-edo $ touch ckanext/edo/public/.gitignore
    ckanext-edo $ mkdir ckanext/edo/fanstatic
    ckanext-edo $ touch ckanext/edo/fanstatic/.gitignore
  

• Commit everything to git and push it to githiub:

    ckanext-edo $ git add .
    ckanext-edo $ git commit -m "Add templates, public and fanstatic dirs"
    ckanext-edo $ git push
  

Enable the Plugin

Now that we've created EdoPlugin and added it to ckanext-edo's setup.py file, we can install the plugin in our virtualenv and use it in CKAN. To install the plugin run:

ckanext-edo $ python setup.py develop

To enable the plugin in CKAN, add it to the plugins list in your development.ini file, eg:

ckan.plugins = stats json_preview recline_preview datastore edo

You should now be able to run CKAN (paster serve development.ini) with your plugin enabled. The extension doesn't do anything yet, but CKAN should run without crashing.

Deploying the Site

The normal process to have one webserver which runs Apache and Nginx, and a dbserver which runs Solr and PostgreSQL. The following process is expected to be run after the bootstrap process which should install all the necessary packages.

Setting up the webserver

Deployments are owned by the okfn user on the server. If you're logged in as a different user, switch to okfn user with

sudo su okfn

Create the directories for the code, configuration, and the filestore.

sudo mkdir -p /etc/ckan/edo
sudo mkdir -p /usr/lib/ckan/edo
sudo mkdir -p /var/lib/ckan/edo
sudo chown okfn /etc/ckan/edo
sudo chown okfn /usr/lib/ckan/edo
sudo chown www-data /var/lib/ckan/edo

Setup a virtualenv in /usr/lib/ckan/edo and activate it

virtualenv --no-site-packages /usr/lib/ckan/edo
source /usr/lib/ckan/edo/bin/activate

Install ckan and switch to the correct branch

pip install -e git://github.com/okfn/ckan.git#egg=ckan
cd /usr/lib/ckan/edo/src/ckan
git branch release-v2.0

Install the ckan-edo extension (and other extensions that need to be installed).

pip install -e git://github.com/okfn/ckanext-edo.git#egg=ckanext_edo

Create a configuration file with

paster make-config ckan /etc/ckan/edo/production.ini

Edit the config file appropriately. Will need to come back to this once the database is set up.

Create /etc/ckan/edo/apache.wsgi using https://github.com/okfn/ckanbuild/blob/master/etc/ckan/apache.wsgi as a template. Correct activate_this to os.path.join('/usr/lib/ckan/edo/bin/activate_this.py')

Setting up the dbserver

The right packages should be installed on the machine already as part of the ansible process.

Clone ckanbuild

git clone https://github.com/okfn/ckanbuild.git

Create the database. A password will be prompted, use that to set the password of the edo user.

 cd ckanbuild/bin
 ./dbserver.sh createdb edo

Edit the configuration on the webserver to add access to the postgresql db just created.

Setting up the DataStore