Skip to content

Create a Collection

Casey Breen edited this page Nov 12, 2023 · 10 revisions

This tutorial will walk you through the process of creating a new collection.

Meta API Credentials

If you have not already done so, create a file ./config/private/master_credentials.csv following the template provided in ./config/_example_private/credentials_master.csv. Your credentials file should contain at least one API Token and Account ID for accessing the Meta Marketing API (see here for instructions). You can store multiple credentials in your master_credentials.csv and assign them to specific collections. The "vm" column should identify the name of the collector, and the "collection" column should contain the name of the collection you want to assign a particular token to.

Create or Reuse a virtual machine

If you would like your new collection to run within an entirely new virtual machine rather than adding it to an existing virtual machine, then create a new sub-folder within ./docker/collectors/ by copying the "_example_collector" template that you will find in the same folder. Be sure to name the new folder with the name that you would like to give to your new collector.

If you would like your new collection to run within an existing virtual machine, then you can skip this step.

Script to Configure Collection

Create a Python script (.py) to configure your collection. The script should be located in ./config/collections/ and should be named for the new collection that you are creating. See ./config/collections/_example_collection.py for an example. This script should create a folder for your collection in ./docker/collectors/[collector_name]/[collection_name] that includes a file named credentials.csv with your Meta API credentials (i.e. access token and account id) as well as a .../specs/ sub-folder that contains .json files defining your collection specs.

Please see the example specs (.json) in ./config/specs to help you design good specs.

Configure docker-compose

Edit ./docker/collectors/[collector_name]/docker-compose.yml to ensure that it defines a docker container for every collection that you would like your collector to run. The docker-compose file in the example collector (./docker/collectors/_example_collector/docker-compose.yml) provides a template, and all you need to change is the name of the collector and collections ensuring that you have added a block of code for every collection.

Note that the docker-compose.yml template defines only one collection. If you would like your collector to run more than one collection, you can simply copy and paste the collection block to create a second collection. Be sure to give them different names and correctly modify their configuration to include the correct collection name.

Please see other collectors for additional examples (./docker/collectors/).

CRON Scheduler

Edit the crontab file for your collector. By default, it will schedule to launch the docker containers for your collections at midnight every day. It will also launch them if the server reboots, so that collections will continue where they left off. Note that the docker-compose up -d command will launch ALL collections defined in your docker-compose.yml. If you would like to collections to launch on different schedules, then you can add new lines into the crontab file to define cron jobs for individual collection containers using the following command to launch individual collections: docker-compose up -d [collection_name].

Shell Script to Deploy Collection

Edit the shell script ./docker/collectors/[collector_name]/deploy.sh which you will use to deploy your collections on a server (or your local machine). The shell script will execute your Python script to configure your collections, build the docker containers defined in your docker-compose (not launch them), and activate the cron scheduler to launch your collectors on a schedule. In the shell script, you need to:

  1. Redefine the VM variable to be the name of your collector,
  2. Redefine the COLLECTIONS variable to be a comma-separated list of your collection names, and
  3. Redefine the DIR variable to the path where you have cloned the mySocialWatcher repository.

Commit to GitHub

Create a new branch in the mySocialWatcher repository on GitHub and commit these changes to that branch. Once you are satisfied that all of the components are in place and working, please submit a pull request to merge your branch into the dev branch.

Allocating Tokens

To allocate API tokens, update the master_credential.csv spreadsheet (config/private/credentials_master.csv) on your local machine. Note that this file cannot go on GitHub, so you will manually secure copy the file onto the server.

Clone this wiki locally