Skip to content
Casey Breen edited this page Oct 18, 2023 · 6 revisions

Welcome to the mySocialWatcher wiki!

mySocialWatcher is a dockerised scalable implementation of the pySocialWatcher package that can be used to query Meta's Marketing API perpetually across many different projects. This wiki provides descriptions of the components of mySocialWatcher and tutorials on how to use it to manage your own large scale collections.

Note: If you are looking to run one-off or short-term collections, consider simply using pySocialWatcher as a stand-alone tool. A good tutorial for pySocialWatcher is available from https://worldbank.github.io/connectivity_mapping/facebook_nbs/pipeline.html. (Please note that both the pysocialWatcher code and the specific instructions in the tutorial (e.g., how to obtain an access token) are out of date.)

Components

mySocialWatcher includes three key components:

  1. Collector
  2. Crawler
  3. Datahub

The collectors are docker containers running a slightly modified version of pySocialWatcher to query the Meta Marketing API, saving the results in the standard .csv format of pySocialWatcher. These docker containers can either be run on local machine or on a virtual machine. A virtual machine may contain multiple collectors.

The crawler periodically searches for and writes any new .csv results from all of the collectors into the centralised datahub.

The datahub is a centralised PostgreSQL database with an API that can be used to write new results into the database, to query results from the database, and to check summary statistics for recent data collected for specific projects. The datahub stores all data previously collected by all of the collectors.

Terminology

collection - A single set of queries to Meta's marketing API that retrieves counts of daily and monthly active users for specified locations, age-sex demographic groups, and other sub-population characteristics suitable for a research project. A collection is defined by one or more "specs".

virtual machine - A Virtual Machine (VM) is an emulation of a physical computer system. It operates based on the architecture and functions of a real or hypothetical computer, and enables a single physical computer to run multiple operating systems or environments.

docker container – a self-contained, portable, and isolated software unit that includes everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings. Containers provide a consistent and isolated environment for applications to run across different systems, making them highly portable and efficient.

specs - Specifications of the data to be included in a collection. The collection specs are stored in .json format. See here for examples.

credentials - Access credentials are required to query the Meta Marketing API, and they consist of an access token and an app ID. See here for instructions how to obtain new credentials.

Illustration

collector_diagram

The figure above provides a visual illustration of one potential setup for running mySocialWatcher. In this setup, we have a server with two virtual machines. Each virtual machine can have one or more collectors. The collectors pull their credentials from a master_credential.csv file.

Clone this wiki locally