Skip to content

Custom features

Casey Breen edited this page Mar 21, 2024 · 3 revisions

Overview of custom features

There are several custom options and features for collecting data in pysocialwatcher. These features are built on top of the base features offered in pysocialwatcher (and haven't undergone as much systematic testing). Please open an issue if these features are not working as intended.

  1. Pysocialwatcher_opt

This optional feature will identify and skip sparse queries. For context, the Facebook API will return a monthly active user (MAU) estimate of 1,000 if a query is less than 1,000. This feature will deduce queries that are sparse and will skip them. For instance, if a query of Men in Angola aged 18+ is sparse, the script will deduce that the query of men in Angola aged 25-35 will also be sparse, and will skip that query. This is especially helpful in reducing the number of calls when there is a large amount of sparsity (e.g., subnational estimation).

Note: this pysocialwatcher code will skip queries where daily active users (DAU) is not sparse. In this sense, this feature is really geared towards users interested in MAU counts.

Creator: Masoomali Fatehkia (adopted for Mysocialwatcher by Casey Breen)

  1. Betterestimation

This customization allows for estimation of sparse MAU queries (MAU = 1,000). This method will estimate the sparse query by running two separate queries:

  • Query 1: sparse query + reference query
  • Query 2: reference query

The estimated MAU for the sparse query will then be: Mau Query 1 - Mau Query 2. This difference will be less than 1,000, allowing for estimation of MAU for an otherwise sparse query. This function is run after all regular queries are completed (and is compatible with pysocialwatcher_opt).

Creator: Masoomali Fatehkia (adopted for Mysocialwatcher by Casey Breen)

  1. Email notifications

This feature will send out an email when a collection has an error and/or when a collection finishes, depending on user preference. Multiple email addresses may be provided for each collection.

Creator: Jiani Yan

Using custom functionality

These custom features by default are turned off. To turn on these features, you will need to update the optional arguments in your collection's docker-compose.yml file. Specifically:

  • pysocialwatcher_opt_flag: "true" turns on the pysocialwatcher_opt feature
  • betterestimates_flag: "true" turns on the betterestimates feature
  • email_notification_enable: "True" will turn on email notifications and email_receiver: "your-email@gmail.com" will specify the email address.

Example

Example of a collection with all features (email notifications, pysocialwatcher_opt, and betterestimates) turned on:

dgg_subnational_age_education:
    container_name: dgg_subnational_age_education
    volumes:
      - ./dgg_subnational_age_education:/app/in
      - ./../../../data/badger/dgg_subnational_age_education:/app/data
    build:
      args:
        pysocialwatcher_opt_flag: "true"
        betterestimates_flag: "true"
        email_notification_enable: "True"
        email_receiver: "casey.breen@demography.ox.ac.uk"
        sleep_time: 0
      context: ./../../../
      dockerfile: ./docker/collectors/Dockerfile
    restart: on-failure

Clone this wiki locally