Repository navigation
Custom features
There are several custom options and features for collecting data in pysocialwatcher. These features are built on top of the base features offered in pysocialwatcher (and haven't undergone as much systematic testing). Please open an issue if these features are not working as intended.
- Pysocialwatcher_opt
This optional feature will identify and skip sparse queries. For context, the Facebook API will return a monthly active user (MAU) estimate of 1,000 if a query is less than 1,000. This feature will deduce queries that are sparse and will skip them. For instance, if a query of Men in Angola aged 18+ is sparse, the script will deduce that the query of men in Angola aged 25-35 will also be sparse, and will skip that query. This is especially helpful in reducing the number of calls when there is a large amount of sparsity (e.g., subnational estimation).
Note: this pysocialwatcher code will skip queries where daily active users (DAU) is not sparse. In this sense, this feature is really geared towards users interested in MAU counts.
Creator: Masoomali Fatehkia (adopted for Mysocialwatcher by Casey Breen)
- Betterestimation
This customization allows for estimation of sparse MAU queries (MAU = 1,000). This method will estimate the sparse query by running two separate queries:
- Query 1: sparse query + reference query
- Query 2: reference query
The estimated MAU for the sparse query will then be: Mau Query 1 - Mau Query 2. This difference will be less than 1,000, allowing for estimation of MAU for an otherwise sparse query. This function is run after all regular queries are completed (and is compatible with pysocialwatcher_opt).
Creator: Masoomali Fatehkia (adopted for Mysocialwatcher by Casey Breen)
- Email notifications
This feature will send out an email when a collection has an error and/or when a collection finishes, depending on user preference. Multiple email addresses may be provided for each collection.
Creator: Jiani Yan
These custom features by default are turned off. To turn on these features, you will need to update the optional arguments in your collection's docker-compose.yml file. Specifically:
-
pysocialwatcher_opt_flag: "true"turns on thepysocialwatcher_optfeature -
betterestimates_flag: "true"turns on thebetterestimatesfeature -
email_notification_enable: "True"will turn on email notifications andemail_receiver: "your-email@gmail.com"will specify the email address.
Example of a collection with all features (email notifications, pysocialwatcher_opt, and betterestimates) turned on:
dgg_subnational_age_education:
container_name: dgg_subnational_age_education
volumes:
- ./dgg_subnational_age_education:/app/in
- ./../../../data/badger/dgg_subnational_age_education:/app/data
build:
args:
pysocialwatcher_opt_flag: "true"
betterestimates_flag: "true"
email_notification_enable: "True"
email_receiver: "casey.breen@demography.ox.ac.uk"
sleep_time: 0
context: ./../../../
dockerfile: ./docker/collectors/Dockerfile
restart: on-failure