Skip to content

protrolium/chartmetric-api-scraper

Repository files navigation

Chartmetric API scraper

Using bash shell, jq, and python to create CSV tables from returned JSON objects

The API is called using a batch curl request with two variables referencing artist name and artist id tables (not included in this repo).

Those raw JSON are then iterated over using jq filters setup for each type of requested object, creating new filtered JSON of only the desired key : value pairs.

Python is used to create and segment out individual tables, with headers based off of the dict keys. For batch processing:
$ find ./ -maxdepth 1 -type f -name "*.json" -exec ./chartmetric.py "{}" ../../temp \;

The generated objects can then be merged by awk to create a merged spreadsheet of data.
$ awk 'FNR == 1 && NR!=1{next;}{print}' *_metric.csv > merged.csv

About

working directory for JSON > CSV workflow

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published