WikiUtils

A set of utility scripts to process Wikipedia related data

parse_mysqldump

A script for parsing wikipedia mysqldump sql.gz files. Can be extended to parse arbitraty mysqldump files.

usage: parse_mysqldump.py [-h] [--column-indexes COLUMN_INDEXES]
                          filename filetype outputfile

positional arguments:
  filename              name of the wikipedia sql.gz file.
  filetype              following filetypes are supported: [categorylinks,
                        pagelinks, redirect, category, page_props, page]
  outputfile            name of the output file

optional arguments:
  -h, --help            show this help message and exit
  --column-indexes COLUMN_INDEXES, -c COLUMN_INDEXES
                        column indexes to use in output file

Inspecting dump files in BASH

Run the following command to parse the dump files in bash. The first and last lines will have some non column information.

zcat enwiki-20170920-categorylinks.sql.gz | grep $'^INSERT INTO ' | sed 's/),(/\n/g' | less -N