Using dumpgenerator.py with Python 3

[EndeavourAccuracy]

Python is currently at version 3.8.6 and your code requires 2.7.

As of 23 April, Ubuntu (Focal Fossa; 20.04) repos no longer carry kitchen for Python 2; only python3-kitchen.
Similarly, as of 27 June, Mint (Ulyana; 20) can no longer access this because it relies on Ubuntu repos.

As a result, modern distros can no longer use dumpgenerator.py.

I'm not a Python programmer.
Nevertheless, I've tried converting dumpgenerator.py from Python 2 to Python 3.
This attempt was unsuccessful.

I've:

• replaced print "" and print '' with print ("")
• replaced ur'' with r'' (This is for Python 3. If this needs to work with both 2 and 3, we'd apparently have to use u'' and escape any backslashes in the strings.)
• replaced cPickle with pickle, and cookielib with http.cookiejar
  But then I ran into this error, and I could not continue:
  "RecursionError: maximum recursion depth exceeded while calling a Python object"

Also, I have my own (C and PHP/JavaScript) FOSS programming projects to work on.

Can you folks work on making a version of dumpgenerator.py that works with Python 3?

[nemobis]

EndeavourAccuracy, 29/09/20 18:30:

can no longer access this because it relies on Ubuntu repos.

I'm not sure what you mean. Does Mint no longer carry pip?

[OAHOR]

For efficiently working with legacy versions of Python, it is recommended to use venv or (my personal preference) miniconda. Miniconda creates 'environments' which can contain any version of (for instance) python without affecting system python.

You can download miniconda:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
And install it:
bash Miniconda3-latest-Linux-x86_64.sh

After restarting your shell window or ssh session. you you can create a new conda environment 'wiki'
conda create -n wiki
Enter it by activating the environemnt:
conda activate wiki
Now you can install your special snowflake version of python:
conda install python=2.7

This python version 2.7 is only accessible from within the conda environment 'wiki' and does not affect the system files. The binaries are stored in your home directory, and none of this requires root access as you are not changing system files.

[jrbray1]

I was caught with this because Mint 20 does not have python2 or pip by default. It seems overkill to install them both use to get dumpgenerator.py working. Why cannot it be upgraded to a modern version?

[tiefpunkt]

@EndeavourAccuracy do you have your modifications available somewhere to work against? So we could try to fix the errors you were running into?

[EndeavourAccuracy]

@EndeavourAccuracy do you have your modifications available somewhere to work against? So we could try to fix the errors you were running into?

Not any more, no. Part of my reasoning not to keep it was that I'm not a Python programmer, and I might therefore have accidentally introduced - perhaps hard to spot - errors in the code.

[elsiehupp]

I tried following the instructions in the main README on both macOS 11 and Debian 11, and in both cases it gave me the following:

$ ./dumpgenerator.py --help
Please install the kitchen module.
Please install or update the Requests module.

...even after running $ pip install --upgrade -r requirements.txt. I assume this is because Python2 is basically EOL, and it's increasingly difficult to set up a working Python2 environment.

Anyway, I'm muddling my way through @OAHOR's instructions for miniconda, and if it's the most reliable way of running dumpgenerator.py these days, then maybe it should be added to the instructions in the main README?

[EndeavourAccuracy]

I hope this ticket won't be closed referencing a work-around to get Python 2 working on modern systems.
(Since it still won't allow using dumpgenerator.py with Python 3.)

[EndeavourAccuracy]

Here's a basic manual for creating backups without shell access, and without using dumpgenerator.py.
This is for users who do have phpMyAdmin and FTP access, and want to distribute a backup without sensitive data.
Use at your own risk.

Creating a MediaWiki Backup Without Sensitive Data, Using phpMyAdmin and FTP
Version 0.1 (June 9, 2021). Public domain.

This backup method does NOT require:
- shell access, or
- a Python 2 environment (e.g. for dumpgenerator.py).

USE THIS MANUAL AT YOUR OWN RISK

--------------------
[1/2] Database
--------------------
1. Launch phpMyAdmin.
2. On the left, click your MediaWiki database.
3. On the right, click tab "Export".
4. Select export method "Custom".
5. Optionally, unselect table "archive", which contains deleted edits. (How? Click "Select All", then Ctrl+click on "archive".)
6. Verify that section "Format-specific options" ends with "structure and data" selected.
7. Verify that section "Data dump options" uses "both of the above" as insert syntax.
8. Press the "Go" button, which will download the .sql file.
9. Remove private information from the .sql file:

Note: What you actually remove is your own decision. Below are suggestions.

Search: CREATE TABLE IF NOT EXISTS `user`
Remove: everything under "Dumping data for table `user`".
(That data could reveal the user_real_name, user_email, user_password, and user_newpassword. See, for example, "SELECT CONVERT(user_email USING utf8) FROM `user`;".)

Search: CREATE TABLE IF NOT EXISTS `watchlist`
Remove: everything under "Dumping data for table `watchlist`".
(That data could reveal which pages are watched/unwatched.)

Search: CREATE TABLE IF NOT EXISTS `recentchanges`
Remove: everything under "Dumping data for table `recentchanges`".
(That data could reveal rc_ip for each change.)

10. Done.

--------------------
[2/2] File system
--------------------
1. Download all files via FTP.
2. Remove private information:

Note: What you actually remove is your own decision. Below are suggestions.

Modify or delete LocalSettings.php.

Maybe delete directory images/archive/.

Maybe delete directory images/deleted/.

Maybe delete directory images/temp/.

Maybe delete cache/.

3. Done.

[elsiehupp]

I just made it so that my existing pull request doesn’t auto-close this issue. I’m working on a Python 3 version right now.

[elsiehupp]

Can all y’all give #409 a spin? Thanks!

[EndeavourAccuracy]

Can all y’all give #409 a spin? Thanks!

Personally, I've just moved to another method. I also lack the time to test-run the updated script, sorry. If this would've come just a bit earlier, I might have made different choices. I've been a bit surprised that so few users have made themselves heard here, even though this ticket has been open since September 2020. I'm guessing most MediaWiki admins have, and use, shell access for backups.

[elsiehupp]

That’s fine. I’ve since figured out how to run the CI tests locally, so I can do most of my testing myself.