helper script for upgrading from 0.10 to 0.11 (ten2eleven.py)

[orbeckst]

Background

The 0.11.0 release will break the API in many places (see also the Release Notes for 0.11.0):

Issue	API Changes
#250 & #252	Summary of Timestep API changes
#310	frame numbers start at 0 instead of 1
#308	time is now a property of Timestep and not of the trajectory reader
#287	Summary of reorganization of modules (MDAnalysis.lib)
#372	Many AtomGroup methods were changed to properties (and some were renamed)
#376	AtomGroup.numberOfAtoms() (and similar methods for residues and segments) is now a property AtomGroup.n_atoms; likewise, Universe.trajectory.numatoms and .numframes are now .n_atoms and .n_frames, respectively.
#389	AtomGroup.selectAtoms() to AtomGroup.select_atoms() (and same for Universe.selectAtoms()
#373	calculating of bond, angle and dihedral values from 2,3, or 4-member AtomGroups will change
#385	AtomGroup.resids (and segids) will return an array the size of the AtomGroup; the old behavior will be obtained with AtomGroup.residues.resids; likewise ResidueGroup.segids will return an array the size of the ResidueGroup
#406	AtomGroup.set_<property> are now pluralized to match the property names, where they were once singular. e.g. ag.set_resnum is now ag.set_resnums, ag.set_radius is now ag.set_radii.

ten2eleven.py script

In order to ease the burden on users we would like to have a script ten2eleven.py that aids the user in the translation of Python code to the new syntax. That might not always be possible but some (tedious) changes could likely be automated.

This should be somewhat similar to the py2to3 script.

Specifications

• The script should take a python source file as input and write out an altered version of the file.

  • Optionally, it should run recursively on a directory and change all py files in the directory.
  • Optionally, it should write all changed files to a new target directory and replicate the original directory structure in the target directory.

• Automatically do substitutions and renamings like the following:

  ag.residues() → ag.residues
  atom.number → atom.index
  import MDAnalysis.core.qcprot → import MDAnalysis.lib.qcprot
  

  (We need a complete list of substitutions – please add to this issue report!)

• It would also be desirable to flag pieces of code that should be manually inspected, e.g. anything related to ts.frame. Optionally, it should insert comment lines in places that should be inspected.

• It should keep a backup file of the original file and it should never overwrite the backup files.

[orbeckst]

(Btw, looking at the code for lib2to3 I should add that my statement "ten2eleven.py should be similar to py2to3" needs to be taken with a grain of salt... 2to3 contains a lot of voodoo, including parsing of the AST to decide where to fix. But maybe one could re-used lib2to3 for our purposes...)

[orbeckst]

Instead of lib2to3 we could also look into rope, a python-based refactoring tool.

[orbeckst]

And more on using lib2to3:
Stack Overflow answer on lib2to3 Architecture Documentation points to Lennart Regebro's Porting to Python 3 book, in particular the chapter Extending 2to3 with your own fixers.

[orbeckst]

Add #389 (selectAtoms() to select_atoms()) to table.

[richardjgowers]

So the camelCase/single_underscore is a nicety, but it's not going to break any scripts, and will issue a nice message.

I think it's a lot more realistic for the script to read a script then give a line of line numbers and possible changes that need making, so something like:

ten2eleven.py my_awesome_analysis.py

line 10: 'print atom.number' - 'atom.number' has changed to 'atom.index'
line 11: 'idx = atomgroup.indices()' - 'atomgroup.indices is now a property not a method'

But if someone can make it replace things in place then that's better. Either way we'll need a list of replacements.

[orbeckst]

We definitely need the list of changes; I hope that @tylerjereddy 's page MDAnalysis 0.11.0 unifying release user guide will have this list.

I think we can write fixes for lib2to3 that do some of the replacements in a smarter way than sed. There are also recipes for rewriting import statements. However, this week I won't have the time to look at this in more detail but Extending 2to3 with your own fixers looked promising.

(rope seems to be designed to do replacements but as part of an interactive frame work, e.g. hooked into vim or emacs. The problem that I saw was that you need to know the offset in the file of the term that you want to refactor, i.e. you put the cursor mark there and then there's a function that guesses the word, passes it through the static code analyzer, and eventually makes the replacement.)

[tylerjereddy]

Let me take a look. If my progress is too slow we can always re-assign to someone else.