Merge branch 'merge-combining-browser'

Author: jquast

README.rst

@@ -19,34 +19,40 @@ Introduction
 ============
 
 This API is mainly for Terminal Emulator implementors -- any python program
-that attempts to determine the printable width of a string on a Terminal.
+that attempts to determine the printable width of a string on a Terminal. It
+is implemented in python (no C library calls) and has no 3rd-party dependencies.
 
-It is certainly possible to use your Operating System's ``wcwidth()`` and
-``wcswidth()`` calls if it is POSIX-conforming, but this would not be possible
+It is certainly possible to use your Operating System's ``wcwidth(3)`` and
+``wcswidth(3)`` calls if it is POSIX-conforming, but this would not be possible
 on non-POSIX platforms, such as Windows, or for alternative Python
-implementations, such as jython.
-
-Furthermore, testing (`wcwidth-libc-comparator.py`_) has shown that libc
-wcwidth() is particularly out of date on most operating systems, reporting -1
-for a great many characters that are actually a displayable width of 1 or 2.
+implementations, such as jython.  It is also commonly many releases older
+than the most current Unicode Standard release files, which this project
+aims to track.
 
 The most current release of this API is based from Unicode Standard release
-_7.0.0_, dated 2014-02-28, 23:15:00 GMT [KW, LI]
+*7.0.0*, dated *2014-02-28, 23:15:00 GMT [KW, LI]* for table generated by
+file ``EastAsianWidth-7.0.0.txt`` and *2014-02-07, 18:42:08 GMT [MD]* for
+``DerivedCombiningClass-7.0.0.txt``.
+
+Installation
+------------
+
+The stable version of this package is maintained on pypi, install using pip::
+
+    pip install wcwidth
 
 Problem
 -------
 
 You may have noticed some characters especially Chinese, Japanese, and
 Korean (collectively known as the *CJK Unified Ideographs*) consume more
-than 1 terminal cell.
-
-In python, if you ask for the length of the string, ``u'コンニチハ'``
-(Japanese: Hello), it is correctly determined to be a length of **5**.
+than 1 terminal cell. If you ask for the length of the string, ``u'コンニチハ'``
+(Japanese: Hello), it is correctly determined to be a length of **5** using
+the ``len()`` built-in.
 
 However, if you were to print this to a Terminal Emulator, such as xterm,
-urxvt, Terminal.app, or PuTTY, it would consume **10** *cells* (columns) --
-two for each symbol.
-
+urxvt, Terminal.app, PuTTY, or iTerm2, it would consume **10** *cells* (columns).
+This causes problems for many of the text-alignment functions, such as ``rjust()``.
 On an 80-wide terminal, the following would wrap along the margin, instead
 of displaying it right-aligned as desired::
 
@@ -65,17 +71,10 @@ that the length of ``wcwidth(u'コ')`` is reported as ``2``, and
 This allows one to determine the printable effects of displaying *CJK*
 characters on a terminal emulator.
 
-Installation
-------------
-
-The stable version of this package is maintained on pypi, install using pip::
-
-    pip install wcwidth
-
 wcwidth, wcswidth
 -----------------
-Use ``wcwidth`` to determine the length of a single character,
-and ``wcswidth`` to determine the length of a string of characters.
+Use ``wcwidth`` to determine the length of a *single character*,
+and ``wcswidth`` to determine the length of a *string of characters*.
 
 To Display ``u'コンニチハ'`` right-adjusted on screen of 80 columns::
 
@@ -88,9 +87,9 @@ To Display ``u'コンニチハ'`` right-adjusted on screen of 80 columns::
 Values
 ------
 
-See the docstring for ``wcwidth()``, general overview of return values:
+A general overview of return values:
 
-   - ``-1``: indeterminate, such as combining_ characters.
+   - ``-1``: indeterminate (see Todo_).
 
    - ``0``: do not advance the cursor, such as NULL.
 
@@ -99,12 +98,37 @@ See the docstring for ``wcwidth()``, general overview of return values:
    - ``1``: all others.
 
 ``wcswidth()`` simply returns the sum of all values along a string, or
-``-1`` if it has occurred for any value returned by ``wcwidth()``.
+``-1`` if it has occurred for any value returned by ``wcwidth()``.  A more
+exacting list of conditions and return values may be found in the docstring
+for ``wcwidth()``.
+
+Discrepacies
+------------
+
+There may be discrepancies with the determined printable width of of characters
+by *wcwidth* and the results of any given terminal emulator -- most commonly,
+emulators are using your Operating System's ``wcwidth(3)`` implementation which
+is often based on tables much older than the most current Unicode Specification.
+Python's determination of non-zero combining_ characters may also be based on an
+older specification.
+
+You may determine an exacting list of these discrepancies using files
+`wcwidth-libc-comparator.py`_ and `wcwidth-combining-comparator.py`_
+
+.. _`wcwidth-libc-comparator.py`: https://github.com/jquast/wcwidth/tree/master/bin/wcwidth-libc-comparator.py
+.. _`wcwidth-combining-comparator.py`: https://github.com/jquast/wcwidth/tree/master/bin/wcwidth-combining-comparator.py
+
 
 ==========
 Developing
 ==========
 
+Execute the command ``python setup.py develop`` to prepare an environment
+for running tests (``python setup.py test``), updating tables (
+``python setup.py update``) or using any of the scripts in the ``bin/``
+sub-folder.  These files are only made available in the source repository.
+
+
 Updating Tables
 ---------------
 
@@ -113,7 +137,10 @@ The command ``python setup.py update`` will fetch the following resources:
     - http://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt
     - http://www.unicode.org/Public/UNIDATA/extracted/DerivedCombiningClass.txt
 
-Generating the table files `wcwidth/table_wide.py`_ and `wcwidth/table_comb.py`_.
+And generate the table files `wcwidth/table_wide.py`_ and `wcwidth/table_comb.py`_.
+
+.. _`wcwidth/table_wide.py`: https://github.com/jquast/wcwidth/tree/master/wcwidth/table_wide.py
+.. _`wcwidth/table_comb.py`: https://github.com/jquast/wcwidth/tree/master/wcwidth/table_comb.py
 
 wcwidth.c
 ---------
@@ -122,9 +149,8 @@ This code was originally derived directly from C code of the same name,
 whose latest version is available at: `wcwidth.c`_ And is authored by
 Markus Kuhn -- 2007-05-26 (Unicode 5.0)
 
-Any subsequent changes were done by directly testing against the various libc
-implementations of POSIX-compliant Operating Systems, such as Mac OSX, Linux,
-and OpenSolaris.
+.. _`wcwidth.c`: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
+
 
 Examples
 --------
@@ -133,17 +159,24 @@ This library is used in:
 
 - `jquast/blessed`_, a simplified wrapper around curses.
 
-- `jonathanslenders/python-prompt-toolkit`_, a Library for building powerful interactive command lines in Python.
+- `jonathanslenders/python-prompt-toolkit`_, a Library for building powerful
+  interactive command lines in Python.
 
 Additional tools for displaying and testing wcwidth is found in the ``bin/``
 folder of this project (github link: `wcwidth/bin`_). They are not distributed
 as a script or part of the module.
 
+.. _`jquast/blessed`: https://github.com/jquast/blessed
+.. _`jonathanslenders/python-prompt-toolkit`: https://github.com/jonathanslenders/python-prompt-toolkit
+.. _`wcwidth/bin`: https://github.com/jquast/wcwidth/tree/master/bin
+
 Todo
 ----
 
-It is my wish that `combining`_ characters are understood. Currently,
-any string containing combining characters will always return ``-1``.
+Though some of the most common ("zero-width") `combining`_ characters
+are understood by wcswidth, there are still many edge cases that need
+to be covered, especially certain kinds of sequences such as those
+containing Control-Sequence-Inducer (CSI).
 
 
 License
@@ -181,31 +214,33 @@ an OSI-approved license that appears most-alike has been chosen, the MIT license
     OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
     THE SOFTWARE.
 
-.. _`jquast/blessed`: https://github.com/jquast/blessed
-.. _`jonathanslenders/python-prompt-toolkit`: https://github.com/jonathanslenders/python-prompt-toolkit
-.. _`wcwidth/bin`: https://github.com/jquast/wcwidth/tree/master/bin
-.. _`wcwidth-libc-comparator.py`: https://github.com/jquast/wcwidth/tree/master/bin/wcwidth-libc-comparator.py
-.. _`wcwidth/table_wide.py`: https://github.com/jquast/wcwidth/tree/master/wcwidth/table_wide.py
-.. _`wcwidth/table_comb.py`: https://github.com/jquast/wcwidth/tree/master/wcwidth/table_comb.py
-.. _`combining`: https://en.wikipedia.org/wiki/Combining_character
-.. _`wcwidth.c`: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
-
 Changes
 -------
 
 0.1.4
+  * **Feature**: ``wcswidth()`` now determines printable length
+    for (most) combining characters.  The developer's tool
+    `bin/wcwidth-browser.py`_ is improved to display combining_
+    characters when provided the ``--combining`` option
+    (`Thomas Ballinger`_ and `Leta Montopoli`_ `PR #5`_).
   * added static analysis (prospector_) to testing framework.
 
 0.1.3
-  * *Bugfix*: 2nd parameter of wcswidth was not honored.
-    (`thomasballinger`_ PR #4).
+  * **Bugfix**: 2nd parameter of wcswidth was not honored.
+    (`Thomas Ballinger`_, `PR #4`).
 
 0.1.2
-  * Updated tables to Unicode Specification 7.0.0
-    (`thomasballinger`_ PR #3).
+  * **Updated** tables to Unicode Specification 7.0.0.
+    (`Thomas Ballinger`_, `PR #3`).
 
 0.1.1
   * Initial release to pypi, Based on Unicode Specification 6.3.0
 
-.. _`thomasballinger`: https://github.com/thomasballinger
 .. _`prospector`: https://github.com/landscapeio/prospector
+.. _`combining`: https://en.wikipedia.org/wiki/Combining_character
+.. _`bin/wcwidth-browser.py`: https://github.com/jquast/wcwidth/tree/master/bin/wcwidth-browser.py
+.. _`Thomas Ballinger`: https://github.com/thomasballinger
+.. _`Leta Montopoli`: https://github.com/lmontopo
+.. _`PR #3`: https://github.com/jquast/wcwidth/pull/3
+.. _`PR #4`: https://github.com/jquast/wcwidth/pull/4
+.. _`PR #5`: https://github.com/jquast/wcwidth/pull/5