Skip to content

Commit d31bd8f

Browse files
authored
Merge pull request #427 from JamesParrott/Slightly_faster
v3.1.3 speed up encoding of utf8 and ascii text
2 parents 12c1034 + 66eed73 commit d31bd8f

5 files changed

Lines changed: 200 additions & 69 deletions

File tree

README.md

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ The Python Shapefile Library (PyShp) reads and writes ESRI Shapefiles in pure Py
88

99
- **Author**: [Joel Lawhead](https://github.com/GeospatialPython)
1010
- **Maintainers**: [James Parrott](https://github.com/JamesParrott) & [Karim Bahgat](https://github.com/karimbahgat)
11-
- **Version**: 3.1.2
12-
- **Date**: 24th June 2026
11+
- **Version**: 3.1.3
12+
- **Date**: 25th June 2026
1313
- **License**: [MIT](https://github.com/GeospatialPython/pyshp/blob/master/LICENSE.TXT)
1414

1515
## Contents
@@ -93,6 +93,9 @@ part of your geospatial project.
9393

9494
# Version Changes
9595

96+
## 3.1.3
97+
- Restore faster text writing paths for single-byte Ascii encodings, and Utf-8.
98+
9699
## 3.1.2
97100
- Raise error in strict mode when creating field with a name, or writing strings, that ends with whole code points
98101
that whose encoding is pad bytes.
@@ -761,7 +764,7 @@ You can call the "fields" attribute of the shapefile as a Python list. Each
761764
field is a Python namedtuple (Field) with the following information:
762765

763766
* name: the name describing the data at this column index (a string).
764-
* field_type: a FieldType enum member determining the type of data at this column index. Names can be:
767+
* field_type: a FieldType determining the type of data at this column index. Names can be:
765768
* "C": Characters, text.
766769
* "N": Numbers, with or without decimals.
767770
* "F": Floats (same as "N").
@@ -773,9 +776,9 @@ field is a Python namedtuple (Field) with the following information:
773776
fields.
774777
* deci: Decimal length. The number of decimal places found in "Number" fields.
775778

776-
A new field can be created directly from the type enum member etc., or as follows:
779+
A new field can be created as follows:
777780

778-
>>> shapefile.Field.from_unchecked("Population", "N", 10,0)
781+
>>> shapefile.Field("Population", "N", 10,0)
779782
Field(name="Population", field_type=FieldType.N, size=10, decimal=0)
780783

781784
Using this method the conversion from string to enum is done automatically.
@@ -1454,10 +1457,16 @@ changes to the user's data, and since version 3.1 PyShp no longer does this.
14541457
False
14551458

14561459
### PyShp's approach
1457-
The simplest most robust solution to support arbitrary codec encoding as ascii bytes, is probably to encode those
1458-
bytes as Base64 (with an alphabet excluding dbf pad bytes), and then encode that Base64 string a third time as bytes.
1459-
PyShp tries to be a little easier to use than this. The intention is to adhere to [Postel's Law](https://en.wikipedia.org/wiki/Robustness_principle) so hopefully PyShp is still fairly lenient when decoding and reading Shapefiles, but either informative or
1460-
stricter, when the user attempts to encoding or write invalid files.
1460+
The simplest, most robust solution to store arbitrary codec encodings as ascii bytes, is probably to encode those
1461+
encodings as Base64 (with an alphabet excluding dbf pad bytes), and then encode that Base64 string a third time as Ascii bytes.
1462+
PyShp tries to be a little easier to use than this. The intention is to adhere to [Postel's Law](https://en.wikipedia.org/wiki/Robustness_principle) so hopefully PyShp is still fairly lenient when decoding and reading Shapefiles. But either informative or
1463+
stricter, when the user attempts to encode or write potentially invalid text data.
1464+
#### UTF-8 and Single byte codecs
1465+
UTF-8 is backwards compatible with ascii, and so far (as we know), is only known to be affected by Problem 4.
1466+
PyShp a provides a faster 'happy' path for Writers and DbfWriters using "utf8" (the default), and when text
1467+
field values are known to have been encoded, each code point to a single byte. Otherwise, a more robust
1468+
strategy is used, to handle the trickier codecs, to hopefully correctly encode data as far as possible,
1469+
otherwise to raise a warning or error.
14611470
#### Warnings
14621471
When possible data corruption is detected when reading or writing a Shapefile (or dbf file), by default
14631472
PyShp will raise a warning. Supported situations are listed below under Decoding and Encoding.

changelog.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
VERSION 3.1.3
2+
3+
2026-06-25
4+
* Restore faster text writing paths for single-byte Ascii encodings, and Utf-8.
5+
16
VERSION 3.1.2
27

38
2026-06-24

0 commit comments

Comments
 (0)