Release candidate: 8.0.0-rc1

About this pre-release

I've been suffering from a health problem that makes it hard for me to do any serious brain work. The 8.0 code has been close to a state of completion for months now, but I didn't trust myself to attempt a release.

Friends suggested a Release Candidate, and... here it is!

This is a big change. Some of your code may need changing:

• Items that have been deprecated for years are now gone.
• Other items have become deprecated.
• There are now nicer ways of doing some very basic things.

Before I go into the details though, let me thank our sponsors: @Laro88, CoffeeSprout, @klaussilveira, and @Tosenaeus. It is wonderful to see all the work truly appreciated.

You too can sponsor the work on libpqxx, by the way: monthly or one-off contributions on Github, or through Buy Me A Coffee.

That said, let's go over what's changed in this release.

Changes in 8.0

Many things have changed. Most code written against 7.x will still work, but there may now be nicer ways of doing things. Some of your code may need updating, and I'm sorry about that! But I hope you'll be happy overall.

Let's go through the main things.

C++20

You will need at least C++20 to compile libpqxx. Support does not need to be absolutely complete, but the basics must be there: concepts, various utility functions in the standard library, the language's new object lifetime rules, and so on.

Thanks to std::span, there are now very few raw pointers in the API (or even internally). This makes libpqxx not only safer, but also easier to prove safe. It is important to be able to run static analysis tools that may detect bugs and vulnerabilities. But, in the C++ tradition, it shouldn't cost us anything in terms of speed.

Retiring deprecated items

Some types and functions were already deprecated and are now gone:

• binarystring. Use blob instead.
• connection_base was an alias. Use just regular connection.
• encrypt_password(). Use the equivalent member functions in connection.
• dynamic_params. Use params.
• stream_from. Use transaction_base::stream() instead.
• Some stream_to constructors. Use static "factory" member functions.
• transaction_base::unesc_raw(). Use unesc_bin().
• transaction_base::quote_raw(). Use quote(), passing a bytes_view.
• Some field constructors.
• Result row splicing. I don't think anyone ever used it.

Result iterators

Lifetimes: result and row iterators no longer reference-count their result object. In libpqxx 7, a result object's data would stay alive in memory for as long as you had an iterator referring to it. It seemed like a good idea once, many years ago, but it made iteration staggeringly inefficient.

So, that's changed now. Just like any other iterator in C++, result and row iterators no longer keep your container alive. It's your own problem to ensure that. However result objects are still reference-counted smart pointers to the underlying data, so copying them is relatively cheap.

Semantics: Iterators for the result and row classes have never fully complied with the standard requirements for iterators.

There was a good reason for this: I wanted these iterators to be convenient and work a lot like C pointers. I wanted you to be able to dereference them as if a result object were just an array of pointers to arrays.

But it really started getting in the way. Much as I hate it, the standard requirements say that for any iterator i, i[n] should mean *(i + n). In libpqxx, if you had a result iterator i, i[n] meant "field n in the row to which i points." Really convenient, but not compliant with the standard.

So, that is no longer the case. If you want "field n in the row to which i points," say (*i)[n] or i->at(n).

By the way, these changes mean that result and row iterators are now proper, standard random-access iterators. Some algorithms from the standard library may magically become more efficient when they detect this.

Comparing results, rows, and fields

When you compare two result, row, or field objects with == or !=, it now only checks whether they refer to (the same row/field in) the same underlying data object. It does not look at the data inside.

These comparisons were meant to be helpful but they were never very well defined. And if you don't know exactly what you're getting, why would you want to invest the compute time?

Row and field references

The row and field classes were cumbersome, inefficient, and hopelessly intertwined with iterators.

To avoid all that, use row_ref instead of row and field_ref instead of field. These assume that you keep the original result around, and in a stable location in memory. Unlike row and field, they do not keep the
underlying data object alive through reference-counting. But neither do any of
the standard C++ containers, so I hope you'll find this intuitive. It's certainly more efficient.

The indexing operations now return row_ref and field_ref instead of row and field respectively. It probably won't affect your code, but you're running static analysis and instrumented builds to check for these things, right?

Binary data

As the documentation predicted, the alias pqxx::bytes has changed to stand for std::vector<std::byte>. It used to be std::basic_string<std::byte>. And pqxx::bytes_view is now an alias for std::span<std::byte>.

This may require changes to your code. The APIs for std::basic_string and std::vector differ, perhaps in more places than they should. Do not read the data using the c_str() member function; use data() instead.

Hate to do this to you. However there were real problems with using std::basic_string the way we did. The basic_string template wasn't built for binary data, and there was no guarantee that it would work with any given compiler. Even where it did, we had to work around differences between compilers and compiler versions. That's not healthy.

But there's also good news! Thanks to C++20's Concepts, most functions that previously only accepted a pqxx::bytes argument will now accept just about any block of binary data, such as std::array<std::byte> or a C-style array.

String conversion API

This is a whole chapter in itself. If you were specialising pqxx::type_name or pqxx::string_traits, things just got simpler, safer, and more efficient. But there's also a change in lifetime rules. This can be important for writing safe and correct code, so read on!

Type names

Let's get the easiest part out of the way first: type_name is now deprecated. Instead of specialising type_name, you now specialise a function called name_type(). It returns a std::string_view, so the underlying data can be a string literal instead of a std::string. Some static analysis tools would report false positives about static deallocation of the type_name strings.

Defining a string conversion

Then there's pqxx::string_traits. Several changes here. If you were defining your own conversions to/from SQL strings, the 8.x API is...

1. leaner, losing several members but mainly into_string().
2. safer, using views and spans rather than raw C-style pointers.
3. simpler, dropping the terminating zero at the end of various strings.
4. friendlier, accepting std::source_location for better error reporting.
5. richer, capable of dealing with different text encodings.
6. faster, because of the lifetime rule changes (described below).

Your existing conversions may still work without changes, but that's only thanks to some specific compatibility shims. These will go away in libpqxx 9 at the latest, so I urge you to update your code.

Lifetime rule changes

In libpqxx 7.x, when you converted a C++ value to an SQL string or vice versa, you could count on having a valid output for at least as long as you kept the object in which you stored it. The details depend on the type, e.g. pqxx::string_traits<bool>::to_string() returns a string constant, either "true" or "false".

We're tightening that up in 8.x. Now, the output remains valid for at least as long as you keep the object in which you stored it, and also keep the original in place. This streamlines some trivial conversions. For example, converting a std::string_view or a C-style string to SQL is now just a matter of returning a view on that same data. It's just the same data in both C++ and SQL.

It also enables some conversions that weren't possible before. You can now convert an SQL string to a std::string_view or a C-style string pointer, because the conversion is allowed to refer to its input data.

Most code won't need to care about this change. A calling function is usually either done very quickly with its converted value, or it immediately arranges for more permanent storage. If you call pqxx::to_string() for example, you get a std::string containing the SQL string. All that changes in that case is the conversion process skipping an internal copy step, making it a bit faster.

Using conversions

There is now a richer, more complete set of global conversion functions. This means you can convert C++ values to SQL strings without calling their pqxx::string_traits functions yourself.

The options are, from easiest to most efficient:

• pqxx::to_string() returns the SQL value as a std::string.
• pqxx::into_buf() renders the SQL v...

7.10.3: Fix MSVC C++17 build

This is a quick update to 7.10.2. It fixes some problems with building libpqxx as C++17 in Microsoft Visual Studio.

The main changes however are what you already got with 7.10.3:

• Fixes to CMake & pkgconfig setup.
• Better handling of weirdness in connection strings.
• Can't pass parameters to stream().
• row::as_tuple() is now public.
• Fixed potential garbage at end of type names in error messages.
• Build fix on macOS related to std::free().
• Fix to workaround when compiler does not support to_chars() for floating point.
• Avoid crash in affected_rows() for empty result.
• Work with compilers that support concepts but not ranges.

7.10.2: Many little fixes

There's no big news in this release... unless one of these happened to affect you:

• CMake used unnecessary, broken PostgreSQL_INCLUDE_DIRS.
• pqxx::connection::connection_string() did not handle quotes and spaces correctly.
• Parameter-passing in stream() was pointless: the underlying protocol doesn't support it.
• Static linkage issue with pkgconfig.
• If you need pqxx::row::as_tuple()... it's public now.
• Demangled type names could have some garbage at the end.
• Build broke in compilers where std::free was not noexcept.
• Floating-point string conversion was broken in compilers that don't support std::to_chars().
• Failure in pqxx::result::affected_rows() when result set is empty.
• Error message in pqxx::result::expect_columns() wasn't quite right.
• Build failed on compilers that support concepts but not ranges.
• Some compilers with some settings issued deprecation warnings for the <ciso646> header.

These have all been fixed. No new problems have come up in a while so it's time to release the fixes, and focus on creating exciting new problems in the 8.0 release branch. :-)

Overall, libpqxx represents... probably more than a decade of unpaid full-time development. You can now support the work on BuyMeACoffee or Github.

libpqxx 7.10.1: Fixes.

This release fixes various problems in 7.10.0. I've been focused on the 8.0 release, which is moving steadily now but it's also just a lot of work.

Improvements in 7.10.1:

• Fix string conversion buffer budget for arrays containing nulls. (#921)
• Remove -fanalyzer option again; gcc is still broken.
• Oops, no, minimum CMake version is not 3.28, but 3.12!
• Fix warnings on compilers that accept [[assume]] with a warning. (#928)
• Can't pass parameters to streaming query.
• CMake build: take libpq path from build target, not absolute path. (#964)
• Added explanation to listen() of when notifications come in. (#963)

7.10.0

libpqxx 7.10.0: Bug fixes, build fixes, and API overhauls

This is about as much of the radical changes I can make before moving on 8.0 (which will require C++20 as a minimum).

I'll start with the fixes, because some of them are important:

• Since macOS doesn't have /bin/true, we'll just call true instead (#885).
• Error reporting could crash during non-blocking connection construction (#894).
• There was a potential buffer overrun when converting an array containing nulls to an SQL string (#906).
• The nullness check for std::optional was broken if the contained type had its own null value (#907).
• An error message for clashes between transaction focuses was misleading (#879).
• Minimum CMake version is now 3.28 3.12 (#851, #874).

Next, we get to the API overhauls!

SQL execution functions go "orthogonal"

There was an enormous and growing body of special functions for executing SQL and getting a result back: Execute SQL statement. Execute statement and expect 0 rows of data. Execute statement and expect 1 row of data (returning the pqxx::row instead of the full pqxx::result). Execute statement and expect n rows of data. All the same functions but with parameters. All the same functions again but with a prepared statement. Not all combinations were actually implemented. And that's not even talking about the various streaming versions.

I didn't touch the streaming calls, but the execute-and-get-a-result functions are more manageable now:

• All of the execution functions are called pqxx::transaction_base::exec(). The arguments determine which version you mean.
• The difference between a regular statement and a parameterised statement is now simply whether you pass a pqxx::params argument.
• A prepared statement differs in that you wrap the statement name in a pqxx::prepped object, to show that it's not an SQL string itself.
• If you want to check for a specific number of rows, you do that on the result: tx.exec("VACUUM mytable").no_rows(); or my_row = tx.exec("SELECT 1").one_row(); and so on.

The old functions, such as exec0() and exec1() and exec_prepared() etc. will still be there for the 8.0 release cycle (as well as the rest of 7.x), but marked as deprecated. They'll be gone in 9.0. Trust me, you won't miss them!

Error/notice handlers get with the times

There's a new mechanism for receiving error, warning, and general notice messages from libpq. Where previously you would derive your own class from pqxx::errorhandler and implement your own virtual function call operator, you can now just register a lambda using a new function, pqxx::connection::set_notice_handler(). Or a callable object. Or a plain old function. Anything that goes in a std::function. As you can see they're called notice handlers now.

This is a whole new mechanism. It's less powerful in one way: it does not support chains of handlers. You can install at most one notice handler on a connection. If you install a new one, it overwrites the old one. I figured if you really want a chain of handlers, that's easy enough to write in your own handler function — but I don't think many people will care.

Lifetimes work differently now. That was a bit of a mess before: inside libpq, any result object gets a copy of whatever handler is installed on the connection, in case any operation on the result object generates a notice — because you might destroy the connection before you were done with the result. So it was easy to get into situations where the mechanism would hand a notice to a connection object that no longer existed. To some degree this has actually improved even for the old mechanism. But the new mechanism simply keeps your handler alive for as long as the connection or any of its results exist.

Finally, unlike the old errorhandler, the new "notice handlers" do not inhibit moving a connection object (e.g. with std::move()). In libpqxx 9.0 I expect pqxx::connection to be fully movable.

You will be able to continue using the old mechanism in libpqxx 7.x and 8.x, but it is marked as deprecated. The mechanisms live side by side, independently from each other. If you use errorhandler, please switch to the new mechanism — and make your own code cleaner and more modern in the process!

Brand-new LISTEN/NOTIFY API

Similar to error handlers, I have replaced notification_receiver with a more modern, simpler, and lambda-friendly API. Just register your callback using pqxx::connection::listen(). (Internally I refer to these callbacks as notification handlers.) As you might expect in a modern API, your callback can be anything that fits in a std::function: a lambda, a function, or a callable object.

Here too you lose a little bit of flexibility: a connection can only have one notification handler per channel at any given time. That means that it's easy to remove or replace handlers after you're done with them — just register a new one to replace it. An empty std::function disables listening on a channel.

You can only register a handler while no transaction is active (not even a pqxx::nontransaction). This keeps the internal administration much simpler than it would otherwise be. Conversely there's a new wrapper function to notify a channel, but this lives in the transaction class. (May seem a little strange but it reflects the slightly mind-bending implications of message buses in a transactional environment.)

Exception behaviour is now well-defined, and as with the new notice handlers, the new handlers do not inhibit moving a connection. I hope you'll find the new mechanism easier to work with. Your handler gets a bit more information, including a reference to the connection object, but it can safely ignore any items in which it is not interested.

The old mechanism will still be around in the 8.x release cycle (as well as the rest of 7.x of course), and live side by side with the old one. That means you can register both an old-style notification_receiver and a new-style notification handler, and get notified twice. However I hope that you will switch to the new style soon.

Phew, that's it! After this there may be 7.10.x bug-fix releases, but I hope to focus on 8.0 which will require C++20 or better and make better use of concepts.

libpqxx 7.9.2: sundry fixes.

Here's what's new in this release:

• Fix CMake documentation install. (#848)
• Bump gcc/clang/postgres minimum version requirements.
• Another fix to the readthedocs documentation build. (#845)
• Remove obsolete CMake workaround kludges. (#851, #866, #867, #869)
• Remove obscure deprecated stream_to constructor that never worked. (#853)
• Support reading a field as an SQL array using as_sql_array(). (#841)
• Make row iterator a proper random_access_iterator. (#846)
• Downgrade result iterator to "bidirectional"; was never really RA. (#846)

You may wonder why result iterators are no longer random access iterators. Turns out they never were! That's because if you have a result iterator i, referring to a row, then i[n] in libpqxx refers to field n in that row. The definition for random access iterators expects it to mean *(i+n) instead.

libpqxx 7.9.1: Re-do documentation

Documentation on ReadTheDocs was broken as of 7.9.0 (see #802). I rebuilt the whole thing from scratch — both local documentation builds and the ones on readthedocs. I think the new build is actually simpler than it was, which I hope will make it less sensitive to changes on ReadTheDocs or in dependencies.

What else changed?

• Fix bad conversion of array of empty strings to string. (#816)
• Move [[likely]] feature check back to compile time, to speed up configure.
• Support [[assume(...)]].
• Fix throw_null_conversion compile error when converting enums. (#807)
• Fix memory leak when failing to connect.
• Improve introductory docs for connections and transactions.
• No more autodetection of doxygen etc. Run it, or don't.
• Docs build now disabled by default; use --enable-documentation to change.

libpqxx 7.9.0: important fixes and new features

This is a pretty big release. As things stand, this looks to be the last release which supports C++17. The plan is to move on to libpqxx 8.0 next, which will require C++20 as a baseline.

Here's what's changed.

Assertion failure while streaming data

There was a bug that triggered an assertion failure when a row in a streaming query result ended in a null string.

The assertions in the code were just a little too strict. It amazes me that this took to long to surface. It has now been fixed.

New type aliases for binary data

We've been using std::basic_string<std::byte> and std::basic_string_view<std::byte> to represent binary data. But @tambry noted that compilers aren't actually required to support this! Worse, libc++18 had already marked it as deprecated.

The authors seem to have changed their minds about that, but Raul contributed a fix anyway. After all the problem may well pop up again.

And so, from libpqxx 7.9.0 onwards, to represent binary data coming from or going into the database, use the new type aliases pqxx::bytes (for a std::string of std::byte) and pqxx::bytes_view (for a std::string_view of std::byte).

If your environment supports the old types, these are just aliases for those, and nothing changes. But if it doesn't, then the aliases will refer to slightly different types that work around the problem. (The alternative definitions use a custom char_traits. The fine print in the C++ standard said that you need this, and that the library is not obligated to offer a generic definition of these traits for std::byte.)

Consistent library naming in CMake

Building CMake projects using libpqxx became a little easier. Thanks to @alexv-ds, you can now just use shared_link_libraries([...] libpqxx::pqxx).

The library name used to vary depending on whether you use find_package() or add_subdirectory() to add libpqxx to your project. Now it's just always libpqxx::pqxx.

Exception-related link errors

If you had a libpqxx built in C++17, and linked it into a project built using C++20, or vice versa, you'd get a lot of link errors about missing classes. They were related to exceptions, such as std::runtime_error.

Linking code built in one C++ version to code built in another is categorically dangerous. Please don't do it. There is no guarantee that it will work. Sadly though all package managers deal with this issue by sticking their heads in the sand.

It turns out that in practice the linking often worked, and various pre-built packaged versions of libpqxx shipped just one binary for all C++ versions. If you built your project in a different C++ version than was used to build libpqxx, two recent changes conspired to break your build:

1. For C++20 and up, I added source_location information to exceptions. If your compiler didn't support that, you just didn't get it — but it affected how an exception object was laid out in memory.
2. Instead of generating a header listing which C++ features were available when you built libpqxx, I just detected features at compile time. But that breaks down when the language changes between inbetween the libpqxx build and your own project's build!

The new release works around this using all-new code to generate a configuration header at build configuration time. The enw code is much more regular, and easier for me to extend and maintain. This should make it easier to add support for most new C++ features in the future. I also believe the build became just slightly faster.

Conversion from std::string_view to SQL string

Converting a std::string_view to an SQL string never actually worked. It wasn't a priority in part because pqxx::zview is likely to be much faster.

Still, this was an annoying irregularity and it has been fixed. You can now pass a std::vector<std::string_view> to a prepared statement, for example.

Expect future libpqxx versions to be a bit more liberal in allowing conversions of various view types. Which does mean that...

• sometimes the conversion may do a bit of extra work under the bonnet and it will be up to you to avoid this when performance is critical; and
• it will be your responsibility to ensure that the data underlying a view or span is still valid whenever you make use of its SQL conversion.

Parameterised versions of query() etc.

It has long bothered me that libpqxx has separate functions for executing a query, and for executing a query with parameters.

There are good reasons why you can't just pass any additional arguments to these functions and have them all converted into SQL parameters. It makes it easier to write code that doesn't mean what you might expect. It also complicates overloading, especially in a future where every query execution function also takes an implicit std::source_location to improve error reporting.

As of libpqxx 7.9.0 however you can now pass a pqxx::params when executing a query, and it will be unambiguously clear that it should be interpreted as a bundle of SQL parameters.

Streaming queries and C++20 pipelines

A streaming query can now act as a std::input_iterator. This removes an obstacle to building C++20 statement pipelines using streaming queries.

Clearer documentation for defining your own data types

The updated documentation makes it a bit easier to see how to go about defining SQL conversions for your own data types, so you can convert them between their SQL and C++ representations.

These conversions are particularly important when you want to pass them to parameterised or prepared statements.

Support for std::span as SQL arrays

This is still somewhat experimental, but libpqxx 8.0 will rely a lot more on std::span.

Thanks to @alexolog and @fallenworld1 you should now be able to pass any std::span (over a supported type of course) as a parameter to a prepared or parameterised statement, and it will automatically convert into an SQL array.

Support for PQfsize() and PQfmod()

You can now query a column's storage size and type modifier. This code was contributed by @TeamPlatform1.

These are only useful for low-level coders. Touch only if you know what you're doing.

Thanks

As you can see, various code changes have been contributed directly by libpqxx users. Others were requested in bug tickets. It would be a bit redundant for me to name them all here, but I am grateful: after all a good bug report is not so much a "customer complaint" as it is real-world feedback on what can be improved.

Further thanks go out to everyone who contributed, and not to forget — the tireless @tt4g and @KayEss who have stepped in to help time and again when people ran into problems.

libpqxx 7.8.1: platform-specific build fixes

Fixes two platform-specific build problems in 7.8.0:

Microsoft Visual Studio had been complaining loudly in C++20 mode when we included the deprecated header <ciso646>. But without that header, it turns out that Microsoft Visual Studio in C++17 mode can't compile libpqxx. So now we include the header only when compiling in C++17 mode.

The configure script built using autoconf did not support Apple's M1 and M2 ARM CPUs. Regenerating the script with a newer autoconf fixed that.

libpqxx 7.8.0: massive feature release

Welcome to libpqxx 7.8.0. Lots of goodies for you. Probably enough that I could have called it 8.0 — except libpqxx 8.0 is going to require C++20. For now you're still fine with C++17.

In 7.8 you get, among other things:

• Streaming large data sets now benchmarks faster than similar C/libpq code!
• New array class for easier parsing of SQL arrays.
• Deprecating stream_from. Use transaction_base::stream().
• Use array_parser only on comma-separated types, i.e. most of them. (#590)
• Bumping requirements versions: need postgres 10.
• Fix array_parser bug when parsing semicolon in an unquoted string.
• Make some zview constructors noexcept if string_view does it.
• Handle result status code for starting streaming replication. (#631)
• Faster text decoding and escaping in data streaming. (#601)
• Deprecate basic_fieldstream and fieldstream.
• Deprecate << operator inserting a field into an ostream.
• New string conversion fields: converts_to_string & converts_from_string.
• Support std::optional<std::string_view> etc. in stream_to. (#596)
• Remove support for single-quoted array/composite elements. No such thing!
• Work around build warning in MinGW: include winsock2.h before windows.h.
• If CMake can't find libpq, fall back to pkg-config. (#664)
• Work around spurious compile error on g++ pre-gcc-10. (#665)
• Include <pqxx/range> and <pqxx/time> headers in <pqxx/pqxx>. (#667)
• Don't use std::function as deleter for smart pointers.
• Work around gcc compile error with regex + address sanitizer + analyzers.
• Fix "double free" on exit when built as shared library on Debian. (#681)
• Stop including <ciso646>; should be built into compilers. (#680)
• New broken_connection exception subclass: protocol_violation. (#686)
• Retired unused blob_already_exists exception class. (#686)
• Support for PQinitOpenSSL(). (#678)
• Slightly more helpful error for unsupported conversions. (#695)
• Replace some C++ feature tests with C++20 feature macros.
• Support moving of stream_to. (#706)
• Incorporate source_location in exceptions.

There were some other small tweaks as well. If you implement the text conversions for your own types, there are two new fields that your string_traits specialisation: converts_to_string and converts_from_string. These are booleans that should say whether the string_traits class implements conversion, respectively, from the type to a string; and from a string to the type.

Enjoy this one! I'm sure somebody will find a problem, with so many changes, in which case we'll to a 7.8.1 soon.