Should alchemlyb guarantee layout of dataframes?

[orbeckst]

In the context of #201 it became apparent that the order of columns in a DataFrame can become important if care is not taken to access the correct columns via .loc (and instead .iloc is used, assuming a specific ordering of columns).

#201 (comment) raises the question if alchemlyb should (or can) make guarantees about how dataframes are structured.

The Standard Forms of Raw Data only state what the column labels are but not the order. The examples seem to imply that, for instance, lambda columns are ordered from 0 to 1, but this is not actually stated, and hence no guarantee is actually made.

• Should we ensure that columns in dfs are ordered in a canonical manner (which would need to be defined)?
• Should we add a warning or note to the docs that order is not guaranteed?

[xiki-tempula]

Given that we are using pandas.DataFrame, using .loc should be able to pinpoint the exact point. So we could say that it is not enforced.
For the developer, it might be good to enforce sorting whenever one can so as to ensure that when one prints out the dataframe, it would be readable.

[orbeckst]

Do we want to start with adding sorting but not necessarily guarantee anything?

We just have to make sure that docs, examples, and alchemlyb and the tests are all using .loc.

[xiki-tempula]

@orbeckst Absolutely, I was checking it and have changed a couple of tests but they failed so I might need to diagnose some issue.

[xiki-tempula]

I was trying to replace the iloc with loc and there seems to be some issue with doing this in the dev side.

For the user, this is not an issue as the user would know the lambda states, but for the developer, the lambda states are unknown.

The way to get it is via delta_f_.columns, so we could replace the delta_f_.iloc[0] with delta_f_.loc[delta_f_.columns[0]], which would still make it susceptible to the issue in #201

To resolve the issue in #201, we could do delta_f_.loc[sorted(delta_f_.columns)[0]], but given that the dataframe is already sorted now, this would make no difference.

So I think in the code we can still keep the iloc

[orbeckst]

I don't quite follow your comment:

To resolve the issue in #201, we could do delta_f_.loc[sorted(delta_f_.columns)[0]], but given that the dataframe is already sorted now, this would make no difference.

In PR #201 we disabled sorting if I read

alchemlyb/src/alchemlyb/parsing/namd.py

Line 269 in 11033ee

u_nk = pd.concat([u_nk, tempDF], sort=False)

correctly together with @dotsdl 's explanation #201 (comment). This change restored the pre-pandas 1.4.3 behavior where we effectively relied on a bug in pandas (concat(sort=True) did not actually work) so that our dataframes are not sorted (at least in the NAMD parser).

@xiki-tempula can you point me to the code where we are actually sorting the df?

---

By the way, this line

alchemlyb/src/alchemlyb/parsing/namd.py

Line 304 in 11033ee

u_nk = pd.concat([u_nk, tempDF], sort=True)

in the NAMD parser looks suspicious: do we need to change it to sort=False, too?

[xiki-tempula]

@orbeckst
Sorry, I think I have misunderstood the issue. Will have another look later on.