Feature Request: Add character-level replacement for custom symbols

[Dan-Ashby]

Feature Request: Character-level replacement for custom symbols

Problem

The current custom_symbols feature is designed to replace entire variable names with a LaTeX friendly string. In some instances it's desirable to swap out individual characters within a variable name. The existing custom_symbols implementation can achieve this, but it can only swap one matching string:

So, for example, if I wanted to introduce parenthesis into a variable by mapping LB and RB to the left and right parenthesis as below:
handcalcs.set_option("custom_symbols", {'LB': '(','RB': ')'})
This would not yield the desired result:
myvarLB1RB would map to myvar(1RB instead of the desired myvar(1)

This occurs because there's a break statement in the swap_custom_symbols method, which prevents subsequent matches being actioned. This is evidently by design, as - if the user is intending only to match whole 'words', rather than a subset of characters within the word - then the variables could quickly become scrambled if multiple matching substrings were found in the custom_symbols list. For example, if the user entered the following:
custom_symbols = { "V_": "V_{", "dot": "\\cdot" "V_dot": "\\dot{V}", }
Then a variable named V_dot would incorrectly be parsed to V_{dot with the first match, and then V_{\\cdot with the second match. The third (desired) string would never match, as - by the time it's reached - the input variable has been scrambled. Note that this risk already exists in the current implementation.

Proposed Solution

1. Restrict the existing swap_custom_symbols to only match and swap whole-word matches (currently there's a risk of unintended scrambling, as described above).
2. Add a new swap_custom_characters function that performs character-level replacement without the break statement, allowing multiple replacements in a single variable name. Having both methods gives the user control over whether to match entire words, or individual characters, increasing flexibility, and reducing the potential for unintended behaviors.

Implementation

I have successfully implemented this feature locally with minimal changes:

• New swap_custom_characters function (similar to existing swap_custom_symbols)
• Added to processing pipeline at the end to avoid conflicts (i.e. the characters swaps are made after all other functions have been performed, so as to avoid conflicting with other existing behaviors).
• New config option custom_characters
• Maintains full backward compatibility

Use Case

This enables clean mathematical notation, utilising parenthesis, commas, and other restricted characters like.
In my current implementation, I've mapped the valid python characters with UniCodes 718, 719, and 716 to the left and right parenthesis, and the comma, respectively, so that I can generate LaTeX representations as below:
handcalcs.set_option("custom_characters", {'ˎ': '(','ˏ': ')', 'ˌ': ','})

• phi_1ˎ3ˌ2ˏ → $φ_{1(3,2)}$
• E_c1ˎ2ˏ → $E_{c1(2)}$

As a structural engineer, you'll perhaps appreciate these symbols; the first is the creep coefficient and the second is the concrete modulus used for tracking the response of element 1 (the precast deck panels) from time index 2 to 3 in a stepwise time-dependent analysis of a composite prestressed concrete bridge girder.

I appreciate that these could have been mapped more simply with the current implementation, but I've found accurately matching the notation to reference texts really helps with avoiding errors! Plus it appeals to my OCD, obviously! haha

If you're interested I'd be happy to provide the implementation details or create a pull request, though as described above it's really just a duplication of the existing swap_custom_symbols functionality, only without a break.

[connorferster]

I like this! If you have an implementation already, please submit a PR and I will review it quickly.

Could you write some tests for this new character-level swapping feature?

[Dan-Ashby]

Hi @connorferster, yeah sure, I'll give this a go.

Note that I'm completely new to both python and github; I've recently started learning these new tricks with a leg-up from AI after being stuck on VBA for years. So this will be my first contribution to a library and in fact the first pull request I'll have made, so let's see how I go... haha

I've given this some more thought today and implemented some changes as below:

Retain and modify existing swap_custom_symbols method

After some thought and follow-up testing I've found that simply modifying the existing swap_custom_symbols to allow this to make multiple swaps is functionally the same as adding a second method to do character-level swaps, as the existing method already allows for both complete and partial string replacement. This is obviously way simpler, and requires minimal changes, so I've progressed with this.

Added function for adding brackets

I've added an additional 'custom_brackets' option that will allow the user to specify a single or multi-character string to be swapped for one of a range of different brackets the user might choose from. I've developed this from the previous example I gave yesterday so that a single character can be used to represent both opening and closing brackets; the function will just sequentially alternate swapping in opening and closing brackets each time it encounters the user-defined character(s). This avoids having to separately define an open and close swap string, which is way more convenient to use. With this approach I've been able to map all the key LaTeX brackets.
The above works fine when combined with the custom_symbols too.

Bug in extend_substrings resurfaced

There's a bug in the extend_substrings method, which tacks on one too many closing braces in some cases. I can see there's a 'discount' hack in this method that's evidently from an earlier effort to solve this same problem that is getting broken again when combined with multiple substitutions. I believe this occurs when prior manipulation of an input string results in the string having curly braces, and then combining this with a subscript. In this case you'll see I've run into it when trying to swap the N_star in combination with a following subscript text. I've run out of steam for today, so planning to just submit this as-is. Would you mind looking at this extend_subscript logic?

I might need some help with implementing unit testing too; I've been battling with this a bit, so have just been doing the testing in a Jupyter Notebook.

[Dan-Ashby]

Found the bug in the extend_subscripts function, so I've made a fix and pushed my changes.

def extend_subscripts(pycode_as_deque: deque, **config_options) -> deque:
    """
    For variables named with a subscript, e.g. V_c, this function ensures that any
    more than one subscript, e.g. s_ze, is included in the latex subscript notation.
    For any item in 'pycode_as_deque' that has more than one character in the subscript,
    e.g. s_ze, then it will be converted to s_{ze}. Also handles nested subscripts.
    """
    swapped_deque = deque([])
    for item in pycode_as_deque:
        if isinstance(item, deque):
            new_item = extend_subscripts(item)  # recursion!
            swapped_deque.append(new_item)
        elif isinstance(item, str) and "_" in item and not "\\int" in item:
            new_item = ""
            for char in item:
                if char == "_":
                    new_item += char
                    new_item += "{"
                else:
                    new_item += char
            # Removed the 'discount' hack and count
            # Revised num_braces to calculate the difference between number of opening and closing braces
            # This now works even if the input string already contained braces.
            num_braces = new_item.count("{") - new_item.count("}")

            new_item += "}" * num_braces
            swapped_deque.append(new_item)
        else:
            swapped_deque.append(item)
    return swapped_deque

[av-engineering]

Found the bug in the extend_subscripts function, so I've made a fix and pushed my changes.

def extend_subscripts(pycode_as_deque: deque, **config_options) -> deque:
"""
For variables named with a subscript, e.g. V_c, this function ensures that any
more than one subscript, e.g. s_ze, is included in the latex subscript notation.
For any item in 'pycode_as_deque' that has more than one character in the subscript,
e.g. s_ze, then it will be converted to s_{ze}. Also handles nested subscripts.
"""
swapped_deque = deque([])
for item in pycode_as_deque:
if isinstance(item, deque):
new_item = extend_subscripts(item) # recursion!
swapped_deque.append(new_item)
elif isinstance(item, str) and "" in item and not "\int" in item:
new_item = ""
for char in item:
if char == "":
new_item += char
new_item += "{"
else:
new_item += char
# Removed the 'discount' hack and count
# Revised num_braces to calculate the difference between number of opening and closing braces
# This now works even if the input string already contained braces.
num_braces = new_item.count("{") - new_item.count("}")

        new_item += "}" * num_braces
        swapped_deque.append(new_item)
    else:
        swapped_deque.append(item)
return swapped_deque

Hello @Dan-Ashby,

I had "solved" the bracket issue in a different way. Maybe want to take a look on that:
#231 (comment).

[Dan-Ashby]

Hi @0av,

Thanks, yeah I hadn't seen this. I think what you'd proposed would have fixed the case I ran into above.

However, I think the use of discount here is a bit brittle though:

custom_symbols = config_options.get("custom_symbols", {})   # ADD LINE
            if "\\mathrm{" in item or "\\operatorname{" in item or any( # ADD BEGINNING WITH LAST OR
                value in item for value in custom_symbols.values()      # ADD LINE
            ):                                                          # ADD LINE
                discount = 1

It's checking for the presence of braces in the input string, and then setting discount = 1 if any are found. This would fail if either

1. There's some additional source of braces beyond mathrm, operatorname, and custom_symbols that we haven't considered, or that might be added in future (this is evidently what broke it when custom_symbols was introduced).
2. There's more than one set of braces in the sources. For example this would fail because it contains both \\mathrm{...} for the formatting and ^{...} braces for the superscript, so in this instance discount should in fact be set to 2, not 1:
   $\mathrm{y}^{2x}$

What I've proposed is to directly count how many opening and closing braces are present, and then add as many closing braces as is necessary to form a well-formed LaTeX string:
num_braces = new_item.count("{") - new_item.count("}")

I think this should work irrespective of how many braces might be present.

I'm completely new to Python and LaTeX though (I had to google what mathrm and operatorname were just now!), and I only just installed and started using handcalcs last week, so I could very well be missing a trick here! If you see any issue with the code above can you flag this and tag in Connor so he can consider it, as I submitted this for his review last night.

Cheers,

Dan

[KY9941]

Hi @Dan-Ashby,

As another structural engineer, this feature looks extremely useful.
Has anything come from your efforts so far in terms of implementing in the next version of handcalcs?

Thanks,
Kevin

[Dan-Ashby]

I submitted a pull request for this a while back which is pending review. I understand from Connor he's planning to incorporate this but evidently he hasn't got to it yet.

[connorferster]

Merged in #235. Thank you!

[connorferster]

Release coming shortly...