main/read: revise the offset calculation on nested input stream#4212
Merged
masatake merged 17 commits intouniversal-ctags:masterfrom May 6, 2025
Conversation
masatake
changed the title
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
b498b9b to
b3b8694
Compare
Member
Author
|
There are too many inconsistencies in the nested-stream related code. |
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
b3b8694 to
0773776
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4212 +/- ##
==========================================
+ Coverage 85.85% 85.86% +0.01%
==========================================
Files 242 242
Lines 62610 62693 +83
==========================================
+ Hits 53752 53833 +81
- Misses 8858 8860 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Closed
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
0773776 to
55a26ca
Compare
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
55a26ca to
71088c3
Compare
Member
Author
|
This must be rebased on #4231. |
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
25717dc to
bc343c5
Compare
Member
Author
|
178140f may be applicable to PEG-based parsers. A test case: baz.md An internal error is reported as a warning: |
Member
|
The CI failes have been fixed. please rebase master branch. |
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
2 times, most recently
from
1607ba1 to
c917fcf
Compare
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…ere no data has been read yet The original code didn't consider the case that the parser calling getInputColumnNumber() run as a guest parser. This commit considers the case and simplifies the logic. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…neral case The original code didn't consider the case that the parser calling getInputColumnNumber() run as a guest parser. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…F case This was original proposed as a part of universal-ctags#4097 by @b4n. However, the fix exposed various hidden breakage in guest parsing. The previous computation could result in invalid values, which could lead to invalid promise ranges and subsequent crashes. This change considers two additional items to the original logic: - the area offset, and - ungetch buffer Co-authored-by: Colomban Wendling <ban@herbesfolles.org> Co-authored-by: Masatake YAMATO <yamato@redhat.com>
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…rent coordinate system
The original code returns a pos in the absolute coordinate system.
For the code expecting the function returns the absolute coordinate
system, this changes adds a more generic function:
MIOPos getInputFilePositionForLineFull (unsigned int line,
enum areaCoord areaCoord);
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…s as a guest Fix universal-ctags#4181. This commit consists of two interdependent parts that could not be separated into two individual commits. Introducing either part first would surface a previously hidden crash—the first sequence triggers one crash, while introducing the other part first triggers a different one. Merging any crash-inducing commit would hamper future git bisect debugging. Therefore, both parts are combined here into a single commit. The first part changes main/promise.c and main/read.c. When a parser made a promise for the same area (thin area) of the parser, the calculation of the area was delayed till the promise is forced. This delaying can be a bug if a guest parser makes a thin area because the guest parser expects the thin area is the same as the current area where the guest parser processes. On the other hand, forcing the promise is always done in the base area. With this change, makePromise() does the calculation of the thin area if the parser runs as a guest. The second part changes main/lxpath.c and parsers/yaml.c. The meta parsers user external libraries (libxml2 and libyaml) for parsing. The libraries provides the way to get the current line numbers. However, the coordinate system of the line number is the current coordinate system. updateTagLine() and setTagEndLine() expect a line number in the absolute coordinate system. In the 2nd part, we utilize translateLineNumber() for convert the line numbers to the values in the absolute coordinate system. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
(Edited by @masatake) Where ideal ctags should extract "post-", the current implementation extacts "post-<?".
Don't include (part of) the PHP open tag in the HTML input.
Fix universal-ctags#4097 JSP tags shouldn't trigger the PHP parser. This used to happen and trigger an infinite loop of a ping-pong effect with the HTML parser asking the PHP parser to parse the JSP block, and the PHP parser, not recognizing the input, asking the HTML one to deal with it. Rinse and repeat.
This function is for fixing peg based parsers. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
c917fcf to
8356581
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR aims to revise the offset calculation on nested input streams by simplifying the use of fpos_t. The changes include updates to test input files for markdown and HTML to support the new stream handling.
Reviewed Changes
Copilot reviewed 23 out of 41 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| Units/parser-markdown.r/packcc-parser-running-within-code-block.d/input.md | Adds a markdown input file with a code block intended for toml testing data; consider clarifying the purpose of the leading "T". |
| Units/parser-html.r/jsp.d/input.html | Adds an HTML input file with a JSP expression; ensure the expression is correctly completed. |
Files not reviewed (18)
- Units/parser-html.r/jsp.d/README: Language not supported
- Units/parser-html.r/jsp.d/args.ctags: Language not supported
- Units/parser-markdown.r/frontmatter.d/expected.tags: Language not supported
- Units/parser-markdown.r/packcc-parser-running-within-code-block.d/args.ctags: Language not supported
- Units/parser-markdown.r/packcc-parser-running-within-code-block.d/expected.tags: Language not supported
- Units/parser-markdown.r/packcc-parser-running-within-code-block.d/features: Language not supported
- Units/parser-php.r/wp-guest.d/args.ctags: Language not supported
- Units/parser-php.r/wp-guest.d/expected.tags: Language not supported
- Units/parser-php.r/wp-guest.d/input.php: Language not supported
- Units/parser-rmarkdown.r/frontmatter.d/expected.tags: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-external-libs.d/args.ctags: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-external-libs.d/expected.tags: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-external-libs.d/features: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-external-libs.d/input-0.sh: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-external-libs.d/input.sh: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-packcc.d/args.ctags: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-packcc.d/expected.tags: Language not supported
- Units/parser-sh.r/sh-heredoc-run-guest-parser-with-packcc.d/features: Language not supported
| @@ -0,0 +1,6 @@ | |||
| T | |||
There was a problem hiding this comment.
[nitpick] The single character 'T' at the start of the file is ambiguous. Please clarify its purpose or include additional context if this is intended test data.
… as a guest Fix universal-ctags#4114. setTagEndLine() takes a tag entry and a line number as arguments. setTagEndLine() expects the line number in the absolute coordinate system. However, a peg-based parser running as a guest parser passed a line number in the current coordinate system. Using the wrong coordinate system caused trouble; the end line number was smaller than the start line number assigned to the tag. The line numbers were derived from file offsets provided by the code generated by packcc. The file offsets were in the current coordinate system. As a result, the derived line numbers were in the current coordinate system. This change uses translateFileOffset() to convert the coordinate system of the offset. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
masatake
force-pushed
the
main/read--revise-offset-calculation-in-nested-input-stream
branch
from
8356581 to
036d135
Compare
…bugs Signed-off-by: Masatake YAMATO <yamato@redhat.com>
…guest parser has read nothing yet getInputLineNumber() returned 1, not 0 if a parser had read nothing yet. This is the expected behaviour IF the parser runs as a host parser. If a parser runs as a guest parser, it should return the start line of the area where the guest parser runs. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Conceptually this pull request is derrived from #4097.
The commit logs are not updated yet.
See also #3727.
fpos_t makes our code too complicated.