Record lines truncation of target PDS Enhancement #2433

pujal0909 · 2025-02-09T17:36:17Z

What It Does

In the scenario when copying from a source PDS to a target PDS where the target does not have enough record lines for a member, initially, it would copy over that member, truncate it's record length, throw a truncation error and would not continue copying over the subsequent members. Now, it will continue copying over the subsequent members and inform the user that member(s)' contents were truncated and they can view the list of members affected in a local file.

How to Test

With the current functionality, copy a source PDS into a target PDS. The target PDS should have a lower record lines value than the source.
Notice that a member with a long record length will be truncated, and subsequent members will not continue to copy over.
Now with the updated functionality, copy the same source PDS into a target PDS with a lower record lines value than the source.
Notice that the user is informed of the truncation and is given a local file of members with this error. The subsequent members were copied over as well.

Review Checklist
I certify that I have:

tested my changes
added/updated automated tests
updated the changelog
followed the contribution guidelines

Additional Comments

Should this technically be a bug fix?
Should the Github security suggestion about using the tmp library be ignored?

Signed-off-by: Pujal <[email protected]>

codecov · 2025-02-09T17:48:52Z

Codecov Report

Attention: Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 91.49%. Comparing base (b91b989) to head (9d8dcce).

Files with missing lines	Patch %	Lines
packages/zosfiles/src/methods/copy/Copy.ts	83.33%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2433      +/-   ##
==========================================
- Coverage   91.50%   91.49%   -0.01%     
==========================================
  Files         641      641              
  Lines       18376    18392      +16     
  Branches     3872     3877       +5     
==========================================
+ Hits        16815    16828      +13     
- Misses       1559     1561       +2     
- Partials        2        3       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Pujal <[email protected]>

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts

packages/zosfiles/src/methods/copy/Copy.ts

Signed-off-by: Pujal <[email protected]>

Signed-off-by: Pujal Gandhi <[email protected]>

Signed-off-by: Pujal <[email protected]>

packages/zosfiles/src/methods/copy/Copy.ts

Signed-off-by: Pujal <[email protected]>

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts

Signed-off-by: Pujal <[email protected]>

github-actions · 2025-02-13T13:24:34Z

📅 Suggested merge-by date: 2/27/2025

packages/zosfiles/src/methods/copy/Copy.ts

traeok · 2025-02-18T20:51:00Z

If we already warn users that some members were truncated from the copy operation, what are your thoughts on listing the affected members as part of the CLI response? Otherwise, users are forced to locate & open the local file to see what members were truncated.

If there were more than, say, 10 members, we could print so many and then add ...and X more at the end of the output, directing the user to the local file to see the rest.

This isn't a request for changes, but I wanted to propose this as an option for warning the user - this may also help with identifying the troublesome members when copying within a CI environment.

Signed-off-by: Pujal <[email protected]>

gejohnston · 2025-02-19T21:41:17Z

packages/zosfiles/src/methods/copy/Copy.ts

+                        ZosFilesMessages.datasetCopiedSuccessfully.message + " " +
+                        ZosFilesMessages.membersContentTruncated.message + "\n\n" +
+                        firstTenMembers.join('\n') +
+                        `\n... and ${numMembers} more` +


Could this logic result in the message "and 0 more" ?

It would. Should that part of the response be removed for the case of when there are not more than ten truncated members? I wasn't sure if the inconsistent response would be confusing for users.

This is definitely a judgement call. If I were to hit the edge-case of exactly 10, I would think it odd to be told that "0 more" occurred. Alternatively if someone were automating on the output, it might be useful to know that "x more" would always be part of the message. I lean to removing the message when it would display "0 more". Perhaps wait a bit to see if others offer opinions.

i agree that adding "...and X more" should only be included if X is greater than 1

Signed-off-by: Pujal <[email protected]>

packages/zosfiles/__tests__/__system__/methods/copy/Copy.system.test.ts

packages/zosfiles/src/methods/copy/Copy.ts

+            const truncatedMembersFile = path.join(tmpdir(), 'truncatedMembers.txt');
+            if(truncatedMembers.length > 0) {
+                const firstTenMembers = truncatedMembers.slice(0, 10);
+                fs.writeFileSync(truncatedMembersFile, truncatedMembers.join('\n'));


Signed-off-by: Pujal <[email protected]>

packages/zosfiles/__tests__/__system__/methods/copy/Copy.system.test.ts

                it("Should copy a partitioned data set", async () => {
                    let error;
                    let response;
-
+                    const truncatedMembersFile = path.join(tmpdir(), 'truncatedMembers.txt');
+                    fs.writeFileSync(truncatedMembersFile, "");


Signed-off-by: Pujal <[email protected]>

ATorrise

agreeing with your discussion with Gene that your truncation response should only include an attached file if truncated items list is greater than 10.

…ssages when x is 0 Signed-off-by: Pujal <[email protected]>

Signed-off-by: Pujal <[email protected]>

sonarqubecloud · 2025-02-20T22:00:51Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
85.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

truncate lrecl functionality

2016201

Signed-off-by: Pujal <[email protected]>

github-actions bot assigned pujal0909 Feb 9, 2025

pujal0909 changed the title ~~Truncate lrecl target~~ Record lines truncation error - target data set Feb 9, 2025

updated system tests

8d2a989

Signed-off-by: Pujal <[email protected]>

pujal0909 changed the title ~~Record lines truncation error - target data set~~ Record lines truncation of target data set Enhancement Feb 10, 2025

pujal0909 changed the title ~~Record lines truncation of target data set Enhancement~~ Record lines truncation of target PDS Enhancement Feb 11, 2025

pujal0909 added 2 commits February 11, 2025 20:22

updated changelogs

37a2941

Signed-off-by: Pujal <[email protected]>

updated system tests

dd84ed7

Signed-off-by: Pujal <[email protected]>

github-advanced-security bot found potential problems Feb 12, 2025

View reviewed changes

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show fixed Hide fixed

updated temp file removal

c62d262

Signed-off-by: Pujal <[email protected]>

pujal0909 force-pushed the truncate-lrecl-target branch from a439165 to c62d262 Compare February 12, 2025 19:51

pujal0909 and others added 2 commits February 12, 2025 14:53

Merge branch 'master' into truncate-lrecl-target

3c2313c

Signed-off-by: Pujal Gandhi <[email protected]>

removed unused var

61be69b

Signed-off-by: Pujal <[email protected]>

github-advanced-security bot found potential problems Feb 12, 2025

View reviewed changes

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show resolved Hide resolved

fixed unit tests

a1714cb

Signed-off-by: Pujal <[email protected]>

github-advanced-security bot found potential problems Feb 13, 2025

View reviewed changes

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

pujal0909 added 2 commits February 12, 2025 21:27

updated integration snapshots

689f61e

Signed-off-by: Pujal <[email protected]>

updated snapshots

2161e6e

Signed-off-by: Pujal <[email protected]>

pujal0909 marked this pull request as ready for review February 13, 2025 13:24

pujal0909 requested review from t1m0thyj, gejohnston, zFernand0, awharn, traeok, ATorrise, adam-wolfe and jace-roell February 13, 2025 13:24

gejohnston approved these changes Feb 18, 2025

View reviewed changes

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show resolved Hide resolved

updated command response to list up to 10 truncated members

8148dc2

Signed-off-by: Pujal <[email protected]>

gejohnston reviewed Feb 19, 2025

View reviewed changes

merged with master

056a276

Signed-off-by: Pujal <[email protected]>

github-advanced-security bot found potential problems Feb 20, 2025

View reviewed changes

pujal0909 added 2 commits February 20, 2025 12:38

fixed system test

0a7e7c2

Signed-off-by: Pujal <[email protected]>

removed unused var

d5c3a7c

Signed-off-by: Pujal <[email protected]>

github-advanced-security bot found potential problems Feb 20, 2025

View reviewed changes

fixed system test

66215ae

Signed-off-by: Pujal <[email protected]>

ATorrise requested changes Feb 20, 2025

View reviewed changes

pujal0909 added 2 commits February 20, 2025 16:20

updated command response to not return local file and 'and x more' me…

8ed3d3e

…ssages when x is 0 Signed-off-by: Pujal <[email protected]>

updated unit tests

9d8dcce

Signed-off-by: Pujal <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record lines truncation of target PDS Enhancement #2433

Record lines truncation of target PDS Enhancement #2433

pujal0909 commented Feb 9, 2025 •

edited

Loading

codecov bot commented Feb 9, 2025 •

edited

Loading

github-actions bot commented Feb 13, 2025

traeok commented Feb 18, 2025 •

edited

Loading

gejohnston Feb 19, 2025

pujal0909 Feb 20, 2025 •

edited

Loading

gejohnston Feb 20, 2025

ATorrise Feb 20, 2025

ATorrise left a comment •

edited

Loading

sonarqubecloud bot commented Feb 20, 2025

Record lines truncation of target PDS Enhancement #2433

Are you sure you want to change the base?

Record lines truncation of target PDS Enhancement #2433

Conversation

pujal0909 commented Feb 9, 2025 • edited Loading

codecov bot commented Feb 9, 2025 • edited Loading

Codecov Report

github-actions bot commented Feb 13, 2025

traeok commented Feb 18, 2025 • edited Loading

gejohnston Feb 19, 2025

Choose a reason for hiding this comment

pujal0909 Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

gejohnston Feb 20, 2025

Choose a reason for hiding this comment

ATorrise Feb 20, 2025

Choose a reason for hiding this comment

ATorrise left a comment • edited Loading

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 20, 2025

Quality Gate passed

pujal0909 commented Feb 9, 2025 •

edited

Loading

codecov bot commented Feb 9, 2025 •

edited

Loading

traeok commented Feb 18, 2025 •

edited

Loading

pujal0909 Feb 20, 2025 •

edited

Loading

ATorrise left a comment •

edited

Loading