Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(dep-resolution): update dep-resolution RPC interface to support lockfiles #312

Merged
merged 5 commits into from
Nov 15, 2024

Conversation

bkettle
Copy link
Contributor

@bkettle bkettle commented Nov 7, 2024

Updates the RPC interface used to resolve dependencies in OCaml to support
lockfiles as well as manifests. This will allow us to begin porting lockfile
parsers to Ocaml as we move forward.

These changes also make it possible to report errors that do not cause dependency resolution to fail entirely.

These changes are not backwards compatible. However, they only affect types
that are used within the CLI, from the Python code to the OCaml code, so as long
as we are careful with releasing the changes to the CLI, it should be safe.

This must be merged in sync with https://github.com/semgrep/semgrep-proprietary/pull/2571 since it has breaking changes.

Copy link

github-actions bot commented Nov 7, 2024

Backwards compatibility summary:

Checking backward compatibility of semgrep_output_v1.atd against past version v1.29.0
Skipping v1.30.0 because commit 78720c795cd5a186f5102c87125ef876c6435a0c has already been checked
Skipping v1.31.0 because commit 78720c795cd5a186f5102c87125ef876c6435a0c has already been checked
Skipping v1.31.1 because commit 78720c795cd5a186f5102c87125ef876c6435a0c has already been checked
Skipping v1.31.2 because commit 78720c795cd5a186f5102c87125ef876c6435a0c has already been checked
Skipping v1.32.0 because commit 78720c795cd5a186f5102c87125ef876c6435a0c has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.33.0
Skipping v1.33.1 because commit 8849e56ddb0977e38a120a6cfbd1c396eb6fa15e has already been checked
Skipping v1.33.2 because commit 8849e56ddb0977e38a120a6cfbd1c396eb6fa15e has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.34.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.37.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.38.0
Skipping v1.38.1 because commit fd294683e7369cabf63738febeaba8a22c925187 has already been checked
Skipping v1.38.2 because commit fd294683e7369cabf63738febeaba8a22c925187 has already been checked
Skipping v1.38.3 because commit fd294683e7369cabf63738febeaba8a22c925187 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.39.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.40.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.41.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.42.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.43.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.44.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.45.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.46.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.47.0
Skipping v1.48.0 because commit 278ed753e0c66b8bfc3f2d805fde53be022dd4b6 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.49.0
Skipping v1.50.0 because commit 857682f41eb09e0b330a247ff1adf3bfeaf9d9ca has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.52.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.53.0
Skipping v1.54.0 because commit 3b72d494260258497e796d094b1a4916501a6df1 has already been checked
Skipping v1.54.1 because commit 3b72d494260258497e796d094b1a4916501a6df1 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.54.2
Skipping v1.54.3 because commit 9f1c50383a9a9969e2fe7a5f9bff9ca0a7c837bb has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.55.0
Skipping v1.55.1 because commit 6dffeaa692153fd33b4f154fddaefde1f2f1ae27 has already been checked
Skipping v1.55.2 because commit 6dffeaa692153fd33b4f154fddaefde1f2f1ae27 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.56.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.57.0
Skipping v1.58.0 because commit 4cc11b00d411c02fc611aa8c78a336520438fb48 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.59.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.59.1
Checking backward compatibility of semgrep_output_v1.atd against past version v1.60.0
Skipping v1.60.1 because commit eed58a091fd7d19e402a6d4cf2d287e137215d03 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.61.0
Skipping v1.61.1 because commit bbfd1c5b91bd411bceffc3de73f5f0b37f04433d has already been checked
Skipping v1.62.0 because commit bbfd1c5b91bd411bceffc3de73f5f0b37f04433d has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.63.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.64.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.65.0
Skipping v1.66.0 because commit 3e7bbafa2b7e722d893303a7fb90a83dab6737a7 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.66.1
Skipping v1.66.2 because commit 215a54782174de84f97188632b4a37e35ba0f827 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.67.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.68.0
Skipping v1.69.0 because commit d5b91fa4f6a03240db31e9bbbc5376a99bc8eeea has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.70.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.71.0
Skipping v1.72.0 because commit 75abf193687b84ab341d8267d865ad68d81a89c9 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.73.0
Skipping v1.74.0 because commit 9f38254957c50c68ea402eebae0f7aa40dd01cbf has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.75.0
Skipping v1.76.0 because commit 9102031608aa4154e1c37f557550ec4eabc8780c has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.77.0
Skipping v1.78.0 because commit dcb5d77b420ddee61f58aadd3c2c7aef38778154 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.79.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.80.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.81.0
Skipping v1.82.0 because commit 9e0f3bec26b07b4fb6753a32cb75277f45f2572c has already been checked
Skipping v1.83.0 because commit 9e0f3bec26b07b4fb6753a32cb75277f45f2572c has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.84.0
Skipping v1.84.1 because commit 3daef49297ada205359cc1d2996354c94b628b0d has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.85.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.86.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.87.0
Skipping v1.88.0 because commit 512c0bd97db59c48a5705b2741662a338776e438 has already been checked
Skipping v1.89.0 because commit 512c0bd97db59c48a5705b2741662a338776e438 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.90.0
--- before.txt
+++ after.txt
@@ -1,7 +1,28 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
 [346efceb] Incompatibility in both directions:
 Incompatible tuple lengths
 The following types are affected:
   function_call
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
 
 [3cb750d6] Backward incompatibility:
 Formerly optional field 'version' is now required.
ERROR: semgrep_output_v1.atd is not backward compatible with v1.90.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.91.0
--- before.txt
+++ after.txt
@@ -1,7 +1,28 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
 [346efceb] Incompatibility in both directions:
 Incompatible tuple lengths
 The following types are affected:
   function_call
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
 
 [3cb750d6] Backward incompatibility:
 Formerly optional field 'version' is now required.
ERROR: semgrep_output_v1.atd is not backward compatible with v1.91.0
Skipping v1.92.0 because commit 2351c5e528cb7430422208dc66707894c066b508 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.93.0
--- before.txt
+++ after.txt
@@ -1,7 +1,28 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
 [346efceb] Incompatibility in both directions:
 Incompatible tuple lengths
 The following types are affected:
   function_call
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
 
 [3cb750d6] Backward incompatibility:
 Formerly optional field 'version' is now required.
ERROR: semgrep_output_v1.atd is not backward compatible with v1.93.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.94.0
--- before.txt
+++ after.txt
@@ -1,5 +1,26 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
 [346efceb] Incompatibility in both directions:
 Incompatible tuple lengths
 The following types are affected:
   function_call
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
 
ERROR: semgrep_output_v1.atd is not backward compatible with v1.94.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.95.0
--- before.txt
+++ after.txt
@@ -1,5 +1,26 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
 [346efceb] Incompatibility in both directions:
 Incompatible tuple lengths
 The following types are affected:
   function_call
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
 
ERROR: semgrep_output_v1.atd is not backward compatible with v1.95.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.96.0
--- before.txt
+++ after.txt
@@ -0,0 +1,21 @@
+[1a8d9110] Incompatibility in both directions:
+Incompatible kinds of types: record/object is now a sum type or enum.
+The following types are affected:
+  dependency_source
+  function_call
+  function_return
+  manifest
+
+[233aede9] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a tuple.
+The following types are affected:
+  function_return
+  resolution_result
+
+[385fcd66] Incompatibility in both directions:
+Incompatible kinds of types: sum type or enum is now a list/array.
+The following types are affected:
+  function_return
+  resolution_error
+  resolution_result
+
ERROR: semgrep_output_v1.atd is not backward compatible with v1.96.0

@bkettle
Copy link
Contributor Author

bkettle commented Nov 7, 2024

@aryx @nmote I have these changes I want to make to the RPC interface. They aren't backwards compatible, so the check fails. As far as I can tell, though, there is no real need for backwards compatibility in these RPC types (if ATD had a module system so that we could use types from semgrep-interfaces I think the RPC types would be their own ATD file like Input_to_core etc.).

Can we ignore the backwards compatibility check here? Or should I restructure these changes to be backwards compatible? (I think the latter would probably look something like adding a new RPC call CallResolveDependenciesV2)

@bkettle bkettle requested review from mmcqd and SayyantR November 7, 2024 22:56
@aryx
Copy link
Collaborator

aryx commented Nov 8, 2024

yes we can ignore the backward check. @mjambon is there a way to adjust the linting to give a list of toplevel type that don't need to go throught the backward compatibility check? (basically the RPC stuff and the semgrep-core output).
Ideally we would use different module for those different behavior but ATD does not have the module system yet so simpler to have everything in one file.

semgrep_output_v1.atd Outdated Show resolved Hide resolved
semgrep_output_v1.atd Outdated Show resolved Hide resolved
@bkettle bkettle mentioned this pull request Nov 12, 2024
2 tasks
@bkettle bkettle force-pushed the bk/ddr/refactor-rpc-interface-for-lockfiles branch from dc8a4a6 to c78ebe8 Compare November 14, 2024 00:00
@bkettle bkettle marked this pull request as ready for review November 14, 2024 18:23
@bkettle
Copy link
Contributor Author

bkettle commented Nov 14, 2024

@aryx I'm finally ready to merge this and the accompanying semgrep-proprietary PR, but it looks like I can't do so because the backwards compatibility check is required. Is this something you can help with?

There's no reason this needs to be in this week's release, so we should probably wait until after.

@aryx
Copy link
Collaborator

aryx commented Nov 15, 2024

Yes I can override the backward compatibility check because I'm admin on this repo I think. @mjambon too.

@aryx
Copy link
Collaborator

aryx commented Nov 15, 2024

@mjambon we need to be able to whitelist a few entry types to not be concerned by the backward compatibility check (e.g., the types for semgrep-core or here the types for Nat's RPC, or we need a module system so we can use
different files for those different purpose.

@aryx
Copy link
Collaborator

aryx commented Nov 15, 2024

The whitelist seems the easiest option.

@aryx aryx merged commit 1c6af38 into main Nov 15, 2024
2 of 3 checks passed
@aryx aryx deleted the bk/ddr/refactor-rpc-interface-for-lockfiles branch November 15, 2024 08:53
@bkettle
Copy link
Contributor Author

bkettle commented Nov 15, 2024

Thanks @aryx ! Agree that a whitelist seems nice; it would be good to be able to make changes to the RPC interfaces more easily.

@aryx
Copy link
Collaborator

aryx commented Nov 15, 2024

yes, and merging the semgrep-interface inside the semgrep-pro repo would accelerate things even more.
That's one of the task of @mjambon for Q4 actually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants