Release v1.18.0

1.18.0 - 2023-04-14

Added

• Metavariable comparison: Added support for **, the exponentiation operator. (gh-7474)

• Pro: Java: Semgrep is now able to track the propagation of taint from the
  arguments of a method, to the object being called. So e.g. given a method

  public void foo(int x) {
      this.x = x;
  }
  

  and a call o.foo(tainted), Semgrep will be able to track that the field
  x of o has been tainted. (pa-2570)

• Kotlin: Class fields will now receive the correct types, and be
  found by typed metavariables correctly

  This applies to examples such as:
  class Foo {
  var x : int
  }
  for the variable x (pa-2684)

• Supply Chain support for package-lock.json version 3 (sc-586)

Fixed

• metavariable-pattern: When used with the nested language key, if there was an
  error parsing the metavariable's content, that error could abort the analysis
  of the current file. If there were other rules that were going to produce findings
  on that file, those findings were not being reported. (gh-7271)

• Matching: Fixed a bug where explicit casts of expressions would produce two matches to
  other explicit casts.

  So for instance, a pattern (int $X) in Java would match twice to (int) 5. (gh-7403)

• taint-mode: Given x = tainted, then x.a = safe, then x.a.b = tainted, Semgrep
  did not report sink(x.a.b). Because x.a was clean, that made Semgrep disregard
  the tainting of any field of x.a such as x.a.b. This now works as expected. (pa-2486)

• When using metavariable-pattern to match embedded PHP code, Semgrep was
  unconditionally adding the <?php opening to the embedded code. When
  <?php was already present, this caused parsing errors. (pa-2696)

• Lockfile-only supply chain findings correctly include line numbers in their match data, improving the appearence of CLI output (sc-658)

• Increase timeout for semgrep install-semgrep-pro to avoid failures when the download is slow. (timeout)

• Fixed the range reported by findings for YAML files that include an anchor, so that the match does not include the original location of the snippet bound to the anchor. (yaml-alias)

Release v1.17.1

1.17.1 - 2023-04-05

Fixed

• Fix an issue that could lead to a crash when printing findings that contain snippets that look like markup to the Rich Python library (rich-markup-crash)

Release v1.17.0

1.17.0 - 2023-04-04

Added

• Scala: Added proper parsing for Scala 3 style imports (pa-2678)

Changed

• taint-mode: Added option taint_assume_safe_comparisons, disabled by default, that
  prevents comparison operators to propagate taint, so e.g. tainted != "something"
  will not be considered tainted. Note that this a syntactic check, if the operator
  is overloaded to perform a different operation this will not be detected. (pa-2645)

Fixed

• Fixed an issue where incorrect ranges for expressions containing parentheses could lead Semgrep to generate invalid autofixes in Python. (gh-2902)
• In rare cases, Semgrep could generate invalid autofixes where Python keyword arguments were placed before positional arguments. When using AST-based autofix, it no longer makes that error. (keywordarg-autofix)

Release v1.16.0

1.16.0 - 2023-03-30

Added

• Kotlin: Added support for typed metavariables. You can write a pattern like:
  ($X : String)
  to find all instances of expressions with type String. (pa-2648)
• Scala: Semgrep can now parse programs that contain quoted expressions, context
  parameter clauses using using, and soft modifiers like inline and open. (pa-2672)
• Scala: Can now parse programs containing matches on types, such as:
  type t = K match {
  case Int => String
  } (pa-2673)
• Parsing rules can take multiple seconds at the start of a scan.
  When running in an interactive terminal with more than 500 rules,
  Semgrep will show a progress bar for this step. (rule-progress)

Changed

• Supply Chain scans will now understand maven_dep_tree.txt files
  that are made of multiple smaller maven_dep_tree.txt files concatenated with cat. (maven-dep-forest)
• Findings of a scan are now printed with section headers per group for the following categories:
  Code Blocking, Code Non-blocking, Supply Chain Reachable, Supply Chain Unreachable findings. (results-headings)
• Switched to using go.mod files to read go dependencies for Semgrep Supply Chain, instead of go.sum files (sc-gomod)

Fixed

• Clojure: parse 'foo/bar' as two separate tokens, so one can use
  metavariable in it and get '$X/bar' to match 'foo/bar' (gh-7311)
• HTML/XML: support attribute as pattern (e.g., foo="true") (gh-7344)
• Improved significantly the time to parse big rulesets such as p/default
  from 20s to a few seconds by parsing patterns lazily and by
  not using /tmp to parse those patterns. (pa-2597)
• Pipfiles with a line comment or inline comment will now parse correctly. (sc-664)

Release v1.15.0

1.15.0 - 2023-03-15

Added

• On full sca scans with dep search feature on, send dependency data for dep search (depsearch)

• metavariable-comparison: Added support for bitwise operators ~, &, | and ^. (gh-7284)

• Taint: pattern-propagators now have optional fields requires and label,
  which are used identically to their counterparts in pattern-sources and pattern-sinks, for the experimental taint labels feature.

  For instance, we can define:

  pattern-propagators:
    - pattern: |
        $TO.foo($FROM)
      from: $FROM
      to: $TO
      requires: A
      replace-labels: [A, C]
      label: B
  

  to denote a propagator which only propagates from $FROM to $TO if $FROM has
  taint label A. In addition, it converts any taints from $TO with labels
  A or C to have label B.

  If label is not specified, the to is tainted with the same label of taint
  that $FROM has. If requires is not specified, it does not require $FROM to
  have a particular label of taint.

  Additionally, replace-labels only restricts the label being propagated if
  the output label is specified. (pa-1633)

• taint-mode: Java: Support for basic field sensitivity via getters and setters.
  Given obj.setX(tainted), Semgrep will identify that a subsequent obj.getX()
  carries the same taint as tainted. It will also differentiate between
  obj.getX() and obj.getY(). Note that Semgrep does not examine the definitions
  for the getter or setter methods, and it does not know whether e.g. some other
  method obj.clearX() clears the taint that obj.setX(tainted) adds. (pa-2585)

• Pro Engine: Semgrep CLI will now download a version of Semgrep Pro Engine
  compatible with the current version of Semgrep CLI, as opposed to the most
  recently released version.

  This behavior is only supported for Semgrep 1.12.1 and later. Previous
  versions will still download the most recently released version, as before. (pa-2595)

Changed

• Pro: semgrep ci will run intrafile interprocedural taint analysis by default
  in differential scans (aka PR scans). (Note that interfile analysis is not run
  in differential scans for performance reasons.) (pa-2565)

• Remove custom entrypoint for returntocorp/semgrep Docker images, now you must
  explicitly call semgrep.

  This won't work now: docker run -v $(pwd):/src returntocorp/semgrep scan ...
  Must do this instead: docker run -v $(pwd):/src returntocorp/semgrep semgrep scan ... (pa-2642)

• Changed Maven version comparison to more closely reflect usage, so versions with more than 3 increments will not be treated as plain strings (sc-656)

Fixed

• The AST dump produced by semgrep-core is now usable from Python
  with the provided ATD interface and the Python code derived from it with
  atdpy. (gh-7296)

• Terraform: Nested blocks can now be used as sources and sinks for taint.
  For instance, the block x in

  resource $A $B {
  x {
  ...
  }
  } (pa-2475)

• CLI: The scan progress bar now shows progress with higher granularity, and has fewer big jumps when using the Pro Engine.

  The abstract unit of 'tasks' has been removed, and now only a percentage number will be displayed. (pa-2526)

• Fix an error with rule targeting for extract mode. Previously, if a ruleset had
  two rules, the first being the extract rule, the second being the rule to run,
  no rules would run on the extracted targets. Additionally, with multiple rules
  the wrong rule might be run on the extracted target, causing errors. Now, in
  extract mode all the rules for the destination language will be run. (pa-2591)

• Metrics: logged in semgrep ci scans now send metrics, as our Privacy.md indicates
  (previously they incorrectly did not, which made it harder for us to track failure events) (pa-2592)

• Rust: Basic let-statement bindings (such as let x = tainted) now properly
  carry taint. (pa-2605)

• Improved error reporting for rule parsing by correctly reporting parse errors
  instead of engine errors in certain cases. (pa-2610)

• Taint: Fixed an issue where an error could be thrown if semgrep-core's output
  contained a dataflow trace without a sink. (pa-2625)

• Julia: Properly allow string literal metavariables like "$A" to be patterns. (pa-2630)

Release v1.14.0

1.14.0 - 2023-03-01

Added

• Add new hashes of a match (finding) to send to the app:
  • code_hash
  • pattern_hash
  • start_line_hash
  • end-line_hash (gh-7218)

Changed

• taint-mode: Historically, the matching of taint sinks has been somewhat imprecise.
  For example, sink(ok if tainted else ok) was flagged. Recently, we made sink-
  matching more precise for sinks like sink(...) declaring that any argument of
  a given function is a sink. Now we make it more precise when specific arguments of
  a function are sinks, like:

  pattern-sinks:
  - patterns:
    - pattern: sink($X, ...)
    - focus-metavariable: $X

  So sink(ok1 if tainted else ok2), sink(not_a_propagator(tainted)), and
  sink(some_array[tainted]), will not be reported as findings. (pa-2477)

• The --gitlab-sast and --gitlab-secrets output formats have been upgraded.
  The output is now valid with the GitLab v15 schema,
  while staying valid with the GitLab v14 schema as well.
  Code findings now include the confidence of the rule.
  Supply Chain findings now include the exposure type. (sc-635)

Fixed

• Fix: Semgrep Pro previously reported a crash for user errors such as
  invalid patterns. It will now give a good error message. (gh-7028)
• Dataflow: Fixed incorrect translation of side-effects inside Boolean expressions,
  this was (for example) causing if (cond && x = 42) S1; S2 to be interpreted as
  x = 42; if (cond && x) S1; S2, thus incorrectly flagging x as a constant
  inside S2. (gh-7199)
• Solidity: support enum and event patterns (gh-7230)
• Kotlin: allow to match extended class in any order
  (e.g., the pattern class $X : Foo will also match class Stuff : Bar, Foo). (gh-7248)
• taint-mode: Code such as sink(sanitizer(source) if source else ok) will not be
  incorrectly reported as a tainted sink. This follows a previous attempt at fixing
  these issues in version 1.1.0. (pa-2509)
• metavariable-pattern: Fixed regression introduced in version 1.9.0 that broke
  the use of pattern-not within metavariable-pattern in some cases. (pa-2510)
• Make Semgrep parse HTML-style comments in JavaScript code. (pa-2560)
• Reduced peak memory usage during inter-file analysis (pa-2563)
• Supply Chain scans on a project using Pipenv
  will now detect transitivity from the committed Pipfile,
  instead of printing an error while trying to parse it. (pa-2577)
• --oss-only previously required --oss-only true to be passed. This PR fixes
  it so that --oss-only will invoke the oss engine. Note that --oss-only true
  will no longer be supported (pa-2587)

Release v1.13.0

1.13.0 - 2023-02-24

Added

• Detect BITBUCKET_TOKEN from environment to authenticate with the Bitbucket API. (app-3691)
• taint-mode: Taint propagators can now specify by-side-effect, just like sources and
  sanitizers. However, the default value of by-side-effect for propagators is true
  (unlike for sources or sanitizers). When using rule option
  taint_assume_safe_functions: true, this allows to specify functions that must
  propagate taint, for example:

      pattern-propagators:
        - by-side-effect: false
          patterns:
            - pattern-inside: $F(..., $X, ...)
            - focus-metavariable: $F
            - pattern-either:
                - pattern: unsafe_function
          from: $X
          to: $F

  Without by-side-effect: true, unsafe_function itself would be tainted by side-
  effect, and subsequent invokations of this function, even if the arguments were safe,
  would be tainted. (pa-2400)
• CLI: SARIF output now includes a tag pertaining to which card of the Rule Board a rule originated from.
  This can be "rule-board-block", "rule-board-audit", or "rule-board-pr-comments". (pa-2519)

Fixed

• Fixed a couple of typos in help/usage. (gh-7184)
• Matching: Fixed a bug where expressions would not match to explicit type casts of matching expressions (pa-1133)
• Golang: Fixed a bug where metavariable ellipses as parameters to functions were not working properly (pa-2545)
• Apex: Fix name resolution of class attributes. Among other things, this will allow
  Semgrep to perform constant propagation of final class attributes.
  See https://semgrep.dev/s/DG6v. (pa-2551)
• Go: Fixed a bug where function arguments in a group that share the same type,
  such as "func foo(x, y, z int)", would parse all arguments after the first as
  having the name ",". For instance, "y" and "z" here would not have the correct
  names. (pa-2558)
• Fixed bug in comparison of Maven versions where multi digit versions would cause a default to raw string comparison (sc-maven-cmp-bug)

Release v1.12.1

1.12.1 - 2023-02-17

Fixed

• Fix local scan hyperlinks by asking git for remote.origin.url if repo_url not provided (gh-7144)
• Improve error messages displayed with --verbose when the contents of a metavariable fails to parse. (pa-2537)
• Fixed parsing bug maven_dep_tree.txt files where dependency specs with 6 fields, or suffixed with (optional) would fail to parse (sc-622)
• Supply Chain rules now correctly understand Maven version strings, as described in https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855 (sc-maven-versions)
• package-lock.json files which contain "resolved": false as a result of a bug in NPM will now parse (sc-npm-bug)

Release v1.12.0

1.12.0 - 2023-02-13

Fixed

• CLI: No longer reports rules as being run with a lack of interfile: true when interfile
  analysis was not requested. (pa-2528)

Release v1.11.0

1.11.0 - 2023-02-10

Changed

• Pro: Removed already deprecated flags --deep (now --pro), --interfile (now --pro),
  and --interproc (now --pro-intrafile). Also removed already deprecated command
  install-deep-semgrep (now install-semgrep-pro). (pa-2518)

Fixed

• Go: Fixed a bug where the location of Go raw string literals were not being reported correctly.
  This could cause issues with Playground range highlighting and autofix (pa-2206)
• CLI: Progress bar for Semgrep Pro Engine interfile scans now reflects actual progress more faithfully (pa-2313)
• Pro: We now check the downloaded binary before installing it, this should prevent
  installation of corrupted binaries due to errors during the download. (pa-2492)