July 2023 release notes

Semgrep OSS Engine

This section of release notes includes upgrades of Semgrep OSS Engine for versions between 1.31.0 and 1.34.1.

Added

• Added rule option interfile: true, which is set under the options key. This is now the preferred method of setting interfile to true. While interfile can still be set under the metadata key, this should be avoided because metadata is not meant to have any effect on how a rule is run.

• Added new --legacy flag to force the use of the Python implementation of Semgrep (also known as 'pysemgrep'). Note that by default most semgrep commands are still using the Python implementation (except semgrep interactive), so in practice you don't need to add this flag, but as Semgrep Inc. ports more commands to OCaml, the new --legacy flag might be useful if you find some regressions.

• Julia: Added support for metavariable type.

• PromQL (Prometheus Query Language): Initial language support. Thank you to Michael Hoffman for his contribution! (#8281)

• PromQL: Added parse_promql_duration function to convert a PromQL duration into milliseconds(#8381). This makes it possible to write comparisons such as:

  - metavariable-comparison:
      metavariable: $RANGE
          comparison: parse_promql_duration(str($RANGE)) > parse_promql_duration("1d")

• .h files now run when C or C++ are selected as the language.

• .cjs and .mjs files now run when JavaScript is selected as the language.

• Rule syntax: Added metavariable type extension for Semgrep Rule Syntax 2.0 (also known as Experimental Semgrep Syntax). This addition introduces recent changes in Semgrep rule syntax 1.0 to the experimental syntax as well. (#8183)

  # previous rule syntax 2.0
  rules:
    - id: no-string-eqeq
      message: find errors
      severity: WARNING
      languages:
        - java
      match:
        all:
          - not: null == (String $Y)
          - $X == (String $Y)

  # new additions to rule syntax 2.0 (experimental syntax)
  rules:
    - id: no-string-eqeq
      message: find errors
      severity: WARNING
      languages:
        - java
      match:
        all:
          - not: null == $Y
          - $X == $Y
        where:
          - metavariable: $Y
            type: String

• Taint analysis: Parameters to functions in languages with pattern matching in function arguments, such as Rust and OCaml, now transmit taint when they are sources. This works with nested patterns too. For example, in Rust:

  fn f ((x, (y, z)): t) {
    let x = 2;
  }

  Tainting the sole argument to this function results in all of the identifiers x, y, and z now being tainted. (#8216)

• Rust: Added support for ellipsis patterns in attribute argument positions. For example, #[get(...)]. (#8148)

• Rust: Added typed metavariable support for Rust. Users can create TypedMetavar using Rust's type annotation syntax :. For example, the following rule works for matching HttpResponseBuilder type of variables:

  rules:
  - id: no-direct-response-write
    patterns:
    - pattern: '($BUILDER : HttpResponseBuilder).body(...)'
    - pattern-not: '($BUILDER : HttpResponseBuilder).body("...".to_string())'
    message: find dangerous codes
    severity: WARNING
    languages: [rust]

• Rust: Added the ability to taint macro calls through its arguments, in macro calls with multiple arguments. (#8209)

• Matching: Added the ability to use metavariables in parameters to match more sophisticated kinds of parameters. In particular, metavariables should now be able to match self parameters, such as in Rust. For example:

  fn $F($X, ...) { ... }

  should match:

  fn $F(self) { }

• Added support for naming propagation when the left-hand side (LHS) of a variable definition is an identifier pattern. In certain languages such as Rust, the variable definition is parsed as a pattern assignment, for example:

  let x: SomeType = SomeFunction();

  This commit ensures that the annotated type is propagated to the identifier pattern on the left-hand side (LHS) of the assignment, thus ensuring proper naming behavior.

• Taint-mode: Added experimental control: true option to pattern-sources. For example:

  pattern-sources:
    - control: true
        pattern: source(...)
  • Such sources taint the "control flow" (or the program counter) so that it is possible to implement reachability queries that do not require the flow of any data. Thus, Semgrep reports a finding in the code below, because after source() the flow of control will reach sink(), even if no data is flowing between both:

  def test():
    source()
    foo()
    bar()
    #ruleid: test
    sink()
  ``` (pa-2958)

• Taint mode: Taint sanitizers will be included in matching explanations. (#8383)

Fixed

• Dockerfile language support: String matching is now done by contents, treating the strings foo, 'foo', or "foo" as equal. (#8229)
• Dockerfile: Single-quoted strings are now parsed without an error. (#7780)
• Julia: Fixed a bug where try-catch patterns would not match properly. Now, you can use an empty try-catch pattern, such as:

  try
    ...
  catch
    ...
  end
  to catch only Julia code which does not specify an identifier for the catch.
  • Otherwise, if you want to match any kind of try-catch, you can specify an ellipsis for the catch identifier instead:

  try
    ...
  catch ...
    ...
  end
  and this matches any try-catch, including those that do not specify an identifier for the catch. It is strictly more general than the previous.
• TypeScript and JavaScript: Fixed an issue leading to incorrect autofix results involving JS/TS async arrow functions, for example, async () => {} (#7353)
• Go: Fixed issue with patterns such as:
  • make(...);
  • make(...,$X);
  • make($A,$B) (#8171)
• Rust: Fixed an issue where implicit returns did not allow taint to flow, and various other small translation issues that would affect taint. (#8325)
• Rust: Fixed attribute patterns to allow matching on simple attribute syntax. (#8234)
• Rust: Fixed a bug where standalone metavariable patterns were not matching as expected. (#8206)
• Rust: Macro calls which involve dereferencing and reference operators (such as foo!(&x) and foo!(*x)) now properly transmit taint.
• Fixed Python Semgrep pattern parsing to also parse match statements, by chaining in the Python tree-sitter parser, and adding metavariable support to the Python tree-sitter parser.
• Aliengrep mode: Fix whitespace bug preventing correct matching of parentheses. (#7990)
• Fixed stack overflow caused by symbolic propagation.

Removed

• Dart has been removed from experimental support.

Semgrep Cloud Platform

Added

• Jira integration is now in private beta for existing customers.
  • You can create Jira tickets from Semgrep Cloud Platform by following steps in the Jira integration documentation.
  • To enable this feature:
    1. Fill out the following form: Request access to the Semgrep Jira integration private beta.
    2. Contact your Technical Account Manager or your Account Executive and let them know you'd like to try out the Jira integration.
• Usage limits are now in effect as of July 31, 2023. See the Usage document to learn more.
• Various bugfixes and improvements.

Semgrep Code

Added

• Added open findings column to the Policies page. This column displays the count of open findings for each rule in your Policies. With this column, you can quickly see which rules are producing the most or least amount of findings.
• Added fix rate column to the Policies page. This column displays a percentage of resolved fixes associated with the rule, or "-" if there are no findings for that rule.
• Added Label as its own column in the Policies page.
• Added icons for Severity and Source columns in the Policies page. The following icons can be seen in the Source columns:
  • Custom rules
  • Community rules
  • Semgrep Pro rules

Changed

• Policies page: Rules are now sorted by mode, then by custom rules, then by Semgrep Pro rules, then by Community rules.
• Targets in a .yarn/ folder or directory are now ignored by the default .semgrepignore patterns.

Fixed

• Fixed an issue with the Findings page > Triage button in which the Ignore button was previously disabled when the Note or Comment textbox was empty. It has been fixed to let users Ignore the finding without filling the text box.

Removed

• The Rule board has been deprecated and removed. The Policies page is now the default and sole page for rule management in Semgrep Cloud Platform.

Semgrep Supply Chain

Fixed

• Fixed bug in gradle.lockfile parser where Semgrep Supply Chain previously threw errors on empty= with nothing after it.
• poetry.lock parsing: Semgrep Supply Chain now correctly handles empty toml tables, quoted table keys, and arbitrarily placed comments.
• Exceptions raised during parsing of manifest files no longer interrupt general parser execution, which previously prevented lockfile parsing if a manifest failed to parse.

Semgrep Assistant

Added

• Semgrep Assistant is now in public beta. Semgrep Assistant is available to any user of Semgrep Cloud Platform. To try it out, see Enabling Semgrep Assistant.
• Semgrep Assistant now suggests rule categories for your rules through the Assistant recommendations in the Dashboard page. Click the Accept button for Semgrep Assistant to automatically update the rule with its suggested category.

Documentation and knowledge base updates

Added

• Semgrep documentation has added ChatGPT-4 as an experimental means for users to learn about Semgrep. Click the floating chat icon and enter a question to receive answers and sources for those answers. This service is powered by Markprompt.
• Added the following documentation articles:
  • Semgrep integration guide for partners: Use these guidelines to add Semgrep OSS to your tooling, developer stack, or infrastructure.
• Added the following knowlege base articles:
  • Semgrep scan troubleshooting
  • Resolving SSO error BadRequest: Missing attribute
  • How to run different versions of Semgrep
  • Importing Semgrep findings into DefectDojo
  • Why did the comments on a PR or MR not appear inline?
  • Why aren't findings populating in the GitHub Advanced Security Dashboard after running Semgrep in CI?
• Added explicit steps to publish private rules in the Semgrep Registry.
• Added a new section on metavariable-type.
• Various piecemeal updates.

Changed

• Migrated troubleshooting articles from Semgrep documentation to the knowledge base.
• Updated Semgrep Assistant for Code docs to reflect its public beta status and new self-serve flow.

Fixed

• Various typographic and layout fixes.