feat: add a check to analyze malicious Python packages (#750)

Signed-off-by: Yao-Wen-Chang <[email protected]>

Author: Yao-Wen-Chang

pyproject.toml

@@ -34,6 +34,7 @@ dependencies = [
     "jsonschema >= 4.22.0,<5.0.0",
     "cyclonedx-bom >=4.0.0,<5.0.0",
     "cyclonedx-python-lib[validation] >=7.3.4,<8.0.0",
+    "beautifulsoup4 >= 4.12.0,<5.0.0",
 ]
 keywords = []
 # https://pypi.org/classifiers/
@@ -74,6 +75,7 @@ dev = [
     "pip-audit >=2.5.6,<3.0.0",
     "pylint >=3.0.3,<4.0.0",
     "cyclonedx-bom >=4.0.0,<5.0.0",
+    "types-beautifulsoup4 >= 4.12.0,<5.0.0",
 ]
 docs = [
     "sphinx >=7.0.0,<8.0.0",

src/macaron/config/defaults.ini

@@ -512,6 +512,10 @@ hostname = registry.npmjs.org
 attestation_endpoint = -/npm/v1/attestations
 request_timeout = 20
 
+[package_registry.pypi]
+request_timeout = 20
+hostname = pypi.org
+
 # Configuration options for selecting the checks to run.
 # Both the exclude and include are defined as list of strings:
 #   - The exclude list is used to specify the checks that will not run.
@@ -547,3 +551,9 @@ request_timeout = 20
 exclude =
 # By default, we run all checks available.
 include = *
+
+[heuristic.pypi]
+releases_frequency_threshold = 2
+# The gap threshold.
+# The timedelta indicate the gap between the date maintainer registers their pypi's account and the date of latest release.
+timedelta_threshold_of_join_release = 5

src/macaron/malware_analyzer/README.md

@@ -0,0 +1,67 @@
+# Implementation of Heuristic Malware Detector
+
+## Check
+
+We schedule the heuristics sequentially:
+
+1. **Empty Project Link**: If the package contains project links (e.g., documentation, Git Repositories),
+the analyzer will further operate the heuristic `Unreachable Project Links` to analyze if all the project links are not reachable.
+2. **One Release**: Checks if there is only one release of the package. If the package contains multiple
+releases, the checker will further check the release frequency through `High Release Frequency` and
+`Unchanged Release` to see if the maintainers release multiple times in a short timeframe (threshold) and
+whether the released contents are identical.
+3. **Closer Release Join Date**: Considers the date when the maintainer registered their account (if
+available). The checker will calculate the gap between the latest release date and the maintainer's account
+registration date.
+4. **Suspicious Setup**: Checks whether the `setup.py` includes suspicious imports, such as `base64` for
+encryption and `requests` for data exfiltration.
+
+## Supported Ecosystem: PyPI
+
+Define Seven Heuristics: `False` means suspicious and `True` means non-suspicious. `SKIP` means some metadata are missing, and the checker skips the heuristic.
+
+1. **Empty Project Link**
+   - **Description**: Checks whether the package contains any project links (e.g., documents or Git
+   Repositories). Many malicious activities do not include any project link.
+   - **Rule**: Return `FALSE` when there is only one project link; otherwise, return `TRUE`.
+
+2. **Unreachable Project Links**
+   - **Description**: Checks the accessibility of the project links. This is considered an auxiliary
+   heuristic since no cases have met this heuristic.
+   - **Rule**: Return `FALSE` if all project links are not reachable; otherwise, return `TRUE`.
+
+3. **One Release**
+   - **Description**: Checks whether the package has only one release.
+   - **Rule**: Return `FALSE` if the package contains only one release; otherwise, return `TRUE`.
+
+4. **High Release Frequency**
+   - **Description**: Checks if the package released multiple versions within a short period. We calculate
+   the release frequency and define a default frequency threshold of 2 days.
+   - **Rule**: Return `FALSE` if the frequency is higher than the threshold; otherwise, return `TRUE`.
+
+5. **Unchanged Release**
+   - **Description**: Checks if the content of releases remains unchanged.
+   - **Rule**: Return `FALSE` if the content of releases is identical; otherwise, return `TRUE`.
+
+6. **Closer Release Join Date**
+   - **Description**: Checks the gap between the date the maintainer registered their account and the date
+   of the latest release. A default threshold of 5 days is defined.
+   - **Rule**: Return `FALSE` if the gap is less than the threshold; otherwise, return `TRUE`.
+
+7. **Suspicious Setup**
+   - **Description**: Checks the `setup.py` to see if there are suspicious imported modules and the
+   `install_requires` packages installed during the package installation process. We define two suspicious
+   keywords as the blacklist.
+   - **Rule**: Return `FALSE` if the package name contains suspicious keywords; otherwise, return `TRUE`.
+
+## Heuristics-Based Analyzer: Scanning 1167 Packages from Trusted Organizations
+
+| Heuristic Name     | Count |
+| ------------------ | ----- |
+| Lower Release      | 102   |
+| Empty Link         | 45    |
+| Links Missing      | 24    |
+| Frequent Release   | 14    |
+| Suspicious Setup   | 5     |
+
+**The result is used as a reference for the confidence score to lower the false positive rate.**

src/macaron/malware_analyzer/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (c) 2022 - 2024, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.

src/macaron/malware_analyzer/checks/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (c) 2022 - 2024, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.