feat: allow defining a git service from defaults.ini (#694)

Signed-off-by: Trong Nhan Mai <[email protected]>

Author: tromai

docs/source/pages/output_files.rst

@@ -131,7 +131,7 @@ to the directory:
     └── git_repos
         └── local_repos
 
-.. note:: Please see :ref:`pages/using:analyzing a locally cloned repository` to know how to set the directory for analyzing local repositories.
+.. note:: Please see :ref:`pages/using:analyzing a repository on the local file system` to know how to set the directory for analyzing local repositories.
 
 .. _output_files_macaron_verify_policy:
 

docs/source/pages/supported_technologies/index.rst

@@ -24,6 +24,8 @@ such as GitHub Actions workflows.
 * Docker
 
 
+.. _supported_git_services:
+
 ------------
 Git Services
 ------------

docs/source/pages/using.rst

@@ -51,7 +51,7 @@ Take the same example as above, to disable analyzing `micronaut-core <https://gi
 
   ./run_macaron.sh analyze -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603 --skip-deps
 
-.. note:: By default, Macaron would generate report files into the ``output`` directory in the current workspace. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
+.. note:: By default, Macaron would generate report files into the ``output`` directory in the current working directory. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
 
 With the example above, the generated output reports can be seen here:
 
@@ -238,7 +238,7 @@ With the example above, the generated output reports can be seen here:
 Analyzing dependencies in the SBOM without the main software component
 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
 
-In the case where the repository URL of the main software component is not available (e.g. the repository is in a private domain where Macaron cannot access),
+In the case where the repository URL of the main software component is not available (e.g. the repository is in a self-hosted git service instance where Macaron cannot access),
 Macaron can still run the analysis on the dependencies listed in the SBOM.
 To do that, you must first create a PURL to present the main software component. This is so that this software component could be referenced later in the :ref:`verify-policy <verify-policy-command-cli>` command.
 For example: ``pkg:private_domain.com/org/name``.
@@ -300,9 +300,9 @@ An example configuration file for utilising this feature:
 
 
 
--------------------------------------
-Analyzing a locally cloned repository
--------------------------------------
+-----------------------------------------------
+Analyzing a repository on the local file system
+-----------------------------------------------
 
 .. warning::
   During the analysis, Macaron can check out different commits, which can reset the index and working tree of the repository.
@@ -312,9 +312,55 @@ Analyzing a locally cloned repository
 .. note::
   We assume that the ``origin`` remote exists in the cloned repository and checkout the relevant commits from ``origin`` only.
 
-If you have a local repository that you want to analyze, Macaron also supports running the analysis against a local repository.
+Macaron supports analyzing a repository on the local file system.
 
-Assume that the dir tree at the local repository has the following components:
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+Analyzing a repository whose git service is not supported by Macaron
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+If the repository remote URL is from an unknown git service (see :ref:`Git Services <supported_git_services>` for a list of supported git services in Macaron), Macaron won't recognize it when analyzing the repository.
+
+You would need to tell Macaron about that git service through the ``defaults.ini`` config.
+For example, let's say you want to analyze a repository hosted at ``https://git.example.com/foo/target``. First, you need to create a ``defaults.ini`` file in the current working directory with the following content:
+
+.. code-block:: ini
+
+  [git_service.local_repo]
+  hostname = git.example.com
+
+In which ``hostname`` contains the hostname of the git service URL. In this example it is ``git.example.com``.
+
+.. note::
+
+  This ``defaults.ini`` section must only be used for analyzing a repository on the local file system. If the hostname has already been supported in other services, it doesn't need to be defined again here.
+
+Assume that the dir tree at the current working directory has the following structure:
+
+.. code-block:: shell
+
+  boo
+  ├── foo
+  │   └── target
+
+We can run Macaron against the local repository at ``target`` by using this command:
+
+.. code-block:: shell
+
+  ./run_macaron.sh --local-repos-path ./boo/foo --defaults-path ./defaults.ini analyze --repo-path target <rest_of_args>
+
+With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
+
+The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
+
+.. note:: If ``--local-repos-path/-lr`` is not provided, Macaron will looks inside ``<current_working_directory>/output/git_repos/local_repos/`` whenever you provide a local path to ``--repo-path/-rp``.
+
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''
+Analyzing a local repository with supported git service
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+If the local repository you want to analyze has a remote origin hosted on a supported git service, you can run the analysis directly without having to prepare ``defaults.ini`` as above.
+
+Assume that the dir tree at the current working directory has the following structure:
 
 .. code-block:: shell
 
@@ -326,13 +372,13 @@ We can run Macaron against the local repository at ``target`` by using this comm
 
 .. code-block:: shell
 
-  ./run_macaron.sh -lr path/to/boo/foo analyze -rp target <rest_of_args>
+  ./run_macaron.sh --local-repos-path ./boo/foo analyze --repo-path target <rest_of_args>
 
-With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``-b``, ``-d`` or ``--skip-deps`` similar to two previous examples)
+With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
 
-The ``-lr`` flag configure Macaron to looks into ``path/to/boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
+The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
 
-.. note:: If ``-lr`` is not provided, Macaron will looks inside ``<working_directory>/output/git_repos/local_repos/`` whenever you provide a local path to ``-rp``.
+.. note:: If ``--local-repos-path/-lr`` is not provided, Macaron will looks inside ``<current_working_directory>/output/git_repos/local_repos/`` whenever you provide a local path to ``--repo-path/-rp``.
 
 -------------------------
 Running the policy engine

scripts/dev_scripts/integration_tests.sh

@@ -434,6 +434,17 @@ echo -e "\n=====================================================================
 echo "Run integration tests with local paths for apache/maven..."
 echo -e "==================================================================================\n"
 
+echo -e "\n----------------------------------------------------------------------------------"
+echo "bitbucket.org/snakeyaml/snakeyaml: Analyzing a repository with un-supported git service as local repo without dependency resolution."
+echo -e "----------------------------------------------------------------------------------\n"
+git clone https://bitbucket.org/snakeyaml/snakeyaml $WORKSPACE/output/local_repos/snakeyaml || log_fail
+DEFAULTS_FILE=$WORKSPACE/tests/e2e/defaults/bitbucket_local_repo.ini
+JSON_EXPECTED=$WORKSPACE/tests/e2e/expected_results/snakeyaml/snakeyaml.json
+JSON_RESULT=$WORKSPACE/output/reports/bitbucket_org/snakeyaml/snakeyaml/snakeyaml.json
+$RUN_MACARON -dp $DEFAULTS_FILE -lr $WORKSPACE/output/local_repos analyze -rp snakeyaml -d a34989252e6f59e36a3aaf788a903b7a37a73d33 --skip-deps || log_fail
+
+check_or_update_expected_output $COMPARE_JSON_OUT $JSON_RESULT $JSON_EXPECTED || log_fail
+
 echo -e "\n----------------------------------------------------------------------------------"
 echo "apache/maven: Analyzing with the branch name, the commit digest and dependency resolution using cyclonedx maven plugin (default)."
 echo -e "----------------------------------------------------------------------------------\n"

src/macaron/config/defaults.ini

@@ -90,6 +90,12 @@ hostname = gitlab.com
 # [git_service.gitlab.self_hosted]
 # hostname = example.org
 
+# This section defines a git service that Macaron doesn't recognize yet.
+# It must only be used for analyzing a locally cloned repository.
+# If the host name is already supported in other services, it doesn't need to be defined again here.
+# [git_service.local_repo]
+# hostname = example.org
+
 # This is the spec for trusted Maven build tools.
 [builder.maven]
 entry_conf = settings.xml

src/macaron/slsa_analyzer/git_service/__init__.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2022 - 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2022 - 2024, Oracle and/or its affiliates. All rights reserved.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
 
 """The git_service package contains the supported git services for Macaron."""
@@ -7,7 +7,14 @@
 from .bitbucket import BitBucket
 from .github import GitHub
 from .gitlab import PubliclyHostedGitLab, SelfHostedGitLab
+from .local_repo_git_service import LocalRepoGitService
 
 # The list of supported git services. The order of the list determines the order
 # in which each git service is checked against the target repository.
-GIT_SERVICES: list[BaseGitService] = [GitHub(), PubliclyHostedGitLab(), SelfHostedGitLab(), BitBucket()]
+GIT_SERVICES: list[BaseGitService] = [
+    GitHub(),
+    PubliclyHostedGitLab(),
+    SelfHostedGitLab(),
+    BitBucket(),
+    LocalRepoGitService(),
+]

src/macaron/slsa_analyzer/git_service/local_repo_git_service.py

@@ -0,0 +1,42 @@
+# Copyright (c) 2024 - 2024, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
+
+"""This module contains the spec for the local repo git service."""
+
+import logging
+
+from pydriller.git import Git
+
+from macaron.errors import ConfigurationError, RepoCheckOutError
+from macaron.slsa_analyzer import git_url
+from macaron.slsa_analyzer.git_service.base_git_service import BaseGitService
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+
+class LocalRepoGitService(BaseGitService):
+    """This class contains the spec of the local repo git service."""
+
+    def __init__(self) -> None:
+        """Initialize instance."""
+        super().__init__("local_repo")
+
+    def load_defaults(self) -> None:
+        """Load the values for this git service from the ini configuration."""
+        try:
+            self.hostname = self.load_hostname(section_name="git_service.local_repo")
+        except ConfigurationError as error:
+            raise error
+
+    def clone_repo(self, _clone_dir: str, _url: str) -> None:
+        """Cloning from a local repo git service is not supported."""
+        raise NotImplementedError
+
+    def check_out_repo(self, git_obj: Git, branch: str, digest: str, offline_mode: bool) -> Git:
+        """Checkout the branch and commit specified by the user of a repository."""
+        if not git_url.check_out_repo_target(git_obj, branch, digest, offline_mode):
+            raise RepoCheckOutError(
+                f"Failed to check out branch {branch} and commit {digest} for repo {git_obj.project_name}."
+            )
+
+        return git_obj

tests/e2e/defaults/bitbucket_local_repo.ini

@@ -0,0 +1,5 @@
+# Copyright (c) 2024 - 2024, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
+
+[git_service.local_repo]
+hostname = bitbucket.org