Run e2e tests on each PR. #584

purple4reina · 2025-04-10T21:14:58Z

What does this PR do?

Run the serverless-e2e-tests on PR, skipping anything that isn't a python test. First publishes the layer build to sandbox account region us-west-2. Then, after gathering the arn for the new layer, pass that to the e2e tests for testing.

Motivation

Better to get alerted right away to potential e2e bugs.

Testing Guidelines

Additional Notes

Because this job is triggering job in a different repo, it is quite tough to access the results of that downstream job. As designed right now, a e2e-status job will be created in addition to the e2e-test job. This status job runs only after the e2e-test job completes. The reason this status job is necessary is so that we can post the results back to github.

The problem with this is the e2e-status job can take hours to fully complete. That's because not only do all the other tests in this repo need to run (integration, unit, layer size, etc), it also needs to run the e2e tests, and it must also wait for the full teardown of aws resources. The latter is what can take so long because these teardown jobs are set to run after a 2 hour delay. So, we technically aren't able to get results posted back to github until all of that completes.

The work around, which was too hard to implement with the time I had, is to use the gitlab api to poll the status of the e2e-test job. This might involve creating a Project Token which is more work that I think it's worth.

An alternative solution is to set the e2e-status test as required by github. There are some major problems with this though. There are times in which we would expect that the e2e tests would fail. For example, if I'm implementing a new feature which will turn an e2e test from not working to working, then the e2e tests themselves will always fail in that case, which is what we want. To get around that, we could remove the xfail strict requirement on the e2e tests downstream. But this would mean we aren't able to keep our feature parity doc in lock step with reality.

At this point, I think the best thing to do is to mark the e2e-status job as required. We are always able to override these failures in the UI, which shouldn't mean we're moving any slower.

Types of Changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog
This PR passes the integration tests (ask a Datadog member to run the tests)

https://datadoghq.atlassian.net/browse/APMSVLS-20

purple4reina · 2025-06-20T20:40:04Z

ci/input_files/build.yaml.tpl


 publish-layer-{{ $environment_name }} ({{ $runtime.name }}-{{ $runtime.arch }}):
  stage: publish
  tags: ["arch:amd64"]
  image: registry.ddbuild.io/images/docker:20.10-py3
  rules:
+    - if: '"{{ $environment_name }}" == "sandbox" && $REGION == "{{ $e2e_region }}" && "{{ $runtime.arch }}" == "amd64"'
+      when: on_success


Always be sure to publish to the sandbox account in us-west-2 for e2e testing.

purple4reina · 2025-06-20T20:40:32Z

ci/input_files/build.yaml.tpl

    - if: '"{{ $environment_name }}" == "sandbox"'
      when: manual
      allow_failure: true
    - if: '$CI_COMMIT_TAG =~ /^v.*/'
+  artifacts:
+    reports:
+      dotenv: {{ $dotenv }}


When saved as a dotenv report, this file will get sourced for its env vars in every downstream job.

purple4reina · 2025-06-20T20:41:43Z

ci/input_files/build.yaml.tpl

+    {{- range (ds "runtimes").runtimes }}
+    {{- if eq .arch "amd64" }}
+    {{- $version := print (.name | strings.Trim "python") }}
+      PYTHON_{{ $version }}_VERSION: $PYTHON_{{ $version }}_VERSION


Explicitly pass environment variables from the sourced publish job dotenv file to the downstream job. These env vars are not passed downstream otherwise.

purple4reina · 2025-06-20T20:42:03Z

ci/input_files/build.yaml.tpl

+    {{- end }}
+  needs: {{ range (ds "runtimes").runtimes }}
+    {{- if eq .arch "amd64" }}
+      - "publish-layer-sandbox ({{ .name }}-{{ .arch }}): [{{ $e2e_region }}]"


Run after all the publish jobs have completed.

purple4reina · 2025-06-20T20:43:32Z

ci/input_files/build.yaml.tpl

+    {{- end }}
+    {{- end }}
+
+e2e-status:


The purpose of this job is to simply relay the status of the downstream job back to the github pull request. Without this, the downstream e2e tests still run, and the e2e-test job will still properly fail. However, none of those failures would be posted to github, because gitlab will only post direct jobs, not jobs that it's triggering in another repo.

purple4reina · 2025-06-20T20:43:58Z

ci/input_files/build.yaml.tpl

+    - e2e-test
+    {{- range (ds "runtimes").runtimes }}
+    {{- if eq .arch "amd64" }}
+    - "publish-layer-sandbox ({{ .name }}-{{ .arch }}): [{{ $e2e_region }}]"


We need to need these in order to get the PYTHON_x_VERSION env vars.

On second thought, I don't actually think these are required. But I've left them in anyway just in case.

purple4reina · 2025-06-20T20:44:14Z

ci/input_files/build.yaml.tpl

+  image: registry.ddbuild.io/images/mirror/alpine:latest
+  tags: ["arch:amd64"]
+  needs:
+    - e2e-test


This job will run once the e2e-test job completes.

purple4reina · 2025-06-20T20:46:00Z

ci/publish_layers.sh

@@ -199,6 +199,8 @@ fi
 while [ $latest_version -lt $VERSION ]; do
    latest_version=$(publish_layer $REGION $layer $aws_cli_python_version_key $layer_path)
    printf "[$REGION] Published version $latest_version for layer $layer in region $REGION\n"
+    latest_arn=$(aws lambda get-layer-version --layer-name $layer --version-number $latest_version --region $REGION --query 'LayerVersionArn' --output text)
+    printf "[$REGION] Published arn $latest_arn\n"


Be sur to always print the arn to stdout when publishing layers. The publish gitlab jobs will do a regex search on this stdout to determine the arn for a newly published layer.

purple4reina force-pushed the rey.abolofia/run-e2e branch 3 times, most recently from f302bd5 to 6889e0f Compare April 10, 2025 21:54

purple4reina force-pushed the rey.abolofia/run-e2e branch from 6ab7639 to 33568a6 Compare May 1, 2025 15:41

purple4reina added 2 commits June 13, 2025 13:39

Always publish to us-east-1.

f19f26c

WIP

abfb1b6

purple4reina force-pushed the rey.abolofia/run-e2e branch from 33568a6 to abfb1b6 Compare June 13, 2025 20:39

Publishes to us-west-2.

2b283fd

purple4reina force-pushed the rey.abolofia/run-e2e branch 22 times, most recently from 49efd46 to 252675f Compare June 16, 2025 21:22

purple4reina added 2 commits June 18, 2025 13:40

Add e2e-status check.

b159dcf

Improve output of job.

0c3f604

purple4reina force-pushed the rey.abolofia/run-e2e branch from 6eeab24 to 91edc88 Compare June 20, 2025 20:23

Needing publish loads dotenv artifact.

7255a2b

purple4reina force-pushed the rey.abolofia/run-e2e branch from 91edc88 to 7255a2b Compare June 20, 2025 20:24

purple4reina commented Jun 20, 2025

View reviewed changes

purple4reina added 7 commits June 20, 2025 14:18

Have e2e-status job start right away then poll.

d158eb0

Comment out tests for now.

afda470

Use apk.

b2b1ba5

Debugging.

92607f2

Try JOB-TOKEN.

a0c1a76

Print out URL.

6924947

Try bridges api.

dfdc510

purple4reina force-pushed the rey.abolofia/run-e2e branch from 1b774ac to dfdc510 Compare June 20, 2025 22:53

Try GITLAB_API_TOKEN.

20a2b56

purple4reina force-pushed the rey.abolofia/run-e2e branch from 4ce0906 to 20a2b56 Compare June 20, 2025 23:02

Use CI_JOB_STATUS to determine pass/fail.

75357ef

purple4reina force-pushed the rey.abolofia/run-e2e branch from cf6e21c to 75357ef Compare June 21, 2025 00:15

purple4reina marked this pull request as ready for review June 21, 2025 00:22

purple4reina requested review from a team as code owners June 21, 2025 00:22

Remove switch.

57056ba

purple4reina force-pushed the rey.abolofia/run-e2e branch from bfe7f03 to 57056ba Compare June 21, 2025 00:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run e2e tests on each PR. #584

Run e2e tests on each PR. #584

purple4reina commented Apr 10, 2025 •

edited

Loading

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 21, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

Uh oh!

Run e2e tests on each PR. #584

Are you sure you want to change the base?

Run e2e tests on each PR. #584

Conversation

purple4reina commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of Changes

Check all that apply

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

purple4reina commented Apr 10, 2025 •

edited

Loading