test(ai): add integration tests #8853

dlarocque · 2025-03-19T19:57:39Z

Adds initial setup for integration tests, and integration tests for count tokens, generate content, and chat. Each test runs against both backends (Vertex AI and Google AI), for each specified model in the config.

These tests aim to catch issues that are caused by changes in the backend (e.g. A required field suddenly no longer being sent), and to allow us to confidently support new models by just adding them to the set of models we test against.

Future PRs will integrate this into our CI.

Sample output:

  Chat Session
    Google AI gemini-2.0-flash
      ✔ startChat and sendMessage: text input, text output
    Vertex AI gemini-2.0-flash
      ✔ startChat and sendMessage: text input, text output
  Count Tokens
    Google AI gemini-2.0-flash
      ✔ text input
      ✔ image input
      ✔ audio input
      ✔ text, image, and audio input
      ✔ public storage reference
    Vertex AI gemini-2.0-flash
      ✔ text input
      ✔ image input
      ✔ audio input
      ✔ text, image, and audio input
      ✔ public storage reference
  Generate Content
    Google AI gemini-2.0-flash
      ✔ generateContent: text input, text output
      ✔ generateContentStream: text input, text output
    Vertex AI gemini-2.0-flash
      ✔ generateContent: text input, text output
      ✔ generateContentStream: text input, text output

changeset-bot · 2025-03-19T19:57:43Z

⚠️ No Changeset found

Latest commit: 63f7215

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

google-oss-bot · 2025-03-19T20:07:10Z

Size Report ¹

Affected Products

@firebase/firestore
Type Base (770e455) Merge (572f535) Diff
browser 385 kB 385 kB +151 B (+0.0%)
module 385 kB 385 kB +151 B (+0.0%)
react-native 385 kB 386 kB +151 B (+0.0%)

`bundle`

Type	Base (`770e455`)	Merge (`572f535`)	Diff
firestore (CSI Auto Indexing Disable and Delete)	279 kB	280 kB	+151 B (+0.1%)
firestore (CSI Auto Indexing Enable)	279 kB	280 kB	+151 B (+0.1%)
firestore (Persistence)	311 kB	311 kB	+151 B (+0.0%)
firestore (Query Cursors)	256 kB	256 kB	+151 B (+0.1%)
firestore (Query)	254 kB	254 kB	+151 B (+0.1%)
firestore (Read data once)	242 kB	242 kB	+151 B (+0.1%)
firestore (Read Write w Persistence)	335 kB	336 kB	+151 B (+0.0%)
firestore (Realtime updates)	244 kB	244 kB	+151 B (+0.1%)
firestore (Transaction)	221 kB	221 kB	+151 B (+0.1%)
firestore (Write data)	220 kB	220 kB	+151 B (+0.1%)

firebase
Type Base (770e455) Merge (572f535) Diff
firebase-compat.js 802 kB 802 kB +149 B (+0.0%)
firebase-firestore-compat.js 347 kB 347 kB +149 B (+0.0%)
firebase-firestore.js 449 kB 449 kB +151 B (+0.0%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/eQugwWJ3mB.html

config/karma.base.js

.gitignore

hsubox76 · 2025-03-19T20:17:50Z

packages/vertexai/package.json

@@ -39,6 +39,7 @@
    "test:ci": "yarn testsetup && node ../../scripts/run_tests_in_ci.js -s test",
    "test:skip-clone": "karma start",
    "test:browser": "yarn testsetup && karma start",
+    "test:integration": "karma start --integration",


You'll have to call this from "test" (line 38) if you want to run it on PRs or do you not want to do that yet?

I would prefer not to add this to test since then it'll be ran when someone runs test in the root directory- which is going to fail if someone hasn't defined the config that's used in these tests. Could be annoying for people working on other SDKs

google-oss-bot · 2025-03-19T20:18:38Z

Size Analysis Report ¹

Affected Products

@firebase/firestore
- addDoc
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 134 kB 135 kB +151 B (+0.1%)
  size-with-ext-deps 206 kB 207 kB +151 B (+0.1%)
- deleteDoc
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 125 kB 125 kB +151 B (+0.1%)
  size-with-ext-deps 197 kB 197 kB +151 B (+0.1%)
- disableNetwork
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 113 kB 113 kB +151 B (+0.1%)
  size-with-ext-deps 185 kB 185 kB +151 B (+0.1%)
- enableIndexedDbPersistence
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 189 kB 189 kB +151 B (+0.1%)
  size-with-ext-deps 261 kB 262 kB +151 B (+0.1%)
- enableMultiTabIndexedDbPersistence
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 225 kB 226 kB +151 B (+0.1%)
  size-with-ext-deps 298 kB 298 kB +151 B (+0.1%)
- enableNetwork
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 113 kB 113 kB +151 B (+0.1%)
  size-with-ext-deps 185 kB 185 kB +151 B (+0.1%)
- executeWrite
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 124 kB 124 kB +151 B (+0.1%)
  size-with-ext-deps 196 kB 196 kB +151 B (+0.1%)
- getAggregateFromServer
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 121 kB 121 kB +151 B (+0.1%)
  size-with-ext-deps 193 kB 193 kB +151 B (+0.1%)
- getCountFromServer
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 121 kB 121 kB +151 B (+0.1%)
  size-with-ext-deps 193 kB 193 kB +151 B (+0.1%)
- getDoc
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 155 kB 156 kB +151 B (+0.1%)
  size-with-ext-deps 227 kB 228 kB +151 B (+0.1%)
- getDocFromServer
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 156 kB 156 kB +151 B (+0.1%)
  size-with-ext-deps 227 kB 228 kB +151 B (+0.1%)
- getDocs
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 157 kB 157 kB +151 B (+0.1%)
  size-with-ext-deps 229 kB 229 kB +151 B (+0.1%)
- getDocsFromServer
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 157 kB 157 kB +151 B (+0.1%)
  size-with-ext-deps 229 kB 229 kB +151 B (+0.1%)
- loadBundle
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 123 kB 123 kB +151 B (+0.1%)
  size-with-ext-deps 195 kB 195 kB +151 B (+0.1%)
- memoryLocalCache
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 107 kB 107 kB +151 B (+0.1%)
  size-with-ext-deps 179 kB 179 kB +151 B (+0.1%)
- onSnapshot
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 158 kB 158 kB +151 B (+0.1%)
  size-with-ext-deps 230 kB 230 kB +151 B (+0.1%)
- onSnapshotsInSync
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 146 kB 147 kB +151 B (+0.1%)
  size-with-ext-deps 218 kB 219 kB +151 B (+0.1%)
- persistentLocalCache
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 185 kB 185 kB +151 B (+0.1%)
  size-with-ext-deps 257 kB 258 kB +151 B (+0.1%)
- persistentMultipleTabManager
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 221 kB 221 kB +151 B (+0.1%)
  size-with-ext-deps 293 kB 293 kB +151 B (+0.1%)
- persistentSingleTabManager
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 185 kB 185 kB +151 B (+0.1%)
  size-with-ext-deps 257 kB 257 kB +151 B (+0.1%)
- runTransaction
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 135 kB 135 kB +151 B (+0.1%)
  size-with-ext-deps 207 kB 207 kB +151 B (+0.1%)
- setDoc
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 134 kB 134 kB +151 B (+0.1%)
  size-with-ext-deps 206 kB 206 kB +151 B (+0.1%)
- updateDoc
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 134 kB 134 kB +151 B (+0.1%)
  size-with-ext-deps 206 kB 206 kB +151 B (+0.1%)
- waitForPendingWrites
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 114 kB 114 kB +151 B (+0.1%)
  size-with-ext-deps 186 kB 186 kB +151 B (+0.1%)
- writeBatch
  Size
  Type Base (770e455) Merge (572f535) Diff
  size 136 kB 136 kB +151 B (+0.1%)
  size-with-ext-deps 208 kB 208 kB +151 B (+0.1%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/6GTZZWxcMS.html

github-actions · 2025-05-16T15:28:06Z

Vertex AI Mock Responses Check ⚠️

A newer major version of the mock responses for Vertex AI unit tests is available. update_vertexai_responses.sh should be updated to clone the latest version of the responses: v13.0

…irebase-js-sdk into dl/vertex-integration

* Version Packages (#9016) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Revert "Fixed scroll behavior (#8977)" (#9053) * Revert "Fixed scroll behavior (#9043)" This reverts commit 9bcd1ea. * Revert "Add Emulator Overlay (#8977) (#9031)" This reverts commit 8593fa0. * Undid revert of util * Create dirty-crews-cross.md * Removed unnecessary markdown file * Updated to only show banner when calling connect*Emulator (#9055) * Version Packages (#9054) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Close webchannel instances on terminate (#9041) Explicitly close any unclosed WebChannel instances on Firestore#terminate() * Reorder reporters to prevent log message duplication in CI (#9040) * docs(ai): Fix 'occured' typo in error codes (#9021) * feat(ai): Add `HarmBlockThreshold.OFF` (#9050) * feat(ai): Schema `title`, `maximum`, `minimum`, and `propertyOrdering` (#9047) * feat(ai): Add support for `minItems` and `maxItems` to `Schema` (#9026) * Build out e2e subdirectories (#9065) * Added test project to fdc (#8924) * Version Packages (#9073) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Release version 11.9.0. * test(ai): add integration tests (#8853) * Update issue template to rename VertexAI > AI (#9074) * ci: Fix Firefox error and streamline cross-browser tests (#9015) * Fixed issue where requestSts wasn't including the Studio cookie in it (#9075) * Fix issue where we ping an endpoint that hasn't been implemented (#9059) * Updated firebase auth changeset (#9088) * build(all): Version Packages v11.9.1 (#9089) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> v11.9.1 release. * test(ai): Upgrade mock responses to v14 (#9091) --------- Co-authored-by: Google Open Source Bot <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Maneesh Tewani <[email protected]> Co-authored-by: Mark Duckworth <[email protected]> Co-authored-by: Daniel La Rocque <[email protected]> Co-authored-by: Christina Holland <[email protected]>

dlarocque added 5 commits March 18, 2025 13:40

Skeleton

98574c2

Integration tests

ae05e53

Import firebase config from secret file

d83b22e

Validate that config file exists.

f78282a

Cleanup

0d1fb41

dlarocque requested a review from a team as a code owner March 19, 2025 19:57

dlarocque requested review from DellaBitta and hsubox76 March 19, 2025 19:57

dlarocque added 2 commits March 19, 2025 15:59

Format

b083d00

Lint fix

4d6a5ce

hsubox76 reviewed Mar 19, 2025

View reviewed changes

config/karma.base.js Outdated Show resolved Hide resolved

Extend preprocessor in vertex ai karma conf

3fbb6d6

hsubox76 reviewed Mar 19, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

dlarocque added 2 commits March 19, 2025 16:12

format

d446c77

Add placeholder integration/firebase-config.ts

fe6528b

hsubox76 reviewed Mar 19, 2025

View reviewed changes

dlarocque requested a review from hsubox76 March 20, 2025 16:07

Format

7ad7833

dlarocque force-pushed the dl/vertex-integration branch from 346de4e to 7ad7833 Compare March 21, 2025 20:07

DellaBitta approved these changes Mar 24, 2025

View reviewed changes

dlarocque changed the title ~~Vertex AI Integration Tests~~ test: add VertexAI integration tests Mar 28, 2025

dlarocque added 2 commits May 16, 2025 09:25

Merge branch 'main' into dl/vertex-integration

11a8b05

Add test configs to run against each test

9f908d2

dlarocque added 3 commits May 16, 2025 11:33

Add test configs to run against each test

abe830c

count tokens tests

472c452

Format and fix lint issues

681b4bd

dlarocque added 8 commits May 20, 2025 14:28

Merge branch 'main' into dl/vertex-integration

62dab2b

Merge branch 'main' into dl/vertex-integration

8f65c98

count tokens tests against both backends

c769317

count tokens test cleanup

d78fc29

remaining integration tests

676e90d

revert changes to gitignore

ddfd7f5

Merge branch 'main' into dl/vertex-integration

46e910d

Merge branch 'dl/vertex-integration' of https://github.com/firebase/f…

3bfcfa7

…irebase-js-sdk into dl/vertex-integration

dlarocque changed the title ~~test: add VertexAI integration tests~~ test(ai): add integration tests May 22, 2025

dlarocque added 6 commits May 22, 2025 13:52

use ci config file

28f843c

import from minified @firebase/ai in integration tests

f851be4

exclude integration from rollup

f327b4b

revert change to import from @firebase/ai

604a419

cleanup

9b97310

format

63f7215

dlarocque merged commit d590889 into main Jun 5, 2025
38 checks passed

dlarocque deleted the dl/vertex-integration branch June 5, 2025 16:50

test(ai): add integration tests #8853

test(ai): add integration tests #8853

Uh oh!

Conversation

dlarocque commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

google-oss-bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size Report 1

Affected Products

@firebase/firestore

bundle

firebase

Test Logs

Uh oh!

Uh oh!

Uh oh!

hsubox76 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

dlarocque Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

google-oss-bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size Analysis Report 1

Affected Products

@firebase/firestore

addDoc

deleteDoc

disableNetwork

enableIndexedDbPersistence

enableMultiTabIndexedDbPersistence

enableNetwork

executeWrite

getAggregateFromServer

getCountFromServer

getDoc

getDocFromServer

getDocs

getDocsFromServer

loadBundle

memoryLocalCache

onSnapshot

onSnapshotsInSync

persistentLocalCache

persistentMultipleTabManager

persistentSingleTabManager

runTransaction

setDoc

updateDoc

waitForPendingWrites

writeBatch

Test Logs

Uh oh!

github-actions bot commented May 16, 2025

Vertex AI Mock Responses Check ⚠️

Uh oh!

Uh oh!

Uh oh!

dlarocque commented Mar 19, 2025 •

edited

Loading

changeset-bot bot commented Mar 19, 2025 •

edited

Loading

google-oss-bot commented Mar 19, 2025 •

edited

Loading

Size Report ¹

`@firebase/firestore`

`bundle`

`firebase`

google-oss-bot commented Mar 19, 2025 •

edited

Loading

Size Analysis Report ¹

`@firebase/firestore`

`addDoc`

`deleteDoc`

`disableNetwork`

`enableIndexedDbPersistence`

`enableMultiTabIndexedDbPersistence`

`enableNetwork`

`executeWrite`

`getAggregateFromServer`

`getCountFromServer`

`getDoc`

`getDocFromServer`

`getDocs`

`getDocsFromServer`

`loadBundle`

`memoryLocalCache`

`onSnapshot`

`onSnapshotsInSync`

`persistentLocalCache`

`persistentMultipleTabManager`

`persistentSingleTabManager`

`runTransaction`

`setDoc`

`updateDoc`

`waitForPendingWrites`

`writeBatch`