From 8cb53456a4fb917116c89b42168e56aab0f2117e Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Thu, 21 Sep 2023 12:37:09 -0700 Subject: [PATCH 1/7] DIFN GA --- .../functions/destination-functions.md | 2 +- src/connections/functions/index.md | 3 - src/connections/functions/insert-functions.md | 172 ++++++++++++++++-- 3 files changed, 161 insertions(+), 16 deletions(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 03a5b4e866..95c8088a44 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -140,7 +140,7 @@ If your function fails, you can check the error details and logs in the **Output Batch handlers are an extension of destination functions. When you define an `onBatch` handler alongside the handler functions for single events (for example: `onTrack` or `onIdentity`), you're telling Segment that the destination function can accept and handle batches of events. > info "" -> Batching is available for destination functions only. +> Batching is available for destination and destination insert functions only. ### When to use batching diff --git a/src/connections/functions/index.md b/src/connections/functions/index.md index 2cb759185d..21505848a4 100644 --- a/src/connections/functions/index.md +++ b/src/connections/functions/index.md @@ -35,9 +35,6 @@ Learn more about [destination functions](/docs/connections/functions/destination #### Destination insert functions Destination insert functions help you enrich your data with code before you send it to downstream destinations. -> info "Destination Insert Functions in Public Beta" -> Destination Insert Functions is in Public Beta, and Segment is actively working on this feature. Some functionality may change before it becomes generally available. [Contact Segment](https://segment.com/help/contact/){:target="_blank"} with any feedback or questions. - Use cases: - Implement custom logic and enrich data with third party sources - Transform outgoing data with advanced filtration and computation diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index 67b13cd5c3..3cdbca8586 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -14,9 +14,6 @@ Use Destination Insert Functions to enrich, transform, or filter your data befor **Customize filtration for your destinations**: Create custom logic with nested if-else statements, regex, custom business rules, and more to filter event data. -> info "Destination Insert Functions in Public Beta" -> Destination Insert Functions is in Public Beta, and Segment is actively working on this feature. Some functionality may change before it becomes generally available. [Contact Segment](https://segment.com/help/contact/){:target="_blank"} with any feedback or questions. - ## Create destination insert functions @@ -237,21 +234,172 @@ To enable your insert function: To prevent your insert function from processing data, toggle Enable Function off. -{% comment %} -## Batching the insert function - -Insert functions support batching with the `onBatch` handler. +## Batching the destination insert function Batch handlers are an extension of insert functions. When you define an `onBatch` handler alongside the handler functions for single events (for example, `onTrack` or `onIdentity`), you're telling Segment that the insert function can accept and handle batches of events. -Note the following limitations for batching with insert functions: -- The batch request and response size is limited to 6mb. -- Max count begins with 100 and goes up to 1,000. +> info "" +> Batching is available for insert and destination functions only. + +### When to use batching + +Consider creating a batch handler if: + +- **You have a high-throughput function and want to reduce cost.** When you define a batch handler, Segment invokes the function once per *batch*, rather than once per event. As long as the function’s execution time isn’t adversely affected, the reduction in invocations should lead to a reduction in cost. + +- **Your destination supports batching**. When your downstream destination supports sending data downstream in batches you can define a batch handler to avoid throttling. Batching for functions is independent of batch size supported by the destination. Segment automatically handles batch formation for destinations internally. > info "" -> Batching is available for insert and destination functions only. Read more about batching [in the Destination Functions docs](/docs/connections/functions/destination-functions/#batching-the-destination-function). +> If a batched function receives too low a volume of events (under one event per second) to be worth batching, Segment may not invoke the batch handler. + +### Define the batch handler + +Segment collects the events over a short period of time and combines them into a batch. The system flushes them when the batch reaches a certain number of events, or when the batch has been waiting for a specified wait time. + +To create a batch handler, define an `onBatch` function within your destination insert function. You can also use the "Default Batch" template found in the Functions editor to get started quickly. + +```js +async function onBatch(events, settings){ + // handle the batch of events +} +``` + +> info "" +> The `onBatch` handler is an optional extension. Destination insert functions must still contain single event handlers as a fallback, in cases where Segment does not receive enough events to execute the batch. + +The handler function receives an array of events. The events can be of any supported type and a single batch may contain more than one event type. Handler functions can also receive function settings. Here is an example of what a batch can look like: + +```json +[ + { + "type": "identify", + "userId": "019mr8mf4r", + "traits": { + "email": "jake@yahoo.com", + "name": "Jake Peterson", + "age": 26 + } + }, + { + "type": "track", + "userId": "019mr8mf4r", + "event": "Song Played", + "properties": { + "name": "Fallin for You", + "artist": "Dierks Bentley" + } + }, + { + "type": "track", + "userId": "971mj8mk7p", + "event": "Song Played", + "properties": { + "name": "Get Right", + "artist": "Jennifer Lopez" + } + } +] +``` + +### Configure the event types within a batch + +Segment batches together any event _of any type_ that it sees over a short period of time to increase batching efficiency and give you the flexibility to decide how batches are created. If you want to split batches by event type, you can implement this in your functions code by writing a handler. + +```js +async function onBatch(events, settings) { + // group events by type + const eventsByType = {} + for (const event of events) { + if (!(event.type in eventsByType)) { + eventsByType[event.type] = [] + } + eventsByType[event.type].push(event) + } + + // concurrently process sub-batches of a specific event type + const promises = Object.entries(eventsByType).map(([type, events]) => { + switch (type) { + case 'track': + return onTrackBatch(events, settings) + case 'identify': + return onIdentifyBatch(events, settings) + // ...handle other event types here... + } + }) + return Promise.all(promises) +} + +async function onTrackBatch(events, settings) { + // handle a batch of track events +} + +async function onIdentifyBatch(events, settings) { + // handle a batch of identify events +} +``` + +### Configure your batch parameters + +By default, Functions waits up to 10 seconds to form a batch of 20 events. You can increase the number of events included in each batch (up to 400 events per batch) by contacting [Segment support](https://segment.com/help/contact/){:target="_blank"}. Segment recommends users who wish to include fewer than 20 events per batch use destination functions without the `onBatch` handler. + +### Test the batch handler + +The [Functions editing environment](/docs/connections/functions/environment/) supports testing batch handlers. + +To test the batch handler: +1. In the right panel of the Functions editor, click **customize the event yourself** to enter Manual Mode. +2. Add events as a JSON array, with one event per element. +3. Click **Run** to preview the batch handler with the specified events. + +> note "" +> The Sample Event option tests single events only. You must use Manual Mode to add more than one event so you can test batch handlers. + +The editor displays logs and request traces from the batch handler. + +The [Public API](/docs/api/public-api) Functions/Preview endpoint also supports testing batch handlers. The payload must be a batch of events as a JSON array. + + +### Handling batching errors + +Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment attempts to retry the batch in the case of Timeout or Retry errors. For all other error types, Segment discards the batch. It's also possible to report a partial failure by returning status of each event in the batch. Segment retries only the failed events in a batch until those events are successful or until they result in a permanent error. + +```json +[ + { + "status": 200 + }, + { + "status": 400, + "errormessage": "Bad Request" + }, + { + "status": 200 + }, + { + "status": 500, + "errormessage": "Error processing request" + }, + { + "status": 500, + "errormessage": "Error processing request" + }, + { + "status": 200 + }, +] +``` + +After receiving the response from the `onBatch` handler, Segment only retries **event_4** and **event_5**. + +| Error Type | Result | +| ---------------------- | ------- | +| Bad Request | Discard | +| Invalid Settings | Discard | +| Message Rejected | Discard | +| RetryError | Retry | +| Timeout | Retry | +| Unsupported Event Type | Discard | -{% endcomment %} {% comment %} From efb8ea9016c7c59028a9ef9c30f3346dea271854 Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Thu, 21 Sep 2023 12:45:19 -0700 Subject: [PATCH 2/7] edits --- src/connections/functions/destination-functions.md | 4 ++-- src/connections/functions/insert-functions.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 95c8088a44..42e0beaee7 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -269,7 +269,7 @@ To test the batch handler: 2. Add events as a JSON array, with one event per element. 3. Click **Run** to preview the batch handler with the specified events. -> note "" +> info "" > The Sample Event option tests single events only. You must use Manual Mode to add more than one event so you can test batch handlers. The editor displays logs and request traces from the batch handler. @@ -306,7 +306,7 @@ Standard [function error types](/docs/connections/functions/destination-function ] ``` -After receiving the response from the `onBatch` handler, Segment only retries **event_4** and **event_5**. +For example, after receiving the responses above from the `onBatch` handler, Segment only retries **event_4** and **event_5**. | Error Type | Result | | ---------------------- | ------- | diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index 3cdbca8586..3b0ef7e193 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -239,7 +239,7 @@ To prevent your insert function from processing data, toggle Enable Function off Batch handlers are an extension of insert functions. When you define an `onBatch` handler alongside the handler functions for single events (for example, `onTrack` or `onIdentity`), you're telling Segment that the insert function can accept and handle batches of events. > info "" -> Batching is available for insert and destination functions only. +> Batching is available for destination and destination insert functions only. ### When to use batching @@ -265,7 +265,7 @@ async function onBatch(events, settings){ ``` > info "" -> The `onBatch` handler is an optional extension. Destination insert functions must still contain single event handlers as a fallback, in cases where Segment does not receive enough events to execute the batch. +> The `onBatch` handler is an optional extension. Destination insert functions must still contain single event handlers as a fallback, in cases where Segment doesn't receive enough events to execute the batch. The handler function receives an array of events. The events can be of any supported type and a single batch may contain more than one event type. Handler functions can also receive function settings. Here is an example of what a batch can look like: @@ -340,7 +340,7 @@ async function onIdentifyBatch(events, settings) { ### Configure your batch parameters -By default, Functions waits up to 10 seconds to form a batch of 20 events. You can increase the number of events included in each batch (up to 400 events per batch) by contacting [Segment support](https://segment.com/help/contact/){:target="_blank"}. Segment recommends users who wish to include fewer than 20 events per batch use destination functions without the `onBatch` handler. +By default, Functions waits up to 10 seconds to form a batch of 20 events. You can increase the number of events included in each batch (up to 400 events per batch) by contacting [Segment support](https://segment.com/help/contact/){:target="_blank"}. Segment recommends users who wish to include fewer than 20 events per batch use destination insert functions without the `onBatch` handler. ### Test the batch handler @@ -351,7 +351,7 @@ To test the batch handler: 2. Add events as a JSON array, with one event per element. 3. Click **Run** to preview the batch handler with the specified events. -> note "" +> info "" > The Sample Event option tests single events only. You must use Manual Mode to add more than one event so you can test batch handlers. The editor displays logs and request traces from the batch handler. @@ -389,7 +389,7 @@ Standard [function error types](/docs/connections/functions/destination-function ] ``` -After receiving the response from the `onBatch` handler, Segment only retries **event_4** and **event_5**. +For example, after receiving the responses above from the `onBatch` handler, Segment only retries **event_4** and **event_5**. | Error Type | Result | | ---------------------- | ------- | From 4308fd105e8118388b71c25f22167d9abf15fc9d Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Thu, 21 Sep 2023 12:46:12 -0700 Subject: [PATCH 3/7] [netlify-build] --- src/connections/functions/insert-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index 375e9060bf..d1119f78f0 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -339,7 +339,7 @@ async function onIdentifyBatch(events, settings) { ``` ### Configure your batch parameters - + By default, Functions waits up to 10 seconds to form a batch of 20 events. You can increase the number of events included in each batch (up to 400 events per batch) by contacting [Segment support](https://segment.com/help/contact/){:target="_blank"}. Segment recommends users who wish to include fewer than 20 events per batch use destination insert functions without the `onBatch` handler. ### Test the batch handler From f71505f7634cb324a9a0fc8009140aa205f6ca7c Mon Sep 17 00:00:00 2001 From: rchinn-segment <93161299+rchinn-segment@users.noreply.github.com> Date: Thu, 21 Sep 2023 17:44:37 -0700 Subject: [PATCH 4/7] Commit suggestion from code review Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- src/connections/functions/insert-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index d1119f78f0..b17f1b0819 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -247,7 +247,7 @@ Consider creating a batch handler if: - **You have a high-throughput function and want to reduce cost.** When you define a batch handler, Segment invokes the function once per *batch*, rather than once per event. As long as the function’s execution time isn’t adversely affected, the reduction in invocations should lead to a reduction in cost. -- **Your destination supports batching**. When your downstream destination supports sending data downstream in batches you can define a batch handler to avoid throttling. Batching for functions is independent of batch size supported by the destination. Segment automatically handles batch formation for destinations internally. +- **Your destination supports batching**. When your downstream destination supports sending data downstream in batches you can define a batch handler to avoid throttling. Batching for functions is independent of batch size supported by the destination. Segment automatically handles batch formation for destinations. > info "" > If a batched function receives too low a volume of events (under one event per second) to be worth batching, Segment may not invoke the batch handler. From bc22aa68a8069eb3382ed89ab93856f87d5114e4 Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Fri, 22 Sep 2023 08:59:19 -0700 Subject: [PATCH 5/7] Updates from PM review [netlify-build] --- src/_includes/content/functions/logs.md | 2 +- src/connections/functions/insert-functions.md | 11 ++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/src/_includes/content/functions/logs.md b/src/_includes/content/functions/logs.md index 210e867185..9ba89f35df 100644 --- a/src/_includes/content/functions/logs.md +++ b/src/_includes/content/functions/logs.md @@ -1,6 +1,6 @@ If your function throws an error, execution halts immediately. Segment captures the event, any outgoing requests/responses, any logs the function might have printed, as well as the error itself. -Segment then displays the captured error information in the [Event Delivery](/docs/connections/event-delivery/) page for your function. You can use this information to find and fix unexpected errors. +Segment then displays the captured error information in the [Event Delivery](/docs/connections/event-delivery/) page for your destination. You can use this information to find and fix unexpected errors. You can throw [an error or a custom error](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error){:target="_blank"} and you can also add helpful context in logs using the [`console` API](https://developer.mozilla.org/en-US/docs/Web/API/console){:target="_blank"}. For example: diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index b17f1b0819..0855d81bec 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -261,6 +261,7 @@ To create a batch handler, define an `onBatch` function within your destination ```js async function onBatch(events, settings){ // handle the batch of events + return events } ``` @@ -326,15 +327,23 @@ async function onBatch(events, settings) { // ...handle other event types here... } }) - return Promise.all(promises) + try { + const results = await Promise.all(promises); + const batchResult = [].concat(...results); // Combine arrays into a single array + return batchResult; + } catch (error) { + throw new RetryError(error.message); + } } async function onTrackBatch(events, settings) { // handle a batch of track events + return events } async function onIdentifyBatch(events, settings) { // handle a batch of identify events + return events } ``` From f26f2449e0781bd97f8b447ccaad3dabbd9ff3c5 Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Fri, 22 Sep 2023 10:12:48 -0700 Subject: [PATCH 6/7] Add pr #5404 update --- src/connections/functions/insert-functions.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index 0855d81bec..7caf411b5a 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -478,11 +478,6 @@ No, destination insert functions are currently available as cloud-mode destinati No, an insert function can only be connected to one destination. -##### How do I publish a destination to the public Segment catalog? - -If you are a partner, looking to publish your destination and distribute your app through Segment catalog, visit the [Developer Center](https://segment.com/partners/developer-center/){:target="_blank"} and check out the Segment [partner docs](/docs/partners/). - - {% comment %} ## Using Segment's Public API From 32e8abcfce90bc3f2a7bcf3178d51d0aa6e3ea0d Mon Sep 17 00:00:00 2001 From: rchinn-segment Date: Mon, 25 Sep 2023 09:44:59 -0700 Subject: [PATCH 7/7] remove beta metadata --- src/connections/functions/insert-functions.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/connections/functions/insert-functions.md b/src/connections/functions/insert-functions.md index 7caf411b5a..a02d1679fa 100644 --- a/src/connections/functions/insert-functions.md +++ b/src/connections/functions/insert-functions.md @@ -1,6 +1,5 @@ --- title: Destination Insert Functions -beta: true ---