Skip to content

[Evals] Support for automated dataset creation #2812

Open
@ssbushi

Description

@ssbushi

Is your feature request related to a problem? Please describe.

Dataset creation / data gathering is a hurdle for most new developers for getting started with evaluations.

Describe the solution you'd like
We can help them out with a dataset-creation flow (powered by LLMs). This feature requires a prompt engineered to consider a target action's schema, its description and intent, and returns a valid JSON format, compatible dataset that we can use to create a dataset.

This can even leverage some recent traces in the prompt as examples for generation.

Considerations to be made here:

  • Where the prompt and associated logic is located
  • Which model to use? How to present the user with choices / decisions to be made (in the UI).
  • Any intermediate validation/quick-look step before creating a dataset?

Describe alternatives you've considered
No other alternatives. The current alternative is to create dataset from traces.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions