Open
Description
Is your feature request related to a problem? Please describe.
Dataset creation / data gathering is a hurdle for most new developers for getting started with evaluations.
Describe the solution you'd like
We can help them out with a dataset-creation flow (powered by LLMs). This feature requires a prompt engineered to consider a target action's schema, its description and intent, and returns a valid JSON format, compatible dataset that we can use to create a dataset.
This can even leverage some recent traces in the prompt as examples for generation.
Considerations to be made here:
- Where the prompt and associated logic is located
- Which model to use? How to present the user with choices / decisions to be made (in the UI).
- Any intermediate validation/quick-look step before creating a dataset?
Describe alternatives you've considered
No other alternatives. The current alternative is to create dataset from traces.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
No status