Share your open ML datasets on Hugging Face Hub!
β’
20
Quentin Lhoest PRO
lhoestq
AI & ML interests
Maintainer of π€Datasets: NLP, Multimodal data processing and sharing
Recent Activity
liked
a dataset
about 5 hours ago
Isotonic/agentinstruct-1Mv1-combined
liked
a dataset
about 5 hours ago
ylecun/mnist
updated
a dataset
about 8 hours ago
infinite-dataset-hub/BreastDiagnosisProgression
Articles
Organizations
Posts
2
Post
3941
Hey ! I'm working on a 100% synthetic Dataset Hub here (you can search for any kind of datasets an the app invents them). The link is here:
infinite-dataset-hub/infinite-dataset-hub
Question for the Community:
Which models should I use to generate images and audio samples for those datasets ? π€
Question for the Community:
Which models should I use to generate images and audio samples for those datasets ? π€
Post
3001
β¨ Easy Synthetic Dataset File Generation using LLM DataGen ! Link: https://huggingface.co/spaces/lhoestq/LLM_DataGen
features + how it works:
βοΈ Generate the dataset content you want just by entering a file name
π‘ Optionally specify the column names you need
π¨ The dataset is streamed and generated on-the-fly in JSON Lines format
β Generation is constrained to always output valid JSON
How does this work ?
1/ Enter a file name
2/ The model generates column names for such a file. Using structured generation, it can generate 2 to 5 column names using lower case characters and underscores. I use a prompt that asks to generate column names for a realistic dataset and low temperature.
3/ The columns are used to update the Finite State Machine for the dataset content structured generation, so that it is used to generate JSON objects using those columns
4/ The model generates JSON objects using structured generation again, using the updated Finite State Machine. I use a prompt that asks for realistic data and a temperature of 1.
> Why update a Finite State Machine instead of re-creating one ?
Creating one can take up to 30sec, while updating one takes 0.1s (though it requires to manipulate a graph which is not easy to implement)
> Batched generation is faster, why not use it ?
Generate in batches is faster but tends to generate duplicates for this demo.
Further work can be to provide different prompts (one per sequence in the batch) to end up with a different distribution of sequences in each batch. Or implement a custom sampler that would forbid generating the same data in sequences of the same batch.
> How does structured generation work ?
I used the
Let me know what you think ! And feel free to duplicate/modify it to try other models/prompts or sampling methods :)
features + how it works:
βοΈ Generate the dataset content you want just by entering a file name
π‘ Optionally specify the column names you need
π¨ The dataset is streamed and generated on-the-fly in JSON Lines format
β Generation is constrained to always output valid JSON
How does this work ?
1/ Enter a file name
2/ The model generates column names for such a file. Using structured generation, it can generate 2 to 5 column names using lower case characters and underscores. I use a prompt that asks to generate column names for a realistic dataset and low temperature.
3/ The columns are used to update the Finite State Machine for the dataset content structured generation, so that it is used to generate JSON objects using those columns
4/ The model generates JSON objects using structured generation again, using the updated Finite State Machine. I use a prompt that asks for realistic data and a temperature of 1.
> Why update a Finite State Machine instead of re-creating one ?
Creating one can take up to 30sec, while updating one takes 0.1s (though it requires to manipulate a graph which is not easy to implement)
> Batched generation is faster, why not use it ?
Generate in batches is faster but tends to generate duplicates for this demo.
Further work can be to provide different prompts (one per sequence in the batch) to end up with a different distribution of sequences in each batch. Or implement a custom sampler that would forbid generating the same data in sequences of the same batch.
> How does structured generation work ?
I used the
outlines library with transformers to to define a JSON schema that the generation has to follow. It uses a Finite State Machine with token_id as transitions.Let me know what you think ! And feel free to duplicate/modify it to try other models/prompts or sampling methods :)
Collections
1
spaces
13
models
1
datasets
43
lhoestq/tmp2
Viewer
β’
Updated
β’
153k
β’
10
lhoestq/NewsEventsPredict-parquet
Viewer
β’
Updated
β’
100
β’
4
lhoestq/tmp_vid3_zip
Viewer
β’
Updated
β’
2
β’
8
lhoestq/tmp_vid2
Viewer
β’
Updated
β’
100
β’
423
lhoestq/tmp_vid
Updated
β’
88
lhoestq/test_video
Viewer
β’
Updated
β’
1
β’
36
lhoestq/tmp_duration
Viewer
β’
Updated
β’
3
β’
40
lhoestq/tmp
Viewer
β’
Updated
β’
2
β’
14
lhoestq/tmp-notebooks
Updated
β’
72
lhoestq/tmp-Infinity-Instruct-3M-dask
Viewer
β’
Updated
β’
1M
β’
89