tutorial · EvolvingLMMs-Lab/lmms-eval@ce6f889 (original) (raw)
``
1
`+
{
`
``
2
`+
"cells": [
`
``
3
`+
{
`
``
4
`+
"cell_type": "markdown",
`
``
5
`+
"metadata": {},
`
``
6
`+
"source": [
`
``
7
`+
"# Make Image dataset on Hugging Face Datasets\n",
`
``
8
`+
"\n",
`
``
9
`+
"
\n",
`
``
10
`+
"\n",
`
``
11
`+
"This notebook will guide you to make correct format of Huggingface dataset, in proper parquet format and visualizable in Huggingface dataset hub.\n",
`
``
12
`+
"\n",
`
``
13
`` +
"We will take the example of the dataset pufanyi/VQAv2_Example and convert it to the proper format."
``
``
14
`+
]
`
``
15
`+
},
`
``
16
`+
{
`
``
17
`+
"cell_type": "markdown",
`
``
18
`+
"metadata": {
`
``
19
`+
"vscode": {
`
``
20
`+
"languageId": "plaintext"
`
``
21
`+
}
`
``
22
`+
},
`
``
23
`+
"source": [
`
``
24
`+
"## Download Dataset\n",
`
``
25
`+
"\n",
`
``
26
`` +
"We have uploaded the zip file of the dataset to Hugging Face for download. This dataset is a subset of the VQAv2 dataset, with 101010 entries each from the val, test, and test-dev splits, for easier downloading."
``
``
27
`+
]
`
``
28
`+
},
`
``
29
`+
{
`
``
30
`+
"cell_type": "code",
`
``
31
`+
"execution_count": 45,
`
``
32
`+
"metadata": {
`
``
33
`+
"vscode": {
`
``
34
`+
"languageId": "bat"
`
``
35
`+
}
`
``
36
`+
},
`
``
37
`+
"outputs": [
`
``
38
`+
{
`
``
39
`+
"name": "stdout",
`
``
40
`+
"output_type": "stream",
`
``
41
`+
"text": [
`
``
42
`+
"--2024-06-19 14:09:51-- https://huggingface.co/datasets/pufanyi/VQAv2_TOY/resolve/main/source_data/sample_data.zip\n",
`
``
43
`+
"Resolving huggingface.co (huggingface.co)... 13.33.30.114, 13.33.30.49, 13.33.30.76, ...\n",
`
``
44
`+
"Connecting to huggingface.co (huggingface.co)|13.33.30.114|:443... connected.\n",
`
``
45
`+
"HTTP request sent, awaiting response... 302 Found\n",
`
``
46
`+
`
``
47
`+
`
``
48
`+
"Resolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 3.165.102.80, 3.165.102.25, 3.165.102.95, ...\n",
`
``
49
`+
"Connecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|3.165.102.80|:443... connected.\n",
`
``
50
`+
"HTTP request sent, awaiting response... 200 OK\n",
`
``
51
`+
"Length: 2678607 (2.6M) [application/zip]\n",
`
``
52
`+
"Saving to: ‘data/sample_data.zip’\n",
`
``
53
`+
"\n",
`
``
54
`+
"sample_data.zip 100%[===================>] 2.55M 7.46MB/s in 0.3s \n",
`
``
55
`+
"\n",
`
``
56
`+
"2024-06-19 14:09:52 (7.46 MB/s) - ‘data/sample_data.zip’ saved [2678607/2678607]\n",
`
``
57
`+
"\n"
`
``
58
`+
]
`
``
59
`+
}
`
``
60
`+
],
`
``
61
`+
"source": [
`
``
62
`+
"!wget https://huggingface.co/datasets/pufanyi/VQAv2_TOY/resolve/main/source_data/sample_data.zip -P data\n",
`
``
63
`+
"!unzip data/sample_data.zip -d data"
`
``
64
`+
]
`
``
65
`+
},
`
``
66
`+
{
`
``
67
`+
"cell_type": "markdown",
`
``
68
`+
"metadata": {},
`
``
69
`+
"source": [
`
``
70
`` +
"We can open data/questions to take a view of the dataset organization. We found that the toy-VQAv2 dataset is organized as follows:\n",
``
``
71
`+
"\n",
`
``
72
"```json\n",
``
73
`+
"{\n",
`
``
74
`+
" "info": { /* some infomation */ },\n",
`
``
75
`+
" "task_type": "TASK_TYPE", "data_type": "mscoco",\n",
`
``
76
`+
" "license": { /* some license */ },\n",
`
``
77
`+
" "questions": [\n",
`
``
78
`+
" {\n",
`
``
79
`+
" "image_id": 262144, // integer id of the image\n",
`
``
80
`+
" "question": "Is the ball flying towards the batter?",\n",
`
``
81
`+
" "question_id": 262144000\n",
`
``
82
`+
" },\n",
`
``
83
`+
" /* ... */\n",
`
``
84
`+
" ]\n",
`
``
85
`+
"}\n",
`
``
86
"```"
``
87
`+
]
`
``
88
`+
},
`
``
89
`+
{
`
``
90
`+
"cell_type": "markdown",
`
``
91
`+
"metadata": {},
`
``
92
`+
"source": [
`
``
93
`+
"## Define Dataset Features _(Optional*)_\n",
`
``
94
`+
"\n",
`
``
95
`+
"You can define the features of the dataset. For more details, please refer to the official documentation.\n",
`
``
96
`+
"\n",
`
``
97
`+
"* Note that if the dataset features are consistent and all entries in your dataset table are non-null for all splits of data, you can skip this step."
`
``
98
`+
]
`
``
99
`+
},
`
``
100
`+
{
`
``
101
`+
"cell_type": "code",
`
``
102
`+
"execution_count": null,
`
``
103
`+
"metadata": {},
`
``
104
`+
"outputs": [],
`
``
105
`+
"source": [
`
``
106
`+
"import datasets\n",
`
``
107
`+
"\n",
`
``
108
`+
"features = datasets.Features(\n",
`
``
109
`+
" {\n",
`
``
110
`+
" "question": datasets.Value("string"),\n",
`
``
111
`+
" "question_id": datasets.Value("int64"),\n",
`
``
112
`+
" "image_id": datasets.Value("string"),\n",
`
``
113
`+
" "image": datasets.Image(),\n",
`
``
114
`+
" "answers": datasets.Sequence(datasets.Sequence(feature={"answer": datasets.Value("string"), "answer_confidence": datasets.Value("string"), "answer_id": datasets.Value("int64")})),\n",
`
``
115
`+
" "answer_type": datasets.Value("string"),\n",
`
``
116
`+
" "multiple_choice_answer": datasets.Value("string"),\n",
`
``
117
`+
" "question_type": datasets.Value("string"),\n",
`
``
118
`+
" }\n",
`
``
119
`+
")"
`
``
120
`+
]
`
``
121
`+
},
`
``
122
`+
{
`
``
123
`+
"cell_type": "markdown",
`
``
124
`+
"metadata": {},
`
``
125
`+
"source": [
`
``
126
`+
"## Define Data Generator\n",
`
``
127
`+
"\n",
`
``
128
`` +
"We use datasets.Dataset.from_generator to create the dataset.\n",
``
``
129
`+
"\n",
`
``
130
`` +
"The generator function should yield dictionaries with the keys corresponding to the dataset features. This can save memory when loading large datasets.\n",
``
``
131
`+
"\n",
`
``
132
`` +
"For the image data, we can convert the image to PIL.Image object.\n",
``
``
133
`+
"\n",
`
``
134
`` +
"Note that if some columns are missing in some splits of the dataset (for example, the answer column is usually missing in the test split), we need to set these columns to null to ensure that all splits have the same features."
``
``
135
`+
]
`
``
136
`+
},
`
``
137
`+
{
`
``
138
`+
"cell_type": "code",
`
``
139
`+
"execution_count": null,
`
``
140
`+
"metadata": {},
`
``
141
`+
"outputs": [],
`
``
142
`+
"source": [
`
``
143
`+
"import os\n",
`
``
144
`+
"import json\n",
`
``
145
`+
"from PIL import Image\n",
`
``
146
`+
"\n",
`
``
147
`+
"KEYS = ["question", "question_id", "image_id", "answers", "answer_type", "multiple_choice_answer", "question_type"]\n",
`
``
148
`+
"\n",
`
``
149
`+
"def generator(qa_file, image_folder, image_prefix):\n",
`
``
150
`+
" # Open and load the question-answer file\n",
`
``
151
`+
" with open(qa_file, "r") as f:\n",
`
``
152
`+
" data = json.load(f)\n",
`
``
153
`+
" qa = data["questions"]\n",
`
``
154
`+
"\n",
`
``
155
`+
" for q in qa:\n",
`
``
156
`+
" # Get the image id\n",
`
``
157
`+
" image_id = q["image_id"]\n",
`
``
158
`+
" # Construct the image path\n",
`
``
159
`+
" image_path = os.path.join(image_folder, f"{image_prefix}_{image_id:012}.jpg")\n",
`
``
160
`+
" # Open the image and add it to the question-answer dictionary\n",
`
``
161
`+
" q["image"] = Image.open(image_path)\n",
`
``
162
`+
" # Check if all keys are present in the question-answer dictionary, if not add them with None value\n",
`
``
163
`+
" for key in KEYS:\n",
`
``
164
`+
" if key not in q:\n",
`
``
165
`+
" q[key] = None\n",
`
``
166
`+
" # Yield the question-answer dictionary\n",
`
``
167
`+
" yield q"
`
``
168
`+
]
`
``
169
`+
},
`
``
170
`+
{
`
``
171
`+
"cell_type": "markdown",
`
``
172
`+
"metadata": {},
`
``
173
`+
"source": [
`
``
174
`+
"## Generate Dataset\n",
`
``
175
`+
"\n",
`
``
176
`+
"We generate the dataset using the generator function.\n",
`
``
177
`+
"\n",
`
``
178
`` +
"Note that if you skip the step of defining dataset features, there is no need to pass the features argument. The dataset infer the features from the dataset automatically."
``
``
179
`+
]
`
``
180
`+
},
`
``
181
`+
{
`
``
182
`+
"cell_type": "code",
`
``
183
`+
"execution_count": null,
`
``
184
`+
"metadata": {},
`
``
185
`+
"outputs": [],
`
``
186
`+
"source": [
`
``
187
`+
"NUM_PROC = 32 # number of processes to use for multiprocessing, set to 1 for no multiprocessing\n",
`
``
188
`+
"\n",
`
``
189
`+
"data_val = datasets.Dataset.from_generator(\n",
`
``
190
`+
" generator,\n",
`
``
191
`+
" gen_kwargs={\n",
`
``
192
`+
" "qa_file": "data/questions/v2_OpenEnded_mscoco_val2014_questions.json",\n",
`
``
193
`+
" "image_folder": "data/images/val2014",\n",
`
``
194
`+
" "image_prefix": "COCO_val2014",\n",
`
``
195
`+
" },\n",
`
``
196
`+
" features=features,\n",
`
``
197
`+
" num_proc=NUM_PROC,\n",
`
``
198
`+
")\n",
`
``
199
`+
"\n",
`
``
200
`+
"data_test = datasets.Dataset.from_generator(\n",
`
``
201
`+
" generator,\n",
`
``
202
`+
" gen_kwargs={\n",
`
``
203
`+
" "qa_file": "data/questions/v2_OpenEnded_mscoco_test2015_questions.json",\n",
`
``
204
`+
" "image_folder": "data/images/test2015",\n",
`
``
205
`+
" "image_prefix": "COCO_test2015",\n",
`
``
206
`+
" },\n",
`
``
207
`+
" features=features,\n",
`
``
208
`+
" num_proc=NUM_PROC,\n",
`
``
209
`+
")\n",
`
``
210
`+
"\n",
`
``
211
`+
"data_test_dev = datasets.Dataset.from_generator(\n",
`
``
212
`+
" generator,\n",
`
``
213
`+
" gen_kwargs={\n",
`
``
214
`+
" "qa_file": "data/questions/v2_OpenEnded_mscoco_test-dev2015_questions.json",\n",
`
``
215
`+
" "image_folder": "data/images/test2015",\n",
`
``
216
`+
" "image_prefix": "COCO_test2015",\n",
`
``
217
`+
" },\n",
`
``
218
`+
" features=features,\n",
`
``
219
`+
" num_proc=NUM_PROC,\n",
`
``
220
`+
")"
`
``
221
`+
]
`
``
222
`+
},
`
``
223
`+
{
`
``
224
`+
"cell_type": "markdown",
`
``
225
`+
"metadata": {},
`
``
226
`+
"source": [
`
``
227
`+
"## Dataset Upload\n",
`
``
228
`+
"\n",
`
``
229
`+
"Finally, we group the dataset with different splits and upload it to the Huggingface dataset hub."
`
``
230
`+
]
`
``
231
`+
},
`
``
232
`+
{
`
``
233
`+
"cell_type": "code",
`
``
234
`+
"execution_count": null,
`
``
235
`+
"metadata": {},
`
``
236
`+
"outputs": [],
`
``
237
`+
"source": [
`
``
238
`+
"data = datasets.DatasetDict({"val": data_val, "test": data_test, "test_dev": data_test_dev})"
`
``
239
`+
]
`
``
240
`+
},
`
``
241
`+
{
`
``
242
`+
"cell_type": "code",
`
``
243
`+
"execution_count": null,
`
``
244
`+
"metadata": {},
`
``
245
`+
"outputs": [],
`
``
246
`+
"source": [
`
``
247
`+
"data.push_to_hub("pufanyi/VQAv2")"
`
``
248
`+
]
`
``
249
`+
},
`
``
250
`+
{
`
``
251
`+
"cell_type": "code",
`
``
252
`+
"execution_count": 44,
`
``
253
`+
"metadata": {},
`
``
254
`+
"outputs": [
`
``
255
`+
{
`
``
256
`+
"data": {
`
``
257
`+
"text/plain": [
`
``
258
`+
"CommitInfo(commit_url='https://huggingface.co/datasets/pufanyi/VQAv2_TOY/commit/b057eff450520a6e3fc7e6be88c3a172c4b5d99b', commit_message='Upload source_data/sample_data.zip with huggingface_hub', commit_description='', oid='b057eff450520a6e3fc7e6be88c3a172c4b5d99b', pr_url=None, pr_revision=None, pr_num=None)"
`
``
259
`+
]
`
``
260
`+
},
`
``
261
`+
"execution_count": 44,
`
``
262
`+
"metadata": {},
`
``
263
`+
"output_type": "execute_result"
`
``
264
`+
}
`
``
265
`+
],
`
``
266
`+
"source": [
`
``
267
`+
"from huggingface_hub import HfApi\n",
`
``
268
`+
"\n",
`
``
269
`+
"api = HfApi()\n",
`
``
270
`+
"api.upload_file(\n",
`
``
271
`+
" path_or_fileobj="/data/pufanyi/project/lmms-eval-public/tools/data/sample_data.zip",\n",
`
``
272
`+
" path_in_repo="source_data/sample_data.zip",\n",
`
``
273
`+
" repo_id="pufanyi/VQAv2_TOY",\n",
`
``
274
`+
" repo_type="dataset",\n",
`
``
275
`+
")"
`
``
276
`+
]
`
``
277
`+
},
`
``
278
`+
{
`
``
279
`+
"cell_type": "code",
`
``
280
`+
"execution_count": null,
`
``
281
`+
"metadata": {},
`
``
282
`+
"outputs": [],
`
``
283
`+
"source": []
`
``
284
`+
}
`
``
285
`+
],
`
``
286
`+
"metadata": {
`
``
287
`+
"kernelspec": {
`
``
288
`+
"display_name": "lmms-eval",
`
``
289
`+
"language": "python",
`
``
290
`+
"name": "python3"
`
``
291
`+
},
`
``
292
`+
"language_info": {
`
``
293
`+
"codemirror_mode": {
`
``
294
`+
"name": "ipython",
`
``
295
`+
"version": 3
`
``
296
`+
},
`
``
297
`+
"file_extension": ".py",
`
``
298
`+
"mimetype": "text/x-python",
`
``
299
`+
"name": "python",
`
``
300
`+
"nbconvert_exporter": "python",
`
``
301
`+
"pygments_lexer": "ipython3",
`
``
302
`+
"version": "3.11.9"
`
``
303
`+
}
`
``
304
`+
},
`
``
305
`+
"nbformat": 4,
`
``
306
`+
"nbformat_minor": 2
`
``
307
`+
}
`