INNER CODE UNIT · Python
generate_dataset
openfoodfacts/openfoodfacts-ai · category-detection/main.py:154
def generate_dataset(
parquet_path: Path,
output_path: Path,
country_filter: str | None = None,
remove_duplicates_from_dataset: Path | None = None,
):
"""Generate a dataset file in JSONL format to be used for batch
processing, using Gemini Batch Inference.
Args:
parquet_path (Path): Path to the Parquet file containing product data.
output_path (Path): Path where to write the generated dataset file.
country_filter (str | None): If provided, only products available in
this country (ISO 3166-1 alpha-2 code) will be included.
remove_duplicates_from_dataset (Path | None): If provided, path to a
JSONL file containing previously processed products. Products
present in this file will be excluded from the generated dataset.
We use the "key" field in each record to identify products.