Added Job Script Tagger and WebDataset Creator

Browse Source

This script processes image files from tar archives or directories, extracts job IDs, fetches job scripts from database, and identifies application tags based on keywords. Valid samples are saved to sharded WebDatasets while problematic ones are logged.

...

This commit is contained in:

Bole Ma

2025-10-14 15:05:15 +02:00

parent 4adda57539

commit 2bd43009c3

1 changed files with 1174 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

job_script_tagger.py

+1174

File diff suppressed because it is too large Load Diff