Revize 3823549c
Přidáno uživatelem Petr Hlaváč před asi 4 roky(ů)
modules/crawler/pipeline.py | ||
---|---|---|
196 | 196 |
ignore_set = database_record_logs.load_ignore_set_loaded(dataset_name) |
197 | 197 |
not_loaded_files = folder_processor.list_of_all_new_files(ignore_set,PROCESSED_DATA_PATH + dataset_path) |
198 | 198 |
|
199 |
print(ignore_set) |
|
200 |
print(not_loaded_files) |
|
201 |
|
|
202 | 199 |
# load every file |
203 | 200 |
for not_loaded_file in not_loaded_files: |
204 | 201 |
# load processed data |
Také k dispozici: Unified diff
Test zpet