Revize fc552acd
Přidáno uživatelem Petr Hlaváč před asi 4 roky(ů)
| modules/crawler/pipeline.py | ||
|---|---|---|
| 196 | 196 |
ignore_set = database_record_logs.load_ignore_set_loaded(dataset_name) |
| 197 | 197 |
not_loaded_files = folder_processor.list_of_all_new_files(ignore_set,PROCESSED_DATA_PATH + dataset_path) |
| 198 | 198 |
|
| 199 |
print(ignore_set) |
|
| 200 |
print(not_loaded_files) |
|
| 201 |
|
|
| 199 | 202 |
# load every file |
| 200 | 203 |
for not_loaded_file in not_loaded_files: |
| 201 | 204 |
# load processed data |
Také k dispozici: Unified diff
test