| 2023-11-18 11:46:07,262 INFO StreamThr :15138 [internal.py:wandb_internal():86] W&B internal server running at pid: 15138, started at: 2023-11-18 11:46:07.260162 | |
| 2023-11-18 11:46:07,264 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status | |
| 2023-11-18 11:46:07,268 INFO WriterThread:15138 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_114607-nijw2jcj/run-nijw2jcj.wandb | |
| 2023-11-18 11:46:07,270 DEBUG SenderThread:15138 [sender.py:send():380] send: header | |
| 2023-11-18 11:46:07,271 DEBUG SenderThread:15138 [sender.py:send():380] send: run | |
| 2023-11-18 11:46:07,725 INFO SenderThread:15138 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files | |
| 2023-11-18 11:46:07,734 INFO SenderThread:15138 [sender.py:_start_run_threads():1124] run started: nijw2jcj with start time 1700300767.260905 | |
| 2023-11-18 11:46:07,739 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: check_version | |
| 2023-11-18 11:46:07,739 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: check_version | |
| 2023-11-18 11:46:07,788 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: run_start | |
| 2023-11-18 11:46:07,795 DEBUG HandlerThread:15138 [system_info.py:__init__():32] System info init | |
| 2023-11-18 11:46:07,795 DEBUG HandlerThread:15138 [system_info.py:__init__():47] System info init done | |
| 2023-11-18 11:46:07,795 INFO HandlerThread:15138 [system_monitor.py:start():194] Starting system monitor | |
| 2023-11-18 11:46:07,796 INFO SystemMonitor:15138 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2023-11-18 11:46:07,796 INFO HandlerThread:15138 [system_monitor.py:probe():214] Collecting system info | |
| 2023-11-18 11:46:07,797 INFO SystemMonitor:15138 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-11-18 11:46:07,799 INFO SystemMonitor:15138 [interfaces.py:start():190] Started disk monitoring | |
| 2023-11-18 11:46:07,800 INFO SystemMonitor:15138 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-11-18 11:46:07,801 INFO SystemMonitor:15138 [interfaces.py:start():190] Started memory monitoring | |
| 2023-11-18 11:46:07,802 INFO SystemMonitor:15138 [interfaces.py:start():190] Started network monitoring | |
| 2023-11-18 11:46:07,807 DEBUG HandlerThread:15138 [system_info.py:probe():196] Probing system | |
| 2023-11-18 11:46:07,927 DEBUG HandlerThread:15138 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2023-11-18 11:46:07,927 DEBUG HandlerThread:15138 [system_info.py:probe():244] Probing system done | |
| 2023-11-18 11:46:07,928 DEBUG HandlerThread:15138 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:46:07.807399', 'startedAt': '2023-11-18T09:46:07.220916', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=4', '--per_device_eval_batch_size=2', '--gradient_accumulation_steps=4', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.274234771728516}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} | |
| 2023-11-18 11:46:07,928 INFO HandlerThread:15138 [system_monitor.py:probe():224] Finished collecting system info | |
| 2023-11-18 11:46:07,928 INFO HandlerThread:15138 [system_monitor.py:probe():227] Publishing system info | |
| 2023-11-18 11:46:07,928 DEBUG HandlerThread:15138 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |
| 2023-11-18 11:46:07,930 DEBUG HandlerThread:15138 [system_info.py:_save_pip():68] Saving pip packages done | |
| 2023-11-18 11:46:07,931 DEBUG HandlerThread:15138 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |
| 2023-11-18 11:46:08,731 INFO Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt | |
| 2023-11-18 11:46:08,734 INFO Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml | |
| 2023-11-18 11:46:10,342 DEBUG HandlerThread:15138 [system_info.py:_save_conda():87] Saving conda packages done | |
| 2023-11-18 11:46:10,345 INFO HandlerThread:15138 [system_monitor.py:probe():229] Finished publishing system info | |
| 2023-11-18 11:46:10,347 DEBUG SenderThread:15138 [sender.py:send():380] send: files | |
| 2023-11-18 11:46:10,348 INFO SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now | |
| 2023-11-18 11:46:10,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:46:10,358 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:46:10,358 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:46:10,585 DEBUG SenderThread:15138 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:46:10,585 DEBUG SenderThread:15138 [sender.py:send():380] send: config | |
| 2023-11-18 11:46:10,586 DEBUG SenderThread:15138 [sender.py:send():380] send: metric | |
| 2023-11-18 11:46:10,586 DEBUG SenderThread:15138 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:46:10,586 DEBUG SenderThread:15138 [sender.py:send():380] send: metric | |
| 2023-11-18 11:46:10,586 WARNING SenderThread:15138 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) | |
| 2023-11-18 11:46:10,587 DEBUG SenderThread:15138 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:46:10,738 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml | |
| 2023-11-18 11:46:10,739 INFO Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:46:10,739 INFO Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-metadata.json | |
| 2023-11-18 11:46:12,338 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:12,731 INFO wandb-upload_0:15138 [upload_job.py:push():131] Uploaded file /tmp/tmpa4run1f0wandb/ypjb894q-wandb-metadata.json | |
| 2023-11-18 11:46:12,746 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:46:14,754 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:46:17,498 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:22,499 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:25,356 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:46:25,357 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:46:25,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:46:27,587 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:32,588 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:37,592 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:37,851 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml | |
| 2023-11-18 11:46:40,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:46:40,357 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:46:40,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:46:43,525 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:48,525 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:53,526 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:46:55,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:46:55,357 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:46:55,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:46:58,604 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:03,604 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:07,802 DEBUG SystemMonitor:15138 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2023-11-18 11:47:07,803 DEBUG SenderThread:15138 [sender.py:send():380] send: stats | |
| 2023-11-18 11:47:08,804 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:10,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:47:10,357 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:47:10,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:47:14,615 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:19,615 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:24,616 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:25,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:47:25,358 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:47:25,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:47:30,557 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:35,558 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:37,804 DEBUG SenderThread:15138 [sender.py:send():380] send: stats | |
| 2023-11-18 11:47:40,357 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:47:40,358 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:47:40,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:47:41,534 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:46,535 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:51,536 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:47:55,358 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:47:55,358 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:47:55,405 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:47:56,604 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:48:01,605 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:48:06,654 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:48:07,194 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:48:07,805 DEBUG SenderThread:15138 [sender.py:send():380] send: stats | |
| 2023-11-18 11:48:09,201 INFO Thread-12 :15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:48:10,113 DEBUG SenderThread:15138 [sender.py:send():380] send: exit | |
| 2023-11-18 11:48:10,113 INFO SenderThread:15138 [sender.py:send_exit():585] handling exit code: 1 | |
| 2023-11-18 11:48:10,113 INFO SenderThread:15138 [sender.py:send_exit():587] handling runtime: 122 | |
| 2023-11-18 11:48:10,125 INFO SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 11:48:10,127 INFO SenderThread:15138 [sender.py:send_exit():593] send defer | |
| 2023-11-18 11:48:10,127 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,127 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2023-11-18 11:48:10,128 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,128 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 0 | |
| 2023-11-18 11:48:10,128 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 1 | |
| 2023-11-18 11:48:10,128 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,128 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2023-11-18 11:48:10,129 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,129 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 1 | |
| 2023-11-18 11:48:10,129 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 2 | |
| 2023-11-18 11:48:10,129 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,130 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2023-11-18 11:48:10,130 INFO HandlerThread:15138 [system_monitor.py:finish():203] Stopping system monitor | |
| 2023-11-18 11:48:10,130 DEBUG SystemMonitor:15138 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2023-11-18 11:48:10,130 DEBUG SystemMonitor:15138 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2023-11-18 11:48:10,131 INFO HandlerThread:15138 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-11-18 11:48:10,131 INFO HandlerThread:15138 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-11-18 11:48:10,138 INFO HandlerThread:15138 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-11-18 11:48:10,138 INFO HandlerThread:15138 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-11-18 11:48:10,138 INFO HandlerThread:15138 [interfaces.py:finish():202] Joined network monitor | |
| 2023-11-18 11:48:10,139 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,139 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 2 | |
| 2023-11-18 11:48:10,139 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 3 | |
| 2023-11-18 11:48:10,140 DEBUG SenderThread:15138 [sender.py:send():380] send: stats | |
| 2023-11-18 11:48:10,140 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,140 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2023-11-18 11:48:10,141 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,141 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 3 | |
| 2023-11-18 11:48:10,141 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 4 | |
| 2023-11-18 11:48:10,141 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,141 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2023-11-18 11:48:10,141 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,142 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 4 | |
| 2023-11-18 11:48:10,142 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 5 | |
| 2023-11-18 11:48:10,142 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,142 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2023-11-18 11:48:10,143 DEBUG SenderThread:15138 [sender.py:send():380] send: summary | |
| 2023-11-18 11:48:10,144 INFO SenderThread:15138 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 11:48:10,145 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,145 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 5 | |
| 2023-11-18 11:48:10,145 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 6 | |
| 2023-11-18 11:48:10,146 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,146 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2023-11-18 11:48:10,146 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,146 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 6 | |
| 2023-11-18 11:48:10,146 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 7 | |
| 2023-11-18 11:48:10,146 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:48:10,147 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,147 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2023-11-18 11:48:10,147 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,147 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 7 | |
| 2023-11-18 11:48:10,205 INFO Thread-12 :15138 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json | |
| 2023-11-18 11:48:10,727 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 8 | |
| 2023-11-18 11:48:10,727 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,728 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2023-11-18 11:48:10,728 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,729 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 8 | |
| 2023-11-18 11:48:10,729 INFO SenderThread:15138 [job_builder.py:build():298] Attempting to build job artifact | |
| 2023-11-18 11:48:10,733 INFO SenderThread:15138 [job_builder.py:_get_source_type():439] no source found | |
| 2023-11-18 11:48:10,734 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 9 | |
| 2023-11-18 11:48:10,734 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:10,734 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2023-11-18 11:48:10,735 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:10,735 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 9 | |
| 2023-11-18 11:48:10,735 INFO SenderThread:15138 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2023-11-18 11:48:11,113 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:48:11,219 INFO SenderThread:15138 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:48:11,220 INFO SenderThread:15138 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files | |
| 2023-11-18 11:48:11,222 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml conda-environment.yaml | |
| 2023-11-18 11:48:11,225 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml config.yaml | |
| 2023-11-18 11:48:11,229 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log output.log | |
| 2023-11-18 11:48:11,233 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt requirements.txt | |
| 2023-11-18 11:48:11,237 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-metadata.json wandb-metadata.json | |
| 2023-11-18 11:48:11,238 INFO SenderThread:15138 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json wandb-summary.json | |
| 2023-11-18 11:48:11,241 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 10 | |
| 2023-11-18 11:48:11,242 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:48:11,242 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:11,242 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2023-11-18 11:48:11,243 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:11,244 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 10 | |
| 2023-11-18 11:48:11,244 INFO SenderThread:15138 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 11:48:11,604 INFO wandb-upload_0:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/conda-environment.yaml | |
| 2023-11-18 11:48:11,638 INFO wandb-upload_1:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/config.yaml | |
| 2023-11-18 11:48:11,717 INFO wandb-upload_3:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/requirements.txt | |
| 2023-11-18 11:48:11,718 INFO wandb-upload_4:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/wandb-summary.json | |
| 2023-11-18 11:48:11,733 INFO wandb-upload_2:15138 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114607-nijw2jcj/files/output.log | |
| 2023-11-18 11:48:11,934 INFO Thread-11 (_thread_body):15138 [sender.py:transition_state():613] send defer: 11 | |
| 2023-11-18 11:48:11,935 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:11,935 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2023-11-18 11:48:11,936 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:11,936 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 11 | |
| 2023-11-18 11:48:11,936 INFO SenderThread:15138 [file_pusher.py:join():181] waiting for file pusher | |
| 2023-11-18 11:48:11,937 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 12 | |
| 2023-11-18 11:48:11,937 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:11,937 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2023-11-18 11:48:11,938 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:11,938 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 12 | |
| 2023-11-18 11:48:11,939 INFO SenderThread:15138 [file_stream.py:finish():595] file stream finish called | |
| 2023-11-18 11:48:12,113 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:48:12,475 INFO SenderThread:15138 [file_stream.py:finish():599] file stream finish is done | |
| 2023-11-18 11:48:12,475 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 13 | |
| 2023-11-18 11:48:12,476 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:48:12,476 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:12,476 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2023-11-18 11:48:12,476 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:12,477 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 13 | |
| 2023-11-18 11:48:12,477 INFO SenderThread:15138 [sender.py:transition_state():613] send defer: 14 | |
| 2023-11-18 11:48:12,477 DEBUG SenderThread:15138 [sender.py:send():380] send: final | |
| 2023-11-18 11:48:12,477 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:48:12,477 DEBUG SenderThread:15138 [sender.py:send():380] send: footer | |
| 2023-11-18 11:48:12,478 INFO HandlerThread:15138 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2023-11-18 11:48:12,478 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:48:12,478 INFO SenderThread:15138 [sender.py:send_request_defer():609] handle sender defer: 14 | |
| 2023-11-18 11:48:12,479 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:48:12,479 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:48:12,479 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:48:12,479 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: server_info | |
| 2023-11-18 11:48:12,480 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:48:12,480 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: get_summary | |
| 2023-11-18 11:48:12,480 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: server_info | |
| 2023-11-18 11:48:12,480 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2023-11-18 11:48:12,482 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:48:12,482 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: job_info | |
| 2023-11-18 11:48:12,642 DEBUG SenderThread:15138 [sender.py:send_request():407] send_request: job_info | |
| 2023-11-18 11:48:12,642 INFO MainThread:15138 [wandb_run.py:_footer_history_summary_info():3760] rendering history | |
| 2023-11-18 11:48:12,642 INFO MainThread:15138 [wandb_run.py:_footer_history_summary_info():3792] rendering summary | |
| 2023-11-18 11:48:12,643 INFO MainThread:15138 [wandb_run.py:_footer_sync_info():3719] logging synced files | |
| 2023-11-18 11:48:12,643 DEBUG HandlerThread:15138 [handler.py:handle_request():146] handle_request: shutdown | |
| 2023-11-18 11:48:12,643 INFO HandlerThread:15138 [handler.py:finish():866] shutting down handler | |
| 2023-11-18 11:48:13,482 INFO WriterThread:15138 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_114607-nijw2jcj/run-nijw2jcj.wandb | |
| 2023-11-18 11:48:13,642 INFO SenderThread:15138 [sender.py:finish():1536] shutting down sender | |
| 2023-11-18 11:48:13,643 INFO SenderThread:15138 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 11:48:13,643 INFO SenderThread:15138 [file_pusher.py:join():181] waiting for file pusher | |